KR102296318B1 - Apparatus and method for classifying videos - Google Patents

Abstract

The present application relates to a video classification apparatus and a video classification method, and the video classification method according to an embodiment of the present invention includes extracting a plurality of frames from an input target video; extracting a motion vector representing an optical flow between the frames; and inputting information on the motion vector into a pre-learned classification model, and determining whether the moving picture corresponds to the slide show type moving picture.

Description

{Apparatus and method for classifying videos}

The present application relates to a video classification apparatus and a video classification method, and more particularly, to a video classification apparatus and a video classification method capable of distinguishing a slideshow type video from a general video.

Recently, as users' demand for video content increases, slide show type videos are rapidly increasing. A slideshow type video is a video in which one or more slide images are displayed, and since it is easy and convenient to produce, it is mainly used for advertisement or publicity.

The slide show type video may be implemented in a form in which sound or narration is added while one slide image is displayed, and it is also possible to sequentially display a plurality of slide images.

Meanwhile, users may use an Internet search service to find information they want, and at this time, a plurality of slideshow type videos may be included among various searched videos. However, since most of the slideshow-type videos contain advertisements, promotional content, or low-quality content, users tend not to prefer slideshow-type videos.

An object of the present application is to provide an apparatus for classifying a video and a method for classifying a video that can distinguish a slide show type video from a general video.

An object of the present application is to provide an apparatus for classifying a video and a method for classifying a video that can distinguish slideshow type videos by utilizing an optical flow between frames included in a video.

A video classification method according to an embodiment of the present invention relates to a video classification method for discriminating a slide show type video, the method comprising: extracting a plurality of frames from an input target video; extracting a motion vector representing an optical flow between the frames; and inputting information on the motion vector to a pre-learned classification model, and determining whether the target video corresponds to the slideshow type video.

A video classification apparatus according to an embodiment of the present invention includes: a frame extracting unit for discriminating a slide show type video, and extracting a plurality of frames from an input target video; a motion vector extractor for extracting a motion vector representing an optical flow between the frames; and a determining unit configured to input information on the motion vector to a pre-learned classification model to determine whether the target video corresponds to the slideshow type video.

Incidentally, the means for solving the above problems do not enumerate all the features of the present invention. Various features of the present invention and its advantages and effects may be understood in more detail with reference to the following specific embodiments.

According to the video classification apparatus and the video classification method according to an embodiment of the present invention, since a slideshow type video can be distinguished from a general video, it is possible to provide the user with the slideshow type video except for the slideshow type video when providing an Internet search service. . Accordingly, it is possible to increase user convenience and increase the accuracy of a search when providing an Internet search service.

According to the video classification apparatus and video classification method according to an embodiment of the present invention, since the light flow in the video is classified using machine learning, more accurate and quick video classification is possible.

However, the effects that can be achieved by the video classification apparatus and the video classification method according to the embodiments of the present invention are not limited to those mentioned above, and other effects not mentioned are from the description below. It will be clearly understood by those of ordinary skill in the art.

1 is a block diagram showing a video classification apparatus according to an embodiment of the present invention.
2 to 4 are schematic diagrams showing a slide show type moving picture according to an embodiment of the present invention.
5 is a schematic diagram showing the optical flow of a slide show type moving picture and a general image according to an embodiment of the present invention.
6 is a schematic diagram illustrating labeling of a learning video for learning a classification model according to an embodiment of the present invention.
7 and 8 are block diagrams showing the operation of the video classification apparatus according to an embodiment of the present invention.
9 is a graph showing a light flow change pattern of a slide show type video and a general video according to an embodiment of the present invention.
10 and 11 are flowcharts illustrating a video classification method according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves. That is, the term 'unit' used in the present invention means a hardware component such as software, FPGA, or ASIC, and 'unit' performs certain roles. However, 'part' is not limited to software or hardware. The 'unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and 'units' may be combined into a smaller number of components and 'units' or further divided into additional components and 'units'.

In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Recently, as users' demand for video content increases, slide show type videos are rapidly increasing. A slideshow type video is a video in which one or more slide images are displayed, and since it is easy and convenient to produce, it is mainly used for advertisement or publicity.

The slide show type video may be implemented in a form in which sound or narration is added while a single slide image is displayed, and it is also possible to sequentially display a plurality of slide images as shown in FIG. 2 .

In addition, as shown in Fig. 3, the slide image moves in a certain direction within the video, or as shown in Fig. 4, a special effect is added to the slide image, such as zooming in the slide image. Even in one case, it can be viewed as a slide show type video.

Here, in addition to zoom-in, the special effects added to the slide image are various, such as fade-in, fade-out, zoom-out, and sticker attachment to the slide image. can exist

Meanwhile, users may use an Internet search service to find information they want, and at this time, a plurality of slideshow type videos may be included among various searched videos. However, since most of the slideshow-type videos contain advertisements, promotional content, or low-quality content, users tend not to prefer slideshow-type videos.

Therefore, for user convenience, it is necessary to exclude a slide show type video when providing an Internet search service, or to provide it with a lower priority when providing search results. That is, it is possible to consider a method of processing the general video and the slideshow-type video by adding a separate annotation to the slideshow-type videos.

Here, according to the moving picture classification apparatus according to an embodiment of the present invention, since slide show type moving pictures can be distinguished from general moving pictures, it is possible to add a comment to each of the differentiated slide show type moving pictures thereafter. Hereinafter, a video classification apparatus according to an embodiment of the present invention will be described.

1 is a block diagram showing a video classification apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , a video classification apparatus 100 according to an embodiment of the present invention may include a frame extraction unit 110 , a motion vector extraction unit 120 , and a determination unit 130 .

The frame extraction unit 110 may extract a plurality of frames from the received target video V. The frame extraction unit 110 may extract a target number of frames by sampling a specific playback section from the target video V at a reference time interval.

For example, a specific playback section may be set to 15 seconds from the playback time of the target video V, and the reference time interval may be set to 0.3 sec. In this case, since 50 frames are extracted, the target number corresponds to 50 frames. That is, by checking the first 50 frames of the target video V, it can be determined whether the target video V corresponds to the slide-type video V1. However, the length of the specific playback section or the reference time interval can be variously changed depending on the embodiment, and it is also possible to set the specific playback section to an arbitrary point in time other than the start point of the target video V.

Meanwhile, depending on the embodiment, the length of the entire playback section of the target video V may be shorter than the length of the specific playback section. For example, when the length of a specific playback section is 15 seconds, when the length of the entire playback section of the target video V is less than 15 seconds, when sampling at a reference time interval of 0.3 sec, the target number of frames cannot be extracted do. In this case, the frame extraction unit 110 may sample the entire playback section of the target video V at a reference time interval to extract only the maximum possible number of frames.

The motion vector extractor 120 may extract a motion vector representing an optical flow between frames. Here, the motion vector extraction unit 120 may utilize the Lucas-Kanade technique for motion vector extraction, and in addition, various techniques for extracting the light flow may be utilized.

Specifically, as shown in FIG. 5 , motion vectors A indicating a movement amount and a movement direction corresponding to the movement of objects included in a frame may be displayed. Here, the motion vectors may be extracted for each time point (t-1, t, t+1).

As shown in Fig. 5(a), in the case of the slide show type video V1, it can be confirmed that the motion vectors A within the same viewpoint have a constant magnitude and direction. That is, if the motion vectors A at the time point (t) are checked, it can be confirmed that each has the same directionality, and the deviation in magnitude is not large. This is the same at time (t-1) and at time (t+1). Also, it can be confirmed that the motion vectors A have a constant pattern in which the magnitudes of the motion vectors A gradually increase in a certain direction according to the change from the time point (t-1) to the time point (t+1).

On the other hand, in the case of the general image V2 of FIG. 5(b), the motion vectors A within the same view have different magnitudes and directions, respectively, from the time point (t-1) to the time point (t+1). In spite of the change to , it can be confirmed that there is no special pattern in the change of the motion vectors (A).

Accordingly, it is possible to distinguish whether the input target video V is a slide show type video V1 or a general video V2 by using the motion vectors extracted by the motion vector extraction unit 120 .

Specifically, the motion vector extracting unit 120 may sequentially select two consecutive frames from among a plurality of frames, and for each time point corresponding to the selected two frames, the movement amount and movement direction of objects included in the two frames. It is possible to extract a plurality of motion vectors indicated. That is, as shown in FIG. 7 , the motion vector extractor 120 selects the frames at the time (t-1) and the time (t), and then extracts motion vectors corresponding to the time point (t-1) therefrom. Then, after selecting the frames at the time (t) and the time (t+1), motion vectors for the time (t) can be extracted from them. Thereafter, motion vectors for respective viewpoints corresponding to the target number of frames may be extracted in the same manner.

In some embodiments, as shown in FIG. 7 , the motion vector extraction unit 120 separates each motion vector into an x-axis component and a y-axis component, and the standard deviation of the x-axis components and the standard of the y-axis components The deviation can be calculated. For example, a plurality of motion vectors generated at time (t-1) may be separated into an x-axis component and a y-axis component, respectively, and the standard deviation of the x-axis components and the standard deviation of the y-axis components may be calculated, respectively. . Here, the calculated results correspond to the x-axis standard deviation (x-std _(t-1) ) and the y-axis standard deviation (y-std _(t-1) ) at time (t-1).

Thereafter, the standard deviation of the x-axis components and the standard deviation of the y-axis components for each time point may be calculated in the same manner, and may be set as the x-axis standard deviation and the y-axis standard deviation at each time point. Here, the generated x-axis standard deviation and y-axis standard deviation for each viewpoint may be provided to the determining unit 130 as a one-dimensional input thereafter.

On the other hand, when the length of the entire playback section of the target video V is shorter than the length of the specific playback section, since only the maximum possible number of frames is extracted, the number of one-dimensional inputs may be insufficient. For example, if the maximum possible number of frames is 20 and the target number is 30, the one-dimensional input according to the target number is [x-std(i=1~29), y-std(i=1~29) )], so the one-dimensional input is dim(1, 58), that is, a total of 58 dimensions must be input. However, since only 20 frames are actually generated, the corresponding one-dimensional input may include [x-std(i=1-19), y-std(i=1-19)]. That is, since the dimension of the one-dimensional input corresponds to dim(1,38), in order to match the dimension difference, the motion vector extractor 120 performs padding on the dimensions after dim(1,38). can In this case, the motion vector extractor 120 may perform padding in various ways, such as inputting 0 or an arbitrary random value.

The determining unit 130 may input motion vector information to the pre-learned classification model C to determine whether the target video V corresponds to the slide show type video V1. Here, if it is determined that it is not the slide show type moving picture V1, it may be classified as a general moving picture V2.

The motion vector information input by the determining unit 130 may be a one-dimensional input provided from the motion vector extracting unit 120 , and the determining unit 130 may input the one-dimensional input to the classification model C. . In this case, the classification model (C) uses the x-axis standard deviation and y-axis standard deviation for each time point included in the one-dimensional input, and the amount of change of the x-axis standard deviation and y-axis standard deviation between consecutive time points. , it may be determined whether the target video V corresponds to the slide show type video V1.

Specifically, referring to FIG. 9 , in the case of a slide show type video, it can be seen that the x-axis standard deviation and the y-axis standard deviation for each viewpoint change while having a constant pattern, but in the case of a general video, it can be confirmed that there is no constant pattern. That is, it is possible to determine whether the classification model C corresponds to the slideshow type video V1 by checking the pattern of changes in the x-axis standard deviation and the y-axis standard deviation.

Here, in the classification model (C), the x-axis standard deviation and y-axis standard deviation for each time of various types of slideshow type videos, and the amount of change of the x-axis standard deviation and the y-axis standard deviation between consecutive viewpoints are may have been pre-learned. Therefore, the classification model (C) calculates the x-axis standard deviation and y-axis standard deviation for each time point of the already learned slideshow type videos, and the amount of change of the x-axis standard deviation and the y-axis standard deviation between successive time points, It may be determined whether the target video corresponds to a slide show type video by comparing the received one-dimensional input.

The classification model (C) uses a plurality of pre-labeled learning videos according to the supervised learning technique, such as DT (Decision Tree), RF (Random Forest), SVM (Support Vector Machine), and DNN (Deep Neural Network). ) can be learned and formed according to machine learning techniques such as

Specifically, as shown in FIG. 6 , whether each of the learning videos is a slideshow type video, and in the case of a slideshow type video, the number of slide images, whether the slide images are moved, whether special effects are included, etc. are displayed. Each may be labeled to do so. Thereafter, the classification model C can be trained using each labeled learning video. According to an embodiment, a classification model ( It is also possible to create C).

On the other hand, as shown in FIG. 8 , the video classification apparatus 100 according to another embodiment of the present invention does not separate the motion vectors into an x-axis component and a y-axis component, but includes motion vectors that change with time. It is also possible to determine whether the corresponding target video corresponds to a slide show type video based on the 3D input.

That is, the motion vector extractor 120 collects motion vectors for each viewpoint, instead of calculating the standard deviation by dividing the motion vectors into an x-axis component and a y-axis component, so that the motion vectors change over time. You can create 3D input.

Thereafter, the determining unit 130 may input a three-dimensional input to the classification model C, and the classification model C compares changes in motion vectors with time with a learned change pattern, and the target video is displayed as a slide show. It can be determined whether it corresponds to a type video. Here, the classification model (C) is a machine learning technique that processes a three-dimensional input such as 3D-ResNet, SlowFast, R(2+1)D, etc. for a plurality of training videos labeled in advance according to a supervised learning technique, especially convolution ( It may be a pre-generated one by learning according to a machine learning model of the convolution series. That is, the classification model (C) can learn three-dimensional inputs formed by extracting motion vectors for each viewpoint of a plurality of learning videos, and through this, a change pattern corresponding to a slide-type video and a change pattern of a general video are distinguished. can do.

10 and 11 are flowcharts illustrating a video classification method according to an embodiment of the present invention. Here, the moving picture classification method according to an embodiment of the present invention distinguishes a slide show type moving picture from a general moving picture, and depending on the embodiment, each step may be performed by the moving picture classification apparatus.

The slideshow type video is a video in which one or more slide images are displayed, and even when a slide image is moved within the video or a special effect on the slide image is displayed, it may be determined to correspond to the slideshow type video. Here, the special effects include fade-in, fade-out, zoom-in, zoom-out, and sticker attachment for slide images. and may be variously added or changed according to embodiments.

Referring to FIG. 10 , in the video classification method according to an embodiment of the present invention, a plurality of frames may be extracted from the first input target video ( S100 ). That is, by sampling a specific playback section from the target video at a reference time interval, a target number of frames can be extracted. For example, the specific playback section may be set to a section from the playback start time of the target video to 15 seconds, and the reference time interval may be set to 0.3 sec. In this case, it is possible to extract a total of 50 frames and determine whether they correspond to a slide show type video.

Depending on the embodiment, the length of the entire playback section of the target video may be shorter than the length of the specific playback section. In this case, the maximum possible number of frames can be extracted by sampling the entire reproduction section at a reference time interval.

After extracting a plurality of frames from the target video, a motion vector representing an optical flow between the frames may be extracted ( S200 ). Here, the motion vectors can be extracted by using the Lucas-Kanade technique, etc., and each of the extracted motion vectors can represent the amount of motion and the direction of motion according to the motion of objects included in the frame. Accordingly, it is possible to determine whether the target video corresponds to the slide show type video using the motion vectors.

Specifically, as shown in FIG. 11( a ), two consecutive frames may be sequentially selected from among a plurality of frames ( S211 ), and for each viewpoint corresponding to the selected two frames, an object included in the two frames It is possible to extract a plurality of motion vectors indicating the movement amount and the movement direction.

Thereafter, a plurality of motion vectors generated at one time point are separated into an x-axis component and a y-axis component, respectively, and the standard deviation of the x-axis components and the standard deviation of the y-axis components are calculated, and the x-axis standard at the time point It can be set as the deviation and the y-axis standard deviation (S213). In addition, by calculating the standard deviation of the x-axis components and the standard deviation of the y-axis components for each time point in the same way, it can be set as the x-axis standard deviation and the y-axis standard deviation at each time point (S214).

After the motion vectors are extracted, it is possible to determine whether the target video corresponds to the slideshow type video by inputting the motion vector information to the pre-trained classification model (S300). Here, the x-axis standard deviation and y-axis standard deviation for each time point can be input to the classification model, and the classification model is the x-axis standard deviation and y-axis standard deviation for each time point, and the x-axis between consecutive time points. Using the standard deviation and the amount of change of the y-axis standard deviation, it is possible to determine whether the target video corresponds to the slideshow type video. At this time, the classification model uses a plurality of pre-labeled learning videos according to a supervised learning technique, such as DT (Decision Tree), RF (Random Forest), SVM (Support Vector Machine), and DNN (Deep Neural Network). ) may be formed by learning with machine learning techniques such as

On the other hand, according to the video classification method according to another embodiment of the present invention, the target object is not divided into an x-axis component and a y-axis component, but based on a three-dimensional input including motion vectors that change with time. It is also possible to determine whether the video corresponds to a slide show type video. That is, as shown in FIG. 11( b ), in the step of extracting the motion vector ( S200 ), two consecutive frames may be sequentially selected from among the plurality of frames ( S221 ), and corresponding to the selected two frames For each viewpoint, a plurality of motion vectors indicating the movement amount and movement direction of the objects included in the two frames may be extracted (S222). Thereafter, by collecting the motion vectors for each viewpoint, a 3D input in which the motion vectors change according to time may be generated ( S223 ).

In this case, in the determining step ( S300 ), a three-dimensional input may be input to the classification model, and the classification model compares the change in motion vectors over time with the learned change pattern, and the target video is converted to a slide show type video. It can be determined whether At this time, the classification model is pre-created by learning a plurality of training videos labeled in advance according to the supervised learning technique according to a machine learning technique that processes 3D inputs such as 3D-ResNet, SlowFast, and R(2+1)D. can That is, the classification model can learn three-dimensional inputs formed by extracting motion vectors for each viewpoint of a plurality of learning videos, and through this, a change pattern corresponding to a slide-type video and a change pattern of a general video can be distinguished. .

The present invention described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium may continuously store a computer-executable program, or may be temporarily stored for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not limited by the above embodiments and the accompanying drawings. For those of ordinary skill in the art to which the present invention pertains, it will be apparent that the components according to the present invention can be substituted, modified and changed without departing from the technical spirit of the present invention.

100: video classification device 110: frame extraction unit
120: motion vector extraction unit 130: determination unit

Claims (13)

In the video classification method for distinguishing a slide show type video,
extracting a plurality of frames from the received target video;
extracting a motion vector representing an optical flow between the frames; and
Inputting information on the motion vector to a pre-learned classification model, and determining whether the target video corresponds to the slideshow type video,
The step of extracting the motion vector is
sequentially selecting two consecutive frames from among the plurality of frames;
extracting a plurality of motion vectors representing movement amounts and movement directions of objects included in the two frames for each viewpoint corresponding to the two selected frames; and
A video classification method comprising the step of collecting motion vectors for each viewpoint, and generating a three-dimensional input in which the motion vectors change with time.
According to claim 1, wherein the slide show type video
A video classification method in which one or more slide images are displayed, wherein the slide image moves within the video or a special effect on the slide image is displayed.
The method of claim 2, wherein the special effect is
Classification of videos comprising at least one of fade-in, fade-out, zoom-in, zoom-out, and sticker attachment for the slide image Way.
The method of claim 1, wherein extracting the frame comprises:
A video classification method, characterized in that by sampling a specific playback section from the target video at a reference time interval, and extracting a target number of frames.
delete delete delete delete delete The method of claim 1, wherein the determining comprises:
The three-dimensional input is input to the classification model, and the classification model determines whether the target video corresponds to the slideshow type video by comparing the change of the motion vectors with time with a learned change pattern. How to classify videos.
11. The method of claim 10, wherein the classification model is
It is characterized in that a plurality of videos labeled in advance according to a supervised learning technique are learned and formed according to a machine learning technique that processes at least one 3D input of 3D-ResNet, SlowFast, and R(2+1)D. video classification method.
A computer program stored in a medium in combination with hardware to execute the moving picture classification method of any one of claims 1 to 4, 10 and 11.
In the video classification device for distinguishing slide show (slide show) type video,
a frame extraction unit for extracting a plurality of frames from the received target video;
a motion vector extractor for extracting a motion vector representing an optical flow between the frames; and
and a determining unit that inputs information about the motion vector to a pre-learned classification model and determines whether the target video corresponds to the slideshow type video,
The motion vector extraction unit
After sequentially selecting two consecutive frames from among the plurality of frames, and extracting a plurality of motion vectors indicating the amount and direction of movement of the objects included in the two frames for each time point corresponding to the two selected frames, A video classification apparatus, characterized in that by collecting motion vectors for each viewpoint, a three-dimensional input in which the motion vectors change according to time is generated.

Images