KR20210027006A - Apparatus, method and computer program for categorizing video based on machine learning - Google Patents

Abstract

Provided is a device for classifying frames in a video, which includes: an input unit for receiving a training dataset for video data; an extraction unit for extracting a characteristic frame of the video data from the input training dataset; and a classification unit configured to classify a frame of the video in consideration of an equal maximum margin proof rate problem with respect to the extracted characteristic frame and the training dataset, and classify the video based on the classified frame.

Description

Apparatus, method, and computer program for classifying video based on machine learning {APPARATUS, METHOD AND COMPUTER PROGRAM FOR CATEGORIZING VIDEO BASED ON MACHINE LEARNING}

The present invention relates to an apparatus, a method, and a computer program for classifying video based on machine learning.

Recently, with the popularization of the Internet and the rapid development of computer communication technology, the demand for new multimedia information services is increasing, and due to this, the emergence of various application fields that require multimedia information processing has resulted in the efficient processing of a large amount of multimedia information. There is an increasing need to develop technologies that can store, search and reproduce.

Among multimedia information, video information is emerging as an important element in various fields such as broadcasting, education, publishing, and libraries, and video search is a technology that studies how to efficiently find desired information among video data stored in a large database. As a result, it has become the core of research on video information processing.

In this regard, Korean Patent Registration No. 10-1826669, which is a prior art, discloses a video search system and method thereof.

In recent years, it is possible to automatically classify video categories by grasping the situations represented by images through artificial intelligence. However, automatic classification of video categories has a disadvantage in that it is very difficult due to ambiguity and insufficient information of the situation represented by the video.

An object of the present invention is to provide a machine learning-based video classification apparatus, method, and computer program for extracting feature frames of video data from the input training dataset when a training dataset for video data is input.

Video classification based on machine learning that classifies frames of video by considering the problem of Balanced Maximal Margin Evidence Rate for characteristic frames and training datasets of video data, and classifies video based on the classified frames. It is intended to provide an apparatus, method and computer program.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides an input unit for receiving a training data set for video data, an extraction unit for extracting a characteristic frame of the video data from the input training data set, and Classification unit for classifying the frame of the video in consideration of a balanced maximal margin evidence rate problem for the extracted feature frame and the training dataset, and classifying the video based on the classified frame It is possible to provide a video classification device including.

Another embodiment of the present invention includes the steps of receiving a training dataset for video data, extracting a feature frame of the video data from the input training dataset, and with respect to the extracted feature frame and the training dataset. It is possible to provide a video classification method including classifying a frame of the video in consideration of a uniform maximum margin evidence rate problem, and classifying the video based on the classified frame.

In another embodiment of the present invention, when the computer program is executed by a computing device, the computer program receives a training dataset for video data, extracts a feature frame of the video data from the input training dataset, and the extracted A computer program stored in a medium comprising a sequence of instructions for classifying a frame of the video based on the classified frame, and classifying the frame of the video in consideration of a characteristic frame and a uniform maximum margin evidence rate problem for the training dataset. Can provide.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, when the training dataset is given in the form of a classified video, frame classification and video classification cannot be performed because frame classification information of the video does not exist. An apparatus, a method, and a computer program for classifying frames and videos in consideration of a maximum margin evidence rate problem can be provided.

For video data, a positive frame is allocated to become a positive video, and the frame of the video is classified by performing simulated annealing in consideration of the equal maximum evidence rate problem so that the positive frames are evenly distributed in each positive video An apparatus, a method, and a computer program for classifying videos based on may be provided.

It is possible to provide an apparatus, a method, and a computer program for classifying a class of a video based on the class of a frame classified most for video data.

1 is a block diagram of an apparatus for classifying a video based on machine learning according to an embodiment of the present invention.
2 is an exemplary diagram for explaining a process of extracting a feature frame in a video classification apparatus according to an embodiment of the present invention.
3 is a flowchart of a method of classifying a video in a machine learning-based video classification apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, and one or more other features, not excluding other components, unless specifically stated to the contrary. It is to be understood that it does not preclude the presence or addition of any number, step, action, component, part, or combination thereof.

In the present specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized by using two or more hardware, or two or more units may be realized by one piece of hardware.

In this specification, some of the operations or functions described as being performed by the terminal or device may be performed instead in a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for classifying a video based on machine learning according to an embodiment of the present invention. Referring to FIG. 1, the video classification apparatus 100 may include an input unit 110, an extraction unit 120, and a classification unit 130.

)를 입력받을 수 있다. The input unit 110 is a training dataset for video data (

) Can be entered.

)는 비디오 데이터로부터 소정의 시간 간격으로 샘플링된 프레임의 트레이닝 데이터(

), 비디오 데이터(

)의 클래스(

), 프레임의 클래스의 수(L), 트레이닝 데이터의 샘플 수(N) 등을 포함할 수 있다. Training dataset (

) Is training data of frames sampled at predetermined time intervals from video data (

), video data (

) Of class(

), the number of classes of frames (L), the number of samples of training data (N), and the like.

The sampled frame may be a frame in which an event can be recognized by including the face of the object. The face of the object plays an important role in characterizing the event, and when using the face of the object, it has the advantage of being able to recognize the event of a video with a single picture without the need for a long image. Accordingly, after excluding a frame without an object, it is possible to reduce input data by sampling at a specific time interval (eg, 3 seconds) and reduce the correlation coefficient between frames to prevent overlapping between features.

The extraction unit 120 may extract a feature frame of video data from the input training data set. Here, the characteristic frame of the video data may be a frame in which a level does not exist. A process of extracting a feature frame of video data will be described in detail with reference to FIG. 2.

2 is an exemplary diagram for explaining a process of extracting a feature frame in a video classification apparatus according to an embodiment of the present invention.

The extraction unit 120 may extract a characteristic frame using a reduced frame network (RFN). The reduced frame network generates sampled frame input, secures the independence between frames by using a feature extraction model to which domain adaptation (DA) is applied, and distribution of the training dataset and the distribution of the test dataset through domain adaptation. By overcoming the difference, the classification accuracy for the test dataset can be improved, thereby minimizing test errors.

The extraction unit 120 may extract component values of the test dataset by applying a kernel preset for the test domain, and generate a partial space for the test domain based on the component values of the extracted test dataset. Here, the test data set may include test data of frames sampled from video data at predetermined time intervals, the number of frames of test data, and the like.

)을 추출할 수 있다. The extraction unit 120 projects the training dataset onto the subspace, so that a feature frame adapted to the test domain from the training dataset projected onto the subspace (

) Can be extracted.

Referring to FIG. 2, the extraction unit 120 performs sampling 210 on the video data 200 at about 3 second intervals, and applies the kernel PCA 240 to each sampled frame 230 to obtain characteristics. The frame 250 can be extracted. At this time, the extraction unit 120 maps the test domain expressed as a subspace so that the kernel PCA is applied to the training data of the training domain for domain adaptation 220 (Domain Adaption). The frame 250 can be extracted.

)의 클래스(

), 트레이닝 데이터의 샘플 수(N), 비디오 데이터(

)의 프레임 수를 이용할 수 있다. Returning to FIG. 1 again, the classification unit 130 classifies the frames of the video in consideration of the problem of a balanced maximal margin evidence rate for the extracted feature frames and training datasets, and based on the classified frames. So you can classify your videos. In this case, in order to classify the frames of the video in consideration of the uniform maximum margin evidence rate problem, the video data (

) Of class(

), number of samples of training data (N), video data (

) Can be used.

)에 대하여 초기화를 수행할 수 있다. 예를 들어, 분류부(130)는 다음의 수학식 1을 이용하여 각 클래스(

)에 대하여 초기화를 수행할 수 있다. The classification unit 130 includes each class of the extracted feature frame (

) Can be initialized. For example, the classification unit 130 uses each class (

) Can be initialized.

)에 기초하여 균등 최대 마진 증거율 문제를 고려하여 비디오의 프레임을 분류할 수 있다. 비디오의 프레임을 분류하는 과정에 대해서는 다음의 수학식 2 및 3을 통해 설명하도록 한다. Thereafter, the classification unit 130 includes each class of the characteristic frame (

Based on ), the frame of the video can be classified in consideration of the problem of the uniform maximum margin evidence rate. The process of classifying the frames of a video will be described through Equations 2 and 3 below.

를

로 초기화할 수 있다. Referring to Equations 2 and 3, the classification unit 130 may convert the equal maximum margin evidence rate problem of Equation 2 into a simple margin problem as shown in Equation 3. At this time, the classification unit 130

To

Can be initialized with

The classifier 130 initializes a frame of negative video data among the video data to a negative frame (-1) based on the converted simple margin problem, and then initializes a frame of negative video data to a frame of positive video data. On the other hand, it can be initialized with a positive frame (+1). At this time, as the simulated annealing proceeds, the frame of the positive video data converges to an optimal solution while the label changes, but the frame of the negative video data may not change to'-1'.

The classifier 130 may derive an objective function value for the equal maximum margin evidence rate problem based on the number to which the least positive frames are allocated among the positive video data and an optimal objective function value for a simple margin problem. The process of deriving the objective function value for the uniform maximum margin evidence rate problem will be described through Equations 4 and 5.

는

가 '+1'로 할당된

의 집합을 나타내며,

가 주어지면,

의 모든 값이 정해질 수 있어 균등 최대 마진 증거율 문제가 RBF 커널의 단순 마진 문제로 줄어들게 될 수 있다. 이 때, 단순 마진 문제의 최적 목적함수 값을

로 두게 되면, 균등 최대 마진 증거율에 대한 목적함수 값은 수학식 5와 같이 도출될 수 있다. 여기서,

는 포지티브 비디오 데이터에 할당된 포지티브 프레임의 수를 나타내고,

는 포지티브 비디오 데이터 중 포지티브 프레임이 가장 적게 할당된 수를 나타낸다. Referring to Equation 4, of Equation 4

Is

Is assigned as'+1'

Represents a set of,

Is given,

Since all values of can be set, the uniform maximum margin evidence rate problem can be reduced to a simple margin problem of the RBF kernel. At this time, the optimal objective function value of the simple margin problem

If set to, the objective function value for the uniform maximum margin evidence rate can be derived as shown in Equation 5. here,

Represents the number of positive frames allocated to the positive video data,

Denotes the number in which positive frames are allocated the least among positive video data.

), 초기 온도(T), 온도 감소율(

), 내부 최대 반복 수(

) 등을 포함하는 파라미터를 이용하여 시뮬레이티드 어닐링(Simulated Annealing)을 수행할 수 있다. 여기서, 마진 에러에 대한 규제 상수(C) 및 커널 파라미터(

)는 커널 SVM(Support Vector Machine)에 관한 규제 상수이며, 최대 증거율에 대한 규제 상수(C')는 포지티브 인스턴스 밸런스를 위한 규제 상수로, 여러 번의 시행을 통해 최적값이 도출될 수 있다. When the initialization is completed, the classification unit 130, a regulation constant for margin error (C), a regulation constant for maximum evidence rate (C'), kernel parameter (

), initial temperature (T), temperature reduction rate (

), internal maximum number of iterations (

) And the like can be used to perform simulated annealing. Here, the regulatory constant for the margin error (C) and the kernel parameter (

) Is a regulatory constant for the kernel SVM (Support Vector Machine), and the regulatory constant (C') for the maximum evidence rate is a regulatory constant for positive instance balance, and an optimum value can be derived through several trials.

):

=0.95, 내부 최대 반복 수(

):

=100으로 설정될 수 있다. 여기서, 내부 최대 반복 수는 주어진 온도에서 최대 반복 가능한 수로 온도 길이라고도 하며, 경우에 따라 더 크게 조정될 수도 있다. The parameters for the simulated annealing are, for example, initial temperature (T): T = 100, temperature reduction rate (

):

=0.95, internal maximum number of iterations (

):

Can be set to =100. Here, the internal maximum repetition number is the maximum repetition number at a given temperature and is also referred to as a temperature length, and may be adjusted to be larger in some cases.

The classifier 130 may derive at least one optimal solution for the equal maximum margin evidence rate problem through the simulated annealing shown in Table 1 below, and classify the frames of the video based on the derived at least one optimal solution. .

repeat -

Phosphorus random

Select

Change the sign of;

-

if

then

else

Random number between

Choose if

then

endif endif

until

until

,

를 도출할 수 있다. 여기서,

는 최종

에 의한

의 레이블로,

로 구성되고,

는 각 클래스(k)에 대한 균등 최대 증거율 문제에 대한 듀얼 최적해를 나타낼 수 있다. Classification unit 130 is at least one optimal solution

,

Can be derived. here,

Is the final

On by

As the label of,

Consists of,

Can represent the dual optimal solution to the uniform maximum evidence rate problem for each class (k).

The classification unit 130 may derive a final output for the problem of the equal maximum margin evidence rate. The final output will be described through Equation 6.

를 도출하고, 균등 최대 마진 증거율 문제에 대한 최종 출력₍

₎을 도출할 수 있다.Referring to Equation 6, the classification unit 130 uses at least one derived optimal solution to intercept

And the final output ₍

₎ Can be derived.

이면,

를

에 더하고(즉,

로 변경함), 반대로

이면,

를

에 뺄 수 있다(즉,

로 변경함).The classifier 130 may change the label by randomly selecting at least one frame from among the positive video data based on Table 1. For example, the classification unit 130

If,

To

Add to (i.e.

Change to), vice versa

If,

To

Can be subtracted from (i.e.

Changed to).

를 풀어,

를 계산함으로써,

의 레이블이 변경된 효과 값을 산출할 수 있다. 이 때, 해가 개선된 경우,

로 옮겨지고, 그렇지 않은 경우,

의 확률로 옮겨질 수 있다. The classifier 130 may calculate an improved effect value according to the changed label for the positive video data, and classify the frame of the video based on the calculated effect value. For example, the classification unit 130

Loosen it,

By calculating

The effect value of which the label of is changed can be calculated. At this time, if the solution is improved,

Moved to, if not,

Can be transferred with a probability of.

The classifier 130 may classify the frames of the video based on the final output of the equal maximum margin evidence rate problem, and classify the video based on the classified frames. For example, the classifier 130 may classify the video class based on the class of the frame that is most classified for the video data. A process of classifying a frame of a video and a class of a video will be described through Equations 7 and 8.

Referring to Equation 7, the classifier 130 may classify the frames of the video through the frame classifier of Equation 7 generated based on the final output of the Equation 6, the evidence rate problem of the equal maximum margin.

The classification unit 130 will describe a process of classifying a video class through Equation 8.

)에 대해 다수결(Majority Voting)에 의해 해당 비디오에 가장 많이 포함된 프레임의 클래스에 기초하여 비디오의 클래스 및 카테고리를 분류할 수 있다. Referring to Equation 8, the classifier 130 uses the video classifier generated through Equation 8 to generate new video data (

), it is possible to classify the class and category of the video based on the class of the frame most included in the corresponding video by majority voting.

The video classification apparatus 100 may be executed by a computer program stored in a medium including a sequence of instructions for classifying frames of a video. When the computer program is executed by the computing device, it receives a training dataset for video data, extracts feature frames of video data from the input training dataset, and equals maximum margin evidence for the extracted feature frames and training datasets. A sequence of instructions for classifying a frame of a video in consideration of a rate problem and classifying a video based on the classified frame may be included.

Through this process, when the training data is given in the form of a classified video, the video classification apparatus 100 does not have frame classification information of the video, so that a video classification technique using frame classification cannot be applied. The video category can be classified using the VCMIL technique that considers the balanced maximal margin evidence rate, which is improved by the simple mi-MIL technique of.

In addition, the video classification apparatus 100 uses the Balanced Evidence Rate (BER) SVM so that as many positive frames as possible are allocated to the positive video and the positive frames are evenly distributed to each positive video. You can classify the categories of.

3 is a flowchart of a method for classifying frames of video in a machine learning-based video classification apparatus according to an embodiment of the present invention. A method of classifying a frame of a video in the video classification apparatus 100 shown in FIG. 3 includes steps processed in a time series by the video classification apparatus 100 according to the embodiments shown in FIGS. 1 and 2. Therefore, even if omitted below, it is also applied to a method of classifying frames of video performed by the video classification apparatus 100 according to the exemplary embodiment illustrated in FIGS. 1 and 2.

In step S310, the video classification apparatus 100 may receive a training dataset for video data.

In step S320, the video classification apparatus 100 may extract a feature frame of video data from the input training dataset.

In step S330, the video classification apparatus 100 may classify a frame of a video in consideration of an equal maximum margin evidence rate problem with respect to the extracted feature frame and the training dataset.

In step S340, the video classification apparatus 100 may classify a video based on the classified frame.

In the above description, steps S310 to S340 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be switched.

The method for classifying frames of video in the video classification apparatus described with reference to FIGS. 1 to 3 may be implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. have. In addition, the method of classifying frames of video in the video classification apparatus described with reference to FIGS. 1 to 3 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: video classification device
110: input unit
120: extraction unit
130: classification unit

Claims (20)

In the device for classifying videos,
An input unit for receiving a training data set for video data;
An extraction unit for extracting a feature frame of the video data from the input training data set; And
Classification unit for classifying the frames of the video in consideration of the problem of a Balanced Maximal Margin Evidence Rate for the extracted feature frames and the training dataset, and classifying the video based on the classified frames
Including a video classification device.
The method of claim 1,
The extraction unit extracts the feature frame adapted to the test domain from the training dataset projected on the subspace by projecting the training dataset onto a subspace of the test domain generated based on component values of the test dataset. That, a video classification device.
The method of claim 2,
The training dataset includes at least one of training data of a frame sampled from the video data at a predetermined time interval, a class and class of the training data, and the number of classes of the frame,
Wherein the test data set includes at least one of test data of frames sampled from the video data at predetermined time intervals and the number of frames of the test data.
The method of claim 2,
The video classification apparatus, wherein the characteristic frame of the video data is a frame in which a level does not exist.
The method of claim 1,
Wherein the classification unit initializes each class of the extracted feature frame.
The method of claim 5,
The classification unit converts the uniform maximum margin evidence rate problem into a simple margin problem based on each class of the characteristic frame, and a frame of negative video data among the video data based on the converted simple margin problem. The video classification apparatus of claim 1, wherein the frame is initialized to a negative frame and a frame of positive video data is initialized to a positive frame.
The method of claim 6,
The classification unit derives an objective function value for the uniform maximum margin evidence rate problem based on the number of the positive frames allocated the least among the positive video data and an optimal objective function value for the simple margin problem. Classification device.
The method of claim 6,
When the initialization is completed, the classification unit is simulated using a parameter including at least one of a regulation constant for a margin error, a regulation constant for a maximum evidence rate, a kernel parameter, an initial temperature, a temperature reduction rate, and an internal maximum number of iterations. To perform annealing (Simulated Anealing), video classification apparatus.
The method of claim 8,
The classification unit derives at least one optimal solution for the equalized maximum margin evidence rate problem through the simulated annealing and classifies the frames of the video based on the derived at least one optimal solution. .
The method of claim 9,
The classification unit randomly selects at least one frame from among the positive video data to change a label, calculates an improved effect value according to the changed label for the positive video data, and calculates the video based on the calculated effect value. The video classification device is to classify the frames of.
The method of claim 1,
The classifying unit classifies the video class based on the class of the frame most classified for the video data.
In a method for classifying a video in a video classification device,
Receiving a training dataset for video data;
Extracting a feature frame of the video data from the input training dataset;
Classifying the frames of the video in consideration of the extracted feature frame and the uniform maximum margin evidence rate problem for the training dataset; And
Classifying the video based on the classified frame
Including, video classification method.
The method of claim 12,
The step of extracting the feature frame of the video data,
Projecting the training dataset onto a subspace of a test domain generated based on component values of the test dataset; And
And extracting the feature frame adapted to the test domain from a training dataset projected onto the subspace.
The method of claim 12,
Classifying the frames of the video,
And performing initialization for each class of the extracted feature frame.
The method of claim 14,
Classifying the frames of the video,
Converting the uniform maximum margin evidence rate problem into a simple margin problem based on each class of the characteristic frame;
Initializing a frame of negative video data among the video data as a negative frame and initializing a frame of positive video data as a positive frame based on the converted simple margin problem Including, video classification method.
The method of claim 15,
Classifying the frames of the video,
Including the step of deriving an objective function value for the uniform maximum margin evidence rate problem based on the number of the positive frames allocated to the least number of the positive video data and an optimal objective function value for the simple margin problem, How to classify videos.
The method of claim 15,
Classifying the frames of the video,
When the initialization is completed, simulated annealing using a parameter including at least one of a regulation constant for a margin error, a regulation constant for a maximum evidence rate, a kernel parameter, an initial temperature, a temperature reduction rate, and an internal maximum number of repetitions. Anealing) of the method comprising the step of performing.
The method of claim 17,
Classifying the frames of the video,
Deriving at least one optimal solution to the uniform maximum margin evidence rate problem through the simulated annealing; And
Classifying the frames of the video based on the derived at least one optimal solution.
The method of claim 12,
Classifying the video,
Classifying the class of the video based on the class of the frame most classified for the video data.
A computer program stored in a computer-readable medium comprising a sequence of instructions for classifying video, comprising:
When the computer program is executed by a computing device,
Receive a training dataset for video data,
Extracting a feature frame of the video data from the input training dataset,
Classify the frames of the video in consideration of the extracted feature frame and the uniform maximum margin evidence rate problem for the training dataset,
A computer program stored on a medium comprising a sequence of instructions for causing classification of the video based on the classified frame.

Images