KR102574273B1 - Method and apparatus for detecting video motion - Google Patents

Abstract

The present invention relates to video motion detection, comprising: a frame input unit for inputting video frames in a streaming manner; a storage unit outputting a probability value for an action time point from the video frame; and an action instance generation unit generating an action instance based on the probability value and outputting a video motion detection result in an instance unit.

Description

Video motion detection device and method {METHOD AND APPARATUS FOR DETECTING VIDEO MOTION}

The present invention relates to techniques for detecting motion in video in real time.

Understanding what's happening in video is a long-standing research problem, and one that remains challenging in computer vision and machine learning.

Recently, advances in deep learning have attracted the attention of many researchers in the field, which has led to significant progress in various video understanding tasks. Many successful advances in video understanding have been made in various tasks such as, for example, action classification, temporal action localization, video question and answering, and video summarization. Most of these tasks deal with video comprehension tasks in an 'offline setting', a state in which the entire video is accessible at the time of inference.

As streaming video formats for live streaming services and surveillance cameras increase, research on how to process video streams continues to increase. In particular, streaming video is prevalent these days as a tool for remote communication such as lectures, video conferences, and YouTube channels.

When performing the task of understanding these streaming videos, it takes a video of unknown length as input, and cannot access the entire video to understand it, only the past frames to the currently streamed frames.

In order to overcome these limitations, a technique of analyzing video streaming in frame units or a technique of estimating video action has been studied, but accurate motion prediction is difficult and most of them are optimized for an offline environment.

Therefore, there is a need for a new method of video motion detection technology capable of analyzing streaming video in real time in an online environment.

Registered Patent Publication No. 10-2420887 (registration announcement on July 15, 2022) Registered Patent Publication No. 10-0420747 (registration announcement on March 02, 2004)

In an embodiment of the present invention, a video motion detection apparatus and method capable of solving a problem of online temporal action localization by analyzing video streaming in real time in units of action instances are proposed.

In an embodiment of the present invention, it is intended to propose a video motion detection apparatus and method capable of securing high accuracy by taking the context of the entire action of an online video.

In an embodiment of the present invention, an apparatus and method for detecting video motions capable of providing low-latency and high-performance detection results by monitoring action contexts and changes in action contexts are proposed.

The problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by those skilled in the art from the description below. will be.

According to an embodiment of the present invention, a frame input unit for inputting a video frame in a streaming manner; a storage unit outputting a probability value for an action time point from the video frame; and an action instance generator generating an action instance based on the probability value and outputting a result of detecting a video action in an instance unit.

Here, the storage unit may include: a feature extraction unit encoding the video frame to extract features; an action end detector detecting an end point of an action by searching for the feature in a forward pass; and an action start detection unit configured to detect a start point of an action by searching for the feature in a backward pass.

In addition, the action end detection unit may include a multi-head detection unit outputting the feature as two types of probability values; and an action end detection improvement unit outputting a final action end probability value based on the two types of probability values.

In addition, the multi-head detection unit may include an action detection head outputting a probability value that an action exists; and an action end detection head outputting an action end probability value where an end point of the action exists.

In addition, each of the action detection head and the action end detection head may include a long short-term memory (LSTM) network, a fully-connected (FC) layer, and a softmax layer.

Also, the action end detection improvement unit may include an FC layer and a Softmax layer.

In addition, the action start detection unit may include an LSTM network, an FC layer, and a Softmax layer.

In addition, the action start detection unit may output an action start probability value where an action start point exists.

In addition, the action start detection unit may maintain the process of searching in the reverse direction until the point at which the action start probability value increases once and then decreases.

In addition, the action start detection unit may determine an action start probability value of a frame previous to the falling time point as a final action start probability value.

In addition, the action instance creation unit may call the action start detection unit when the action end probability value exceeds a preset threshold value.

In addition, the action instance creation unit generates the action instance based on the start point of the action and the end point of the action, and the action instance may include the final action end probability value, the final action start probability value, and an action class. can

Also, the feature extraction unit may store the features in a sequence form.

A video motion detection method of a video motion detection apparatus according to an embodiment of the present invention, comprising: inputting a video frame in a streaming manner; outputting a probability value for an action time point from the video frame; and generating an action instance based on the probability value and outputting a video motion detection result in an instance unit.

Here, the outputting of the probability value may include extracting a feature by encoding the video frame; detecting an end point of an action by searching the feature in a forward direction; and detecting a start point of an action by searching the feature in a reverse direction.

In addition, the detecting of the end point may include outputting the feature as two types of probability values; and outputting a final action end probability value based on the two types of probability values.

In addition, the step of outputting the two types of probability values may include outputting a probability value for which an action exists; and outputting an action end probability value where an end point of the action exists.

Also, the detecting of the starting point may include searching in the reverse direction until the point at which the action starting probability value increases once and then decreases.

Also, the detecting of the starting point may include determining an action starting probability value of a previous frame at the falling time point as a final action starting probability value.

In addition, the step of detecting the starting point of the action may be performed when the action end probability value exceeds a preset threshold value.

According to an embodiment of the present invention, a computer readable recording medium storing a computer program, the computer program including instructions for causing a processor to perform a video motion detection method, the method comprising: streaming video frames; input in a manner; outputting a probability value for an action time point from the video frame; and generating an action instance based on the probability value and outputting a result of detecting a video action in an instance unit.

According to an embodiment of the present invention, a computer program stored on a computer-readable recording medium, the computer program including instructions for causing a processor to perform a video motion detection method, the method comprising: converting video frames into a streaming method; input; outputting a probability value for an action time point from the video frame; and generating an action instance based on the probability value and outputting a result of detecting a video action in an instance unit.

According to an embodiment of the present invention, by solving the problem of estimating online action time, it is possible to perform real-time motion detection in units of instances by generating a starting point, an ending point, and a class of an action in a streaming video. In addition, according to an embodiment of the present invention, while solving the high responsiveness required in online work by first finding the end point of an action in a forward pass, it is possible to find a start point of an action through a backward pass and take the context of the entire action to ensure high accuracy. can In addition, according to the embodiment of the present invention, when the end of an action is detected, the context of the action is grasped by the multi-head detector, and the changing aspect of the context is observed through the end of action detection improvement unit, so that the delay is lower than that of using a single end of action detector. High-performance detection can be performed.

1 is a block diagram illustrating a video motion detection device 1 according to an embodiment of the present invention.
FIG. 2 is a block diagram showing a specific configuration of the storage unit 20 of FIG. 1 .
FIG. 3 is a block diagram showing a specific configuration of the action end detection unit 210 of FIG. 2 .
FIG. 4 is a block diagram showing a specific configuration of the multi-head detection unit 212 of FIG. 3 .
5 is a conceptual diagram illustrating the detailed configuration and operation process of the storage unit 20 of the video motion detection apparatus 1 according to an embodiment of the present invention by way of example.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms, only the present embodiments are intended to complete the disclosure of the present invention, and those of ordinary skill in the art to which the present invention belongs It is provided to fully inform the person of the scope of the invention, and the scope of the invention is only defined by the claims.

In describing the embodiments of the present invention, detailed descriptions of well-known functions or configurations will be omitted unless actually necessary in describing the embodiments of the present invention. In addition, terms to be described later are terms defined in consideration of functions in the embodiment of the present invention, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification.

As a technology for analyzing streaming video, there are research methods for analyzing streaming video by dividing it into frames, such as online action detection and online detection of action start.

Such online action detection and online detection of action start predict action classes or action start points in units of frames, but they are not necessarily linked to understanding in units of action instances. For example, for a 30-frame video clip containing 3 consecutive jump actions, the approach of online action detection outputs 30 action predictions in 30 frames, and cannot predict that there are 3 jump actions.

Accordingly, in an embodiment of the present invention, a process of grouping frames into action instances is applied to perform an online action time estimation task as an approach of online action detection.

In an embodiment of the present invention, a technique for extending online detection of action start to online detection of action end point, detecting start point and end point, and applying it to an online action time estimation task is proposed.

In an embodiment of the present invention, in order to detect the start and end points of an action as a consistent pair, a technique for determining the most reliable end point candidate of an action necessary for grouping frames between an action start and an action instance is proposed. want to do

In an embodiment of the present invention, it is possible to solve the online-based action time estimation problem by analyzing video streaming in real-time in units of action instances, and to obtain high accuracy by taking the context of the entire action of the online video, and to obtain the context of the action. An apparatus and method for detecting video motions capable of providing low-latency and high-performance detection results by monitoring changes in motion and action contexts are proposed.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a video motion detection device 1 according to an embodiment of the present invention.

As shown in FIG. 1 , an apparatus 1 for detecting video motion according to an embodiment of the present invention may include a frame input unit 10 , a storage unit 20 and an action instance generator 30 .

The frame input unit 10 may receive video frames in real time in a streaming manner.

The storage unit 20 may output a probability value for an action time point from a video frame input through the frame input unit 10 .

The action instance generation unit 30 may generate an action instance based on the probability value stored in the storage unit 20 and output a video motion detection result in an instance unit.

2 is a block diagram showing a specific configuration of such a storage unit 20.

As shown in FIG. 2 , the storage unit 20 may include a feature extractor 200, an action end detector 210, and an action start detector 220. .

The feature extractor 200 may extract a feature by receiving a video frame from the frame input unit 10 in real time.

The action end detection unit 210 may detect the end point of the action by searching for the features extracted by the feature extraction unit 200 in a forward pass.

As shown in FIG. 3 , the action end detector 210 may include a multi-head detector 212 and an end detection refinement 214 .

The multi-head detection unit 212 may serve to output the features extracted by the feature extraction unit 200 with two different probability values.

The action end detection improvement unit 214 may serve to output a final action end probability value based on two types of probability values output from the multi-head detection unit 212 .

4 is a block diagram showing a specific configuration of the multi-head detection unit 212 among the configurations of the action end detection unit 210 of FIG. 3. The multi-head detection unit 212 according to an embodiment of the present invention is an action detection head head 212a) and an action end detection head 212b.

The action detection head 212a may play a role of outputting a probability value of the existence of an action.

The action end detection head 212b may serve to output an action end probability value where an end point of the action exists.

Referring back to FIGS. 1 and 2 , the action start detection unit 220 is a means for outputting an action start probability value where an action start point exists, and searches for features extracted by the feature extractor 200 in a backward pass. It can play a role in detecting the starting point of an action.

The action start detection unit 220 may maintain backward search until the action start probability value rises once and then decreases, and determines the action start probability value of the previous frame at the drop point as the final action start probability value.

In this case, the action start detector 220 may be called by the action instance generator 30 when the action end probability value output from the action end detection head 212b of FIG. 4 exceeds a preset threshold. That is, the action instance generator 30 may monitor the action end probability value of the action end detector 210 and command the action start detector 220 to be called depending on the action end probability value.

Here, the action instance generator 30 may generate the above-described action instance based on the start point of the action and the end point of the action.

The action instance generated by the action instance generator 30 may include the final action end probability value generated by the action end detector 210 and the final action start probability value generated by the action start detector 220 . Also, an action instance may further include an action class.

Accordingly, the action instance generator 30 may generate one action instance by combining the final action end probability value, the final action start probability value, and the action class.

5 is a conceptual diagram illustrating the detailed configuration and operation process of the storage unit 20 of the video motion detection apparatus 1 according to an embodiment of the present invention by way of example.

As shown in FIG. 5 , the feature extraction unit 200 may collect and encode a predetermined number of video frames input in real time through the frame input unit 10 . The feature extractor 200 may generate features through a process of encoding a predetermined number of video frames, and may store the created features in the form of a sequence.

The action end detection unit 210 is a means for detecting an end point of an action by searching for features extracted from the feature extraction unit 200 in a forward direction, and may include a multi-head detection unit 212 and an action end detection improvement unit 214. can

The multi-head detection unit 212 may further include an action detection head 212a and an action end detection head 212b, and each of the action detection head 212a and the action end detection head 212b is LSTM (long short) -term memory) network, a fully-connected (FC) layer, and a softmax (softmax) layer. Accordingly, a probability value of the existence of an action may be output through the action detection head 212a, and an action end probability value of the existence of an end point of the action may be output through the action end detection head 212b.

The action end detection improving unit 214 is a means for outputting a final action end probability value based on two types of probability values output from each of the action end detection head 212a and the action end detection head 212b of the multi-head detection unit 212. , FC layer and softmax layer may be made of a neural network structure.

The action start detection unit 220 is a means for detecting the start point of an action by searching the features extracted from the feature extraction unit 200 in a reverse direction, and may be formed of a neural network structure including an LSTM network, an FC layer, and a softmax layer. .

The action instance generator 30 may generate one action instance by combining the detected action end point (final action end probability value) and the action start point (final action start probability value).

In FIG. 5 , the upper graph represents the final action end probability value, and the lower graph represents the final action start probability value. In each graph, the x-axis represents a video frame and the y-axis represents a probability value.

Here, the action instance may include a vector of an action class, a class probability value, a start point frame, and an end point frame. Accordingly, an output result provided from the video motion detection apparatus 1 by the action instance generated by the action instance generator 30 may be expressed as an action range, an action type, and the like. The action range may mean the above-described action start point and action end point, and the action type may mean a specific sports event such as volleyball and basketball, for example.

According to the embodiment of the present invention as described above, it is possible to perform real-time motion detection in an instance unit by solving the online action time estimation problem, generating the start point, end point, and class of an action in a streaming video, and While solving the high responsiveness required in online work by finding the end point first, it finds the start point of the action through the reverse pass and takes the context of the entire action to ensure high accuracy. is identified, and the changing aspect of the context is observed through the action end detection improvement unit to implement low-latency and high-performance detection rather than using a single action end detector.

Meanwhile, combinations of each block of the accompanying block diagram and each step of the flowchart may be performed by computer program instructions. Since these computer program instructions may be loaded into a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, the instructions executed by the processor of the computer or other programmable data processing equipment are described in each block of the block diagram. It creates means to perform functions.

These computer program instructions may be stored on a computer usable or computer readable medium (or memory) or the like that may be directed to a computer or other programmable data processing equipment to implement functions in a particular manner, so that the computer usable Alternatively, the instructions stored in a computer readable recording medium (or memory) may produce an article of manufacture containing instruction means for performing a function described in each block of the block diagram.

In addition, since the computer program instructions can be loaded on a computer or other programmable data processing equipment, a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to generate a computer or other programmable data processing equipment. Instructions performing possible data processing equipment may also provide steps for executing the functions described in each block of the block diagram.

Also, each block may represent a module, segment, or portion of code including at least one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative embodiments, it is possible for the functions mentioned in the blocks to occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on their function.

1: video motion detection device
10: frame input unit
20: storage unit
30: action instance creation unit
200: feature extraction unit
210: action end detection unit
212: multi-head detection unit
212a: action detection head
212b: action end detection head
214: action end detection improvement unit
220: action start detection unit

Claims (22)

a frame input unit for inputting video frames in a streaming manner;
a storage unit outputting a probability value for an action time point from the video frame; and
An action instance generating unit generating an action instance based on the probability value and outputting a video motion detection result in an instance unit;
the storage unit,
a feature extraction unit for extracting features by encoding the video frame;
an action end detector detecting an end point of an action by searching for the feature in a forward pass; and
An action start detection unit configured to detect a start point of an action by searching for the feature in a backward pass;
The action end detection unit,
a multi-head detection unit outputting the feature with two types of probability values; and
An action end detection improvement unit outputting a final action end probability value based on the two types of probability values;
The action start detection unit outputs an action start probability value where the action start point exists, and maintains the backward search process until the action start probability value increases once and then decreases, and the previous frame at the drop point. Determine the action start probability value as the final action start probability value,
The action instance creation unit creates the action instance based on the start point of the action and the end point of the action, and ends the action by calling the action start detection unit when the action end probability value exceeds a preset threshold value. Depending on the probability value, the action start detection unit is commanded to be called, and the action instance includes the final action end probability value, the final action start probability value, the class of the action, and the class probability value.
Video motion detection device.
delete delete According to claim 1,
The multi-head detection unit,
an action detection head outputting a probability value of an action present; and
An action end detection head outputting an action end probability value at which the end point of the action exists.
Video motion detection device.
According to claim 4,
Each of the action detection head and the action end detection head,
It includes a long short-term memory (LSTM) network, a fully-connected (FC) layer, and a softmax layer.
Video motion detection device.
According to claim 1,
The action end detection improvement unit,
including the FC layer and the Softmax layer.
Video motion detection device.
According to claim 1,
The action start detection unit,
LSTM network, FC layer and Softmax layer.
Video motion detection device.
delete delete delete delete delete According to claim 1,
The feature extraction unit,
Storing the features in sequence form
Video motion detection device.
A video motion detection method of a video motion detection device, comprising:
inputting a video frame in a streaming manner;
outputting a probability value for an action time point from the video frame; and
Generating an action instance based on the probability value and outputting a video motion detection result in an instance unit; Including,
The step of outputting the probability value is,
encoding the video frames to extract features;
detecting an end point of an action by searching the feature in a forward direction; and
Including; detecting a start point of an action by searching the feature in a reverse direction;
Detecting the end point of the action,
outputting the features as two types of probability values; and
Based on the two types of probability values, outputting a final action end probability value; includes,
The step of detecting the starting point of the action includes searching in the reverse direction until a point in time when an action starting probability value at which the starting point of the action exists increases once and then decreases,
The step of detecting the starting point includes determining an action start probability value of a previous frame at the falling time point as a final action start probability value,
The outputting of the video motion detection result in units of instances may include generating the action instance based on a start point of the action and an end point of the action, and when the action end probability value exceeds a preset threshold value. By including the step of detecting the start point of the action in, it is called to detect the action start point dependent on the action end probability value,
The action instance includes the final action end probability value, the final action start probability value, a class of the action, and a class probability value.
Video motion detection method.
delete delete 15. The method of claim 14,
The step of outputting the two types of probability values,
outputting a probability value where an action exists; and
Outputting an action end probability value where an end point of the action exists;
Video motion detection method.
delete delete delete A computer-readable recording medium storing a computer program,
The computer program,
Including instructions for causing a processor to perform a video motion detection method;
The method,
inputting a video frame in a streaming manner;
outputting a probability value for an action time point from the video frame; and
Generating an action instance based on the probability value and outputting a video motion detection result in an instance unit; Including,
The step of outputting the probability value is,
encoding the video frames to extract features;
detecting an end point of an action by searching the feature in a forward direction; and
Including; detecting a start point of an action by searching the feature in a reverse direction;
Detecting the end point of the action,
outputting the features as two types of probability values; and
Based on the two types of probability values, outputting a final action end probability value; includes,
The step of detecting the starting point of the action includes searching in the reverse direction until a point in time when an action starting probability value at which the starting point of the action exists increases once and then decreases,
The step of detecting the starting point includes determining an action start probability value of a previous frame at the falling time point as a final action start probability value,
The outputting of the video motion detection result in units of instances may include generating the action instance based on a start point of the action and an end point of the action, and when the action end probability value exceeds a preset threshold value. Including the step of detecting the starting point of the action, it is called to detect the starting point of the action depending on the action end probability value,
The action instance includes the final action end probability value, the final action start probability value, a class of the action, and a class probability value.
A computer-readable recording medium.
A computer program stored on a computer readable recording medium,
The computer program,
Including instructions for causing a processor to perform a video motion detection method;
The method,
inputting a video frame in a streaming manner;
outputting a probability value for an action time point from the video frame; and
Generating an action instance based on the probability value and outputting a video motion detection result in an instance unit; Including,
The step of outputting the probability value is,
encoding the video frames to extract features;
detecting an end point of an action by searching the feature in a forward direction; and
Including; detecting a start point of an action by searching the feature in a reverse direction;
Detecting the end point of the action,
outputting the features as two types of probability values; and
Based on the two types of probability values, outputting a final action end probability value; includes,
The step of detecting the start point of the action includes searching in the reverse direction until a point in time when an action start probability value at which the action start point exists increases once and then decreases,
The step of detecting the starting point includes determining an action start probability value of a previous frame at the falling time point as a final action start probability value,
The outputting of the video motion detection result in units of instances may include generating the action instance based on a start point of the action and an end point of the action, and when the action end probability value exceeds a preset threshold value. Including the step of detecting the starting point of the action, it is called to detect the starting point of the action depending on the action end probability value,
The action instance includes the final action end probability value, the final action start probability value, the class of the action, and the class probability value.
A computer program stored on a recording medium.

Images