KR20160124948A - Tensor Divergence Feature Extraction System based on HoG and HOF for video obejct action classification - Google Patents

Abstract

The present invention relates to a method of extracting feature information for classification of object behavior from video, comprising the steps of: calculating a gradient vector and an optical optical flow vector for a selected key point for video frames of a video to be processed; Obtaining a tensor product of the optical flow vector, calculating a tensor divergence with respect to the tensor product so as to lower the dimension of the tensor product, and determining the calculated tensor divergence as a feature vector for motion classification. According to the feature information extraction method and feature extractor for classifying the video object behavior, it is possible to reflect changes in space and time by fusing the features of HOG and HOF into one feature, while not increasing the dimension of the feature vector, It is possible to improve the performance of the behavior classification in video without increasing the computational complexity of the classifier.

Description

TECHNICAL FIELD The present invention relates to a method and an apparatus for extracting feature information based on HOG / HOF for classification of video object behavior,

The present invention relates to HOG / HOF-based feature information extraction method and feature extractor for video object behavior classification, and more particularly, to feature information extraction method and feature extractor for video object behavior classification for improving classification performance of motion in an object .

A variety of feature extraction methods have been proposed for classifying the behavior of an object such as a human action in a video.

In the case of human behavior, there are various behaviors such as walking, running, jogging, and straightening of the arm. Techniques have been developed to recognize these behaviors.

Particularly, excellent features for expressing object behavior have recently been developed. HOG (histogram of the Grandient) based on the gradient of the image, histogram of optical flow (HOF) based on the optical flow of the image in time, Motion boundary histogram (MBH) and scale invariant feature transformation (SIFT), which use optical flow derivatives that are robust to camera motion.

MBH is obtained for the x-axis and the y-axis, respectively, which is called MBHx and MBHy.

A classifier should be used to classify the behavior of objects from extracted features. Classifiers are HMM (hidden markov model), SVM (support vector machine), Fisher vector based GMM (Gaussian mixture model), and histogram feature based BOF (bag of feature).

For methods of extracting object behavior features, the latest article {H. Wang, A. Klitsch, C. Schmid, and C.-L. Liu, " Dense Trajectories and Motion Boundary Descriptors for Action Recognition, "International Journal of Computer Vision, vol. 103, no. 1, pp. 60-79, (May, 2013).

On the other hand, there is a demand for a feature extraction method in video that can improve the classification performance while reducing the amount of computation of the classifier applied for classifying the behavior of the object.

The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a video object behavior classification method capable of reducing calculation amount while combining gradient information and optical flow information of an image as feature information for classifying a behavior of an object in a video The feature information extracting method and the feature extracting device.

According to an aspect of the present invention, there is provided a feature information extracting method for classifying a video object behavior according to the present invention,

According to the feature information extraction method and feature extractor for video object behavior classification according to the present invention, it is possible to reflect changes in space and time by fusing features of HOG and HOF into one feature, , It provides the advantage that the performance of the behavior classification in video can be improved without increasing the amount of calculation of the classifier as the recognizer.

FIG. 1 is a flowchart illustrating a feature information extraction process for classifying a video object behavior according to the present invention.
FIG. 2 is a flow chart showing in detail the feature information extraction process of FIG. 1,
FIG. 3 is a diagram for explaining an example of application to behavior recognition from feature information extracted according to the present invention,
4 and 5 are diagrams illustrating a process of calculating a histogram from feature information extracted according to the present invention,
6 is a block diagram illustrating a feature information extractor for classifying a video object behavior according to the present invention.

Hereinafter, a feature information extracting method and a feature extractor for classifying a video object behavior according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart showing a feature information extraction process for classifying a video object behavior according to the present invention.

First, a key point for image frames of an input video to be processed is selected (step 10), and a gradient vector g and an optical optical flow vector f are calculated for the selected key point (step 20).

Next, a tensor product of the calculated gradient vector (g) and the optical optical flow vector (f) is calculated (step 30) and a tensor divergence is calculated for the calculated tensor product to lower the dimension of the tensor product ( Step 40).

Then, the calculated tensor divergence is determined as a feature vector for motion classification, and the determined feature vector data for motion classification is provided to a classifier of BoF or SVM to classify the behavior (step 50).

Hereinafter, this processing will be described in more detail with reference to FIGS. 2 to 5. FIG.

라고 하자. 여기서

는 그레디언트(gradient) 벡터이고

는 옵티컬 플로우(opttical flow) 벡터이다. First, when the key points, which are the main pixel points in the video frame of the video, are selected, the feature vector of the key point

Let's say. here

Is a gradient vector and

Is an opttical flow vector.

The tensor product of the gradient vector g and the optical flow vector f can be expressed by the following equation (2).

는 텐서곱을 의미하고, 수학식 2는 아래의 수학식 3과 같다.here

Denotes the tensor product, and the equation (2) is the following equation (3).

는

축(가로축)의 단위벡터,

는

축(세로축)의 단위벡터이다. 결국 두 벡터의 텐서(tensor) 곱은 2x2의 행렬로 표현된다. here

The

The unit vector of the axis (horizontal axis)

The

Is the unit vector of the axis (vertical axis). As a result, the tensor product of the two vectors is expressed by a matrix of 2x2.

Therefore, the gradient vector (g) and the optical flow vector (f) are expressed as an extension of a matrix, and each factor in the matrix is expressed as a correlation between two vector elements.

In this patent, tensor divergence is used to convert a feature extended by a tensor product into a low-dimensional vector.

라고 할 때, 텐서다이버전스(tensor divergence)는 아래의 수학식4 및 5와 정의된다.That is,

, The tensor divergence is defined by the following equations (4) and (5).

Therefore, the gradient-flow tensor divergence (TDGF), which is the feature information generated through step 40, is defined by the following equation (5).

는 2차원 벡터로서, 특징벡터의 차원을 낮추어주며, 그레디언트 벡터(g) 정보와 옵티컬 플로우 벡터(f) 정보를 압축적으로 보여준다. ≪ RTI ID = 0.0 >

Is a two-dimensional vector that lowers the dimension of the feature vector and compressively shows the gradient vector (g) information and the optical flow vector (f) information.

The feature information thus extracted can be used as it is in the next learning and recognition step, and can be applied to the conventional SVM or BOF recognition method by converting it to a feature such as a histogram.

라고 하자. 여기서

는 영상 프레임 인덱스이며,

는 영상으로서

의 행렬로서 표시된다. An example of a key point selection process is described below.

Let's say. here

Is an image frame index,

As an image

Lt; / RTI >

Various methods known in the art may be used to select the keypoints to be traced for the gradient and optical flow for such image frames.

Hereinafter, a method of using the eigenvalues of the 2 ^nd gradient matrix will be described as an example of selecting a key point.

First, the key point is obtained by using the equation (6) below the second gradient (2 ^nd gradient) for each pixel from the input image to be processed video, calculating the eigenvalues (eigen value) of the second gradient, and then, And a pixel having a calculated eigenvalue greater than a set threshold value is determined by a key point.

와

는 임의 픽셀에서

축과

축 방향의 그레디언트(gradient)를 의미하고,

와

는 2^nd gradient이다. 실제 디지털 영상에 있어 이러한 값들은

축 방향에 대하여

그리고

축에 대하여

커널들과의 필터링을 통해 얻어진다. here

Wow

≪ / RTI >

Axis

Means a gradient in the axial direction,

Wow

Is a 2 ^nd gradient. For real digital images these values

About axial direction

And

About axis

It is obtained through filtering with kernels.

에 대하여 고유벡터분할(eigen-vector decomposition)을 구하고, 두 개의 고유값들이 특정 임계값(

)보다 큰 픽셀들을 키포인트(key points)로 정한다. 여기서 특정임계값(

)은 경험적으로 결정된다. 위의 과정에서 그레디언트(gradient) 벡터가 얻어지는데, 그레디언트 벡터는 각 키포인트(key point)에 대하여

와 같다. Given a matrix for each pixel

Eigen-vector decomposition is performed on the two eigenvalues, and two eigenvalues are set to a specific threshold value

) Are defined as key points. Here, a specific threshold value (

) Is determined empirically. In the above procedure, a gradient vector is obtained, and the gradient vector is obtained for each key point

.

에서

번째 특징점

는 다음 프레임

에서 추적되며 아래의 수학식 7과 같이 표현된다.Next, time

in

Second feature point

The next frame

And is expressed by Equation (7) below.

는 시간

에서 키포인트(key point) 위치이며,

은에디안(median) 필터 커널(kernel)이고,ω는

로서 옵티컬 플로우 커널(optical flow kernel)이다. 그러면 옵니컬 플로우 벡터(optical flow vector)는 아래의 수학식 8과 같이 표현된다.here,

Time

Key point position in the < RTI ID = 0.0 >

Is a median filter kernel, and < RTI ID = 0.0 >

Is an optical flow kernel. Then, the optical flow vector is expressed by Equation (8) below.

A method of obtaining an optical flow path is variously known and a detailed description thereof will be omitted.

The gradient vector (g) and the optical flow vector (f) are defined in time and space because they are defined on all frames and on all key points.

,

로 표현된다. 여기서

는 프레임 인덱스

는 키포인트(key points) 인덱스를 의미한다. In other words

,

Lt; / RTI > here

Frame index

Means an index of key points.

Next, the tensor product of the gradient vector and the optical flow vector can be rewritten as Equation (9) below, reflecting the frame index and the keypoint index as described above.

)를 아래의 수학식 10을 통해 얻는다.In addition, the feature vector to which the tensor divergence is applied

) Is obtained by the following equation (10).

Once the feature vectors for behavior classification are obtained as described above, behavior recognition can be performed according to general learning and recognition procedures.

를 행동인식에 적용하는 예를 보이기 위해 BOF 및 SVM(support vector machine)을 사용하는 방법을 도 3 내지 도 5를 함께 참조하여 설명한다.In the present invention,

To support the behavior recognition, a method of using BOF and SVM (support vector machine) will be described with reference to FIG. 3 to FIG. 5. FIG.

를 적용한 행동분류(action classification)의 방법에 대하여 설명한다. Referring first to Figure 3,

And a method of action classification using the method.

에 대하여

서술자(descriptor)를 계산한다. 이를

라고 하자. 이를 위하여 먼저

특징이 2차원 공간의 벡터이므로, 이를 양자화 한다. 이를 위해 벡터양자화기(vector quantizer)를 사용하면 되고, 모든

특징 벡터들이 모여진 TDGF 특징벡터 풀로부터 k-means 집단화를 통해

개의 대표 벡터를 얻는다. 이를

라고 하자. 그러면 입력되는 하나의

는 가장 작은 거리를 갖는

에 할당된다. 여기서

는 가장 작은 거리를 갖는 대표벡터의 인덱스이다. As shown in FIG. 3,

about

Calculates a descriptor. This

Let's say. To do this,

Since the feature is a vector of two-dimensional space, it is quantized. To do this, you can use a vector quantizer,

Through the k-means aggregation from TDGF feature vector pools with feature vectors

Of representative vectors are obtained. This

Let's say. Then,

Has the smallest distance

Lt; / RTI > here

Is the index of the representative vector having the smallest distance.

의 벡터코딩은

과 같다.then

The vector coding of

Respectively.

그리고

의 시공간상 블록을 설정한다. 그리고 이 블록은 도 4에 예시된 바와 같이

의 부블록(subblock)으로 나누어진다. 다음 이 부블록 내에 존재하는 각 픽셀들에 대하여

를 계산하여 이들을 연쇄시킨다. 이를

로 정의하는데,

의 차원은

가 된다. For each of the following key points:

And

Block in the space-time. 4,

The subblocks are divided into subblocks. Next, for each pixel in the subblock

Are calculated, and these are concatenated. This

In addition,

The dimension of

.

를 프레임 지속시간

내 모든 키포인트(key points) 및 부블록들에 대하여 계산하고 히스토그램

를 계산한다. 이를 위해 먼저

에 대한 대표 벡터를

특징벡터풀로부터 k-means 집단화를 통해 얻는다. 단 대표의 개수를

라고 하자. 다음 입력되는

프레임내 모든 키포인트(key points) 및 부블록들에 대하여

들을 계산하고, 이 정보와

의 벡터정보를 이용하여 하나의 특징벡터

를 계산한다. 이를 위해 한 특징벡터

가 대표

번째에 할당되면

의

번째 값을 1가 증가 시키면 된다. 그러면

프레임에 대해 하나의

를 얻을 수 있는데, 이

는

의 벡터이다. 물론 이렇게 얻어진 히스토그램(histogram)에 대해서 정규화(normalization)을 수행한다. 그러면 하나의 액션(action) 비디오 클립에 대하여 하나의

의 특징벡터를 얻을 수 있다. 이 특징 벡터를

라고 하자. 그러면 특징벡터는

의 인덱스를 사용하여 표현되는데,

전체 행동의 가지수로서

이고

은 각 행동에 대한 표본 비디오 클립 수로서

와 같이 표현된다. 마지막으로 특징벡터

를 사용한 인식은 비선형커널(non-linear kernel)을 채용한 SVM(support vector machine)을 이용하여 이루어진다.Next, as shown in Figs. 4 and 5,

Frame duration

Calculate for all my key points and subblocks and use the histogram

. First,

Representative vector for

The feature vector is obtained from the pool by k-means aggregation. However,

Let's say. Then entered

For all key points and subblocks in a frame

, And the information

Using one vector < RTI ID = 0.0 >

. For this purpose,

Representative

Th

of

1 < / RTI > then

One for frame

Can be obtained.

The

. Of course, normalization is performed on the thus obtained histogram. Then, for a single action video clip,

Can be obtained. This feature vector

Let's say. Then the feature vector

≪ RTI ID = 0.0 >

As a branch of total action

ego

Is the number of sample video clips for each action

. Finally,

Is performed using a support vector machine (SVM) employing a non-linear kernel.

The characteristic information extractor of the present invention to which this extraction method is applied is shown in Fig.

6, the feature information extractor 100 includes an input unit 110, a keypoint extractor 120, a feature information extractor 130, and a classifier 140.

The input unit 110 receives the image data to be processed.

The keypoint extraction unit 120 extracts keypoints from the image data input to the input unit 110 in the manner described above.

The feature information extracting unit 130 obtains a tensor product of the gradient vector g and the flow vector f with respect to the keypoint extracted by the keypoint extractor 120 and calculates a tensor product of the tensor product to lower the dimension of the tensor product. Calculates the divergence, and determines the calculated tensor divergence as a feature vector for motion classification, and provides it to the classifier 140.

The classifier 140 classifies the motion of the object from the feature vector information received from the feature information extractor 130.

를 채용한 결과, 아래의 표1에 나타낸 바와 같이 제안 기술의 특징을 사용한 경우 가장 우수한 성능을 보였음.In order to verify this extraction method, the recognition experiment was performed on the KTH database. As a result, HOG, HOF, MBHx, MBHy,

As shown in Table 1 below, the best performance was obtained when the features of the proposed technique were used.

HOG HOF MbhX MbhY

KTH 93.98 94.44 95.83 94.44 96.30

For reference, a sample image of the KTH data set used in the recognition experiment is as follows.

As described above, according to the feature information extraction method and feature extractor for video object behavior classification according to the present invention, it is possible to reflect changes in space and time by integrating the features of HOG and HOF into one feature, It is possible to improve the performance of the intra-video motion classification without increasing the amount of calculations of the recognizer or classifier.

110: Input unit 120: Keypoint extraction unit
130: Feature information extracting unit 140:

Claims (4)

A method for extracting feature information for object behavior classification from video,
end. Calculating a gradient vector and an optical optical flow vector for a selected key point for video frames of a video to be processed;
I. Obtaining a tensor product of the gradient vector and the optical flow vector;
All. Calculating a tensor divergence with respect to the tensor product to reduce the dimension of the tensor product, and determining the calculated tensor divergence as a feature vector for motion classification. Way. The method of claim 1, wherein the key point is obtained by obtaining a second gradient for each pixel from an input image of a video to be processed, calculating a unique value for the second gradient, and then calculating a pixel whose calculated eigenvalue is greater than a set threshold value And extracting characteristic information for video object behavior classification. The method of claim 1, wherein the feature vector data for motion classification is provided to a classifier of a BoF or an SVM, and the behavior is classified and processed. An input unit for receiving image data;
A key point extracting unit for extracting a key point from the image data input to the input unit;
Calculating a tensor product of the gradient vector with respect to the key vector extracted by the keypoint extractor, calculating a tensor product of the tensor product to lower the dimension of the product of the tensor product, And a feature information extracting unit for extracting feature information from the feature information extracted by the extracting unit.

Images