KR101512048B1 - Action recognition method and apparatus based on sparse representation - Google Patents

Abstract

Action recognition method and apparatus based on sparse representation according to the present invention includes the following steps: building up an action feature of a learning video to an action area library and building up a background feature of the learning video into a background area library; extracting features from inputted videos; extracting background usefulness by adjusting the extracted features with the background features of the background area library; when the extracted background usefulness meets the standard threshold values, recognizing the action of the inputted videos on the basis of learning video′s background and action features; and when the extracted background usefulness does not meet the standard threshold values, recognizing the action of the inputted video on the basis of learning video′s action features.

Description

[0001] ACTION RECOGNITION METHOD AND APPARATUS BASED ON SPARSE REPRESENTATION [0002]

The present invention relates to a technique for recognizing human behavior based on a rare expression, and more particularly, to a technique for recognizing a behavior based on a rare expression using a dictionary of behavior-background features acquired from a learning image To a method and apparatus for recognizing a behavior based on a rare expression.

In recent years, the human behavior recognition model, which automatically understands human behavior in video (video), has attracted much attention and research in the field of video analysis due to the interest in the utilization of intelligent video surveillance and human-object interaction. ought.

As is well known, there is a conventional method of recognizing human behavior using a feature vector. In this conventional method, a learning dictionary is extracted from a learning image and a learning dictionary is constructed in advance. When an input image is input, And extracts the input feature vector from the input feature vector.

Then, human behavior is recognized (classified) by matching (adapting) between input feature vectors and learning feature vectors.

That is, the conventional method recognizes human behavior through global matching of feature vectors (learning feature vectors and input feature vectors) extracted from the entire image of learning and input, The conventional method for recognizing the behavior of the image has a fundamental problem in that the real-time property is degraded because all the feature vectors in the image must be matched and the classification performance (recognition performance) is degraded due to the global matching of all the feature vectors I have a problem.

T. Guha and R. K. Ward, " Learning Sparse Representations for Human Action Recognition, " IEEE Trans. Pattern Anal. Mach. Intell., Vol. 34, no. 8, August 2012.

The present invention recognizes human behavior by using both the background feature and the behavior feature of the learning image when the background information is useful based on the result of the concentration measurement of the sparse coefficient, and when the background information is not useful, We propose a new behavior recognition method that can recognize human behavior using only behavioral features.

The problems to be solved by the present invention are not limited to those mentioned above, and another problem to be solved by the present invention can be clearly understood by those skilled in the art from the following description will be.

According to one aspect of the present invention, there is provided a method for generating a learning image, the method comprising: constructing a behavioral feature of a learning image as a behavioral region dictionary and constructing a background feature of the learning image as a background region dictionary; Detecting a background availability degree by adapting the feature to a background feature of the background region dictionary; and determining, based on the background feature and the behavior feature of the learning image, Recognizing a behavior of an input image, and recognizing a behavior of the input image based on a behavior feature of the learning image when the calculated background usability does not satisfy the reference threshold value, Based behavior recognition method.

The learning image of the present invention can be separated into the action area and the background area through background subtraction or motion-based object separation techniques.

According to another aspect of the present invention, there is provided a method for constructing a learning image, the method comprising: dividing the learning image into a behavior region and a background region; extracting a behavior feature and a background feature from the separated behavior region and background region; And constructing the extracted background feature as the background area dictionary.

The behavioral features of the present invention may include global features and local features.

The global characteristic of the present invention can be extracted using a motion history image (MHI).

The background utility of the present invention can be detected based on the background area of behavior classification and the maximum concentration of sparse coefficients associated with the behavioral area.

According to another aspect of the present invention, there is provided an information processing apparatus comprising dictionary construction means for constructing a background region dictionary composed of a behavior region dictionary having behavior characteristics extracted from a learning image and background features extracted from the learning image, A feature adaptation unit adapted to adaptively adapt the feature extracted to the background feature of the background region dictionary to detect a background usefulness; and a feature adaptation unit that, when the detected background usability satisfies a reference threshold value, Recognizing a behavior of the input image based on a behavior feature and a background feature of the background area dictionary, and when the detected background usefulness does not satisfy the reference threshold value, There is provided a behavior recognition apparatus based on a rare expression that includes a behavior recognition unit that recognizes a behavior of an input image.

The dictionary construction means of the present invention may further comprise a region separation unit that separates the learning image into a behavior region and a background region, a behavior feature extraction unit that extracts the behavior feature from the separated behavior region, A background feature extracting unit for extracting the background feature, the behavior area dictionary constructed by the extracted behavior feature, and the background area dictionary constructed with the extracted background feature.

The region dividing unit of the present invention can separate the action region and the background region through background subtraction or motion-based object separation.

The behavior feature extraction unit of the present invention can extract the global feature using a motion history image (MHI).

The feature adaptation unit of the present invention can detect the background usefulness based on the background region of the behavior classification and the maximum concentration of the sparse coefficients associated with the behavioral region.

The present invention separates behavioral features and background features of a learning image into respective dictionaries, and when background information is useful based on the result of measurement of concentration of sparse coefficients, human behavior If the background information is not useful, it is possible to realize real-time human behavior recognition by recognizing human behavior using only the behavior characteristic of the learning image. By applying the matching selectively, it is possible to further enhance the classification performance of the behavior recognition.

FIG. 1 is a block diagram of a rare-expression-based behavior recognition apparatus according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a main process of constructing a behavior area dictionary and a background area dictionary by dividing a learning image into a behavior area and a background area according to an embodiment of the present invention.
3 is a flowchart illustrating a main process of recognizing human behavior based on background features and behavior characteristics of a learning image or recognition of human behavior based on behavior characteristics of a learning image according to an embodiment of the present invention.
4A shows an example of a learning image, FIG. 4B shows an example of a background area separated from a learning image, and FIG. 4C shows an example of a behavior area separated from a learning image.
FIG. 5A shows an example of learning images, FIG. 5B shows an example of background areas and action areas separated from learning images, and FIG. 5C shows a behavioral deconvolution dictionary.
FIG. 6A shows a sparse coefficient distribution graph obtained from a test video of a dive, and FIG. 6B shows a sparse coefficient distribution graph obtained from an example image of a golf swing.

First, the advantages and features of the present invention, and how to accomplish them, will be clarified with reference to the embodiments to be described in detail with reference to the accompanying drawings. While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. It is to be understood that the following terms are defined in consideration of the functions of the present invention, and may be changed according to intentions or customs of a user, an operator, and the like. Therefore, the definition should be based on the technical idea described throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of a rare-expression-based behavior recognition apparatus according to an embodiment of the present invention, and may include a learning image processing module 110, a behavior recognition module 120, and the like.

1, the learning image processing module 110 includes an area separation unit 111, a behavior feature extraction unit 112, a background feature extraction unit 113, a behavior area dictionary 114, and a background area dictionary 115, And the behavior recognition module 120 may include an input feature extraction unit 122, a feature adaptation unit 124, a behavior recognition unit 126, and the like.

First, a behavior image (action video) can be divided into a behavior area (or a foreground area) in which the behavior is included and a background area in which the behavior is performed. The area separation unit 111, as an example, It is possible to provide functions such as dividing the image into a background area and a behavior area shown in Figs. 4B and 4C as an example. Here, separating the regions is to more clearly distinguish the characteristics of the behavior region and the background region.

That is, the present invention proposes a new behavior recognition technique to overcome the absence of a high-performance area automatic separation technique, that is, to provide information of a signal to very small base signals (elements) extracted from a large set of basic signals , atom) as a linear combination of sparse representation is used. Here, the technique of separating the background region and the action region can be achieved by techniques such as background subtraction, motion-based object separation, and the like.

The behavior feature extraction unit 112 may provide a function of extracting a behavior feature from a separated behavior area transmitted from the region separation unit 111. The behavior feature extracted by the feature feature extraction unit 112 may include, It is possible to provide a function of constructing the dictionary 114 and the like.

The background feature extraction unit 113 may provide a function of extracting a background feature from a separated background area transmitted from the region separation unit 111. The background feature extracted by the feature extraction unit 113 may include a background area It is possible to provide a function of constructing the dictionary 115 and the like.

That is, even if the size and position of the action area of the input image (test video) are not known in the classification step, the present invention utilizes the advantage that a dictionary can be constructed using the size and position of the action area of the learning images in the learning step do.

For example, we propose a preconfiguration method for behavior recognition based on rare expressions that is divided into a behavioral domain dictionary and a background domain dictionary to overcome the absence of an appropriate domain separation technique in the testing phase. Here, the behavior area dictionary expresses the characteristic of the non-divided behavioral area of the input image (test video), and the background area dictionary is used to express the characteristic of the background area of the input image (test video). In this way, it is possible to separate the behavioral and background information of the test video before intuition, so that the classification performance can be improved by applying the background information adaptive classification method.

Then, after the background separation, the features describing human behavior characteristics are extracted. These features can be divided into global features and regional features according to the extracted regional characteristics.

Here, the global feature can extract a global feature using, for example, a motion history image (MHI) using the characteristics of the entire space-time volume in which the two-dimensional frame is connected along the time axis. And the pixel intensity at MHI can be obtained as a function of the temporal movement of the point, or the background is static, or the motion of the object can be separated from motion or distracting motion by the camera. Human behavior recognition methods based on global features are sensitive to motion noise, background changes and overlaps in the background. On the other hand, regional features are very robust to background changes, overlapping, rotation, and ratio changes, but they do not capture all of the behavior because they can not express spatial relationships between features.

The Sparse Representation aims to represent the input signal as a linear combination of the base signals (atoms) in a dictionary, which is typically a sparse coefficient vector with most elements consisting of zeros sparse coefficient vector). These rare expressions are useful not only for signal acquisition and compression but also for computer vision such as signal classification, character classification, and face recognition. The following is a description of a more formal SR-based signal classification.

는 i 번째 행동에 속하는 j 번째 학습용 영상의 특징 벡터를 뜻한다. 일반적인 SR을 사용함에 있어서, 시험용 비디오를 표현할 때 사용되는 사전 D는 다음의 수학식 1과 같이 정의될 수 있다.

Is the feature vector of the jth training image belonging to the i- th action. In using a general SR, a dictionary D used for expressing a test video can be defined as the following Equation 1: " (1) "

[Equation 1]

In the above equation (1), N _i and K denote the number of learning images (video) and the total number of behavior classifications belonging to the i-th behavior classification, d denotes the dimension of the feature vector representing the behavior of the image, (Ie, the sum of the number of videos for learning by all behavior categories). The feature vector y of the test video (input video) V can be expressed by the following equation (2) using the given dictionary D.

&Quot; (2) "

는 V의 희소 선형 표현을 나타낸다. 또한

는 i번째 행동 분류의 j번째 학습용 비디오와 관련된 희소계수 값을 의미한다. 희소계수 X를 이용하여 다음의 수학식 3에서와 같이 각 행동 분류의 잔류 오차(residual error)를 연산할 수 있다.In Equation (2)

Represents a sparse linear representation of V. Also

Is the sparse coefficient value associated with the jth learning video of the i-th behavior classification. The residual error of each behavior classification can be calculated by using the scarcity factor X as shown in the following Equation 3. " (3) "

&Quot; (3) "

는 i번째 행동의 잔류오차를 의미하고

는 X에서 i번째 행동 분류에 해당하는 원소들만 0이 아닌 값을 갖는 신규벡터를 나타낸다. 위 과정을 통해 최소 잔류 오차를 갖는 행동으로 시험용 비디오 V를 분류할 수 있다.In the above equation (3)

Denotes the residual error of the i-th action

Represents a new vector having a non-zero value only in the elements corresponding to the i-th behavior classification in X. Through the above procedure, we can classify the test video V into behavior with minimum residual error.

That is, according to the present invention, learning videos are used for constructing an action region-aware dictionary (action region-separated), and in the test stage, behaviors expressed in the test video are classified using a dictionary configured.

For example, in the classification process, the size and position of the action area of the test video is unknown, but the exact action area size and position of the learning videos are known. Based on such an observation, a pre-construction method separated by a behavior area dictionary and a background area dictionary as shown in FIGS. 5A and 5B is presented. These two dictionaries (background area dictionary and action area dictionary) 4 and 5, respectively.

That is, FIG. 5A shows an example of learning images, FIG. 5B shows an example of background areas and action areas separated from learning images, and FIG. 5C shows a behavioral deconvolution dictionary.

&Quot; (4) "

&Quot; (5) "

와

는 i번째 행동 분류의 j번째 학습용 비디오의 배경 특징 벡터와 행동 특징 벡터를 각각 의미한다. 또한

와

는 배경 영역과 행동 영역을 표현하는 사전을 각각 뜻한다.In the above equation

Wow

Is a background feature vector and a behavior feature vector of the jth learning video of the i-th behavior classification, respectively. Also

Wow

Refers to a dictionary representing a background region and a behavior region, respectively.

Using this, the behavioral domain separation type dictionary can be defined as the following Equation (6).

&Quot; (6) "

That is, by expressing the background region and the behavior region information of the preliminary test video of Equation (6) as two separate dictionaries, the information extracted from the background region of the test video and the information extracted from the action region are automatically separated I can do it.

Referring again to FIG. 1, the input feature extraction unit 122 in the behavior recognition module 120 may provide a function of extracting a feature from an input image (test video) and transmitting it to the feature adaptation unit 124 .

The characteristic adaptation unit 124 adapts the characteristic of the test video (input image) transmitted from the input feature extraction unit 122 to the background feature of the learning video (learning image) constructed in the background area dictionary 115, Detection of usefulness, that is, whether or not background information is useful based on the measurement result of the maximum concentration of sparse coefficients related to the background area and the behavior area of the behavior classification, and if the detected background usefulness meets a preset reference threshold value And transmits a result of the determination to the behavior recognition unit 126. [0051]

Next, the behavior recognition unit 126 acquires the behavior characteristics of the behavior area dictionary 114 and the background of the background area dictionary 115 when the background usefulness is notified from the feature adaptation unit 124 that the reference threshold satisfies the reference threshold value A function of recognizing a behavior of an input image (test video) based on a feature and outputting a behavior recognition level value (that is, a behavior classification name expressed using a virtual code) generated as a result of the behavior recognition Can be provided.

In addition, when the behavior recognition unit 126 is notified from the feature adaptation unit 124 that the background availability degree does not satisfy the reference threshold value, the behavior recognition unit 126 acquires the input image (test video) based on the behavior characteristics of the behavior area dictionary 114, And outputting a behavior recognition level value (that is, a behavior classification name expressed using a virtual code) generated as a result of the behavior recognition.

More specifically, given the behavior vector separating type dictionary D and the feature vector y of the test video V, the rare expression of y is as follows.

&Quot; (7) "

와

는 각각 V의 배경과 행동영역을 표현하는 희소선형 표현을 의미한다. 또한

와

는 i번째 행동 분류의 j번째 학습용 비디오의 배경 영역 및 행동 영역과 관련된 희소계수를 뜻한다.From here,

Wow

Represents a sparse linear representation representing the background and behavioral region of V, respectively. Also

Wow

Is the sparse coefficient associated with the background area and the action area of the jth learning video of the i-th behavior classification.

FIG. 6A shows a sparse coefficient distribution graph obtained from a test video for a dive, and FIG. 6B shows a sparse coefficient distribution graph obtained from an example image of a golf swing.

Referring to FIG. 6A, the sparse coefficients corresponding to the background of the test video for the dive are concentrated in a ture class-class 1 interval, which indicates that the background area of the diving is different from the background area of other behaviors it means.

On the other hand, it can be seen that the sparse coefficients corresponding to the background of the test video of the golf swing are spread over several categories, as shown in Figure 6b, because the background of the golf swing is kicking, Riding, walking, and so on.

This observation suggests that the background region information may be effective for behavior classification when the sparse coefficients of the background region are concentrated in the normal classification region. Based on this evidence, the context adaptive classification technique present.

In order to consider the background information in the classification process, it is necessary to measure the concentration of the sparse coefficients. For this, the maximum sparse matrix concentration (MSCC) defined by the following equation (8) can be used.

&Quot; (8) "

That is, given Equation (8), it can be determined whether background information can be used for behavioral classification (availability of background information) according to the criterion of Equation (9).

&Quot; (9) "

는 임계값이며, 본 발명에서는 수학식 9의 값이 참일 경우에만 배경 정보가 유용한 것으로 판단하는데, 이것은 배경 정보가 유용하지 않을 경우, 시험용 비디오의 배경 영역에 해당하는 희소계수들은 시험용 비디오의 행동 영역과 관계된 희소계수들보다 집중도가 낮다는 근거에 기반한다.In the above Equation 9,

In the present invention, it is determined that the background information is useful only when the value of Equation (9) is true. If the background information is not useful, the sparse coefficients corresponding to the background region of the test video are divided into the action region Based on the fact that the concentration is lower than the scarcity coefficients associated with.

In view of the fact that it is difficult to use the residual error of the background region and the behavior region in a balanced manner, the classification scheme based on the conventional rare expression classifies the behavior using the obtained degree of concentration of the sparse coefficients Present a new technique.

That is, when the rare solution of the above-described formula (7) is obtained, the concentration degree of the sparse coefficient of each behavioral classification i can be obtained by the following equations (10) and (11).

&Quot; (10) "

&Quot; (11) "

와

는 각각 i번째 행동 분류의 배경 영역과 행동 영역과 관계된 희소계수의 집중도를 뜻한다. 따라서, 수학식 10과 11의 결과를 이용하여 시험용 비디오 V를, 아래의 수학식 12에서와 같이, 희소계수의 최대 집중도가 나타나는 행동으로 분류할 수 있다.In the above Equations 10 and 11,

Wow

Is the concentration of the sparse coefficients associated with the background and behavioral regions of the i-th behavioral classification, respectively. Therefore, by using the results of Equations 10 and 11, the test video V can be classified as a behavior in which the maximum concentration of the sparse coefficients appears, as shown in Equation 12 below.

&Quot; (12) "

In Equation (12), w denotes a weight, and a value of 0 when the result of Equation (9) is false (that is, when background information is not useful) , And when the background information is useful).

Next, a series of processes for recognizing human behavior based on the rare expression using the behavior recognition apparatus of the present invention having the above-described configuration will be described in detail.

FIG. 2 is a flowchart illustrating a main process of constructing a behavior area dictionary and a background area dictionary by dividing a learning image into a behavior area and a background area according to an embodiment of the present invention.

Referring to FIG. 2, when a learning image (learning video) as shown in FIG. 4A is input as an example (step 202), a technique such as background subtraction, motion-based object separation, 4B and 4C, the background region and the behavior region are separated (Step 204), and the behavior region is transferred to the behavior feature extraction section 112 and the background region to the background feature extraction section 113, respectively .

Next, the behavior feature extraction unit 112 extracts behavior characteristics describing a behavior characteristic of a person from the separated behavior area, and the background feature extraction unit 113 extracts background features from the separated background area (step 206 ).

Then, the action area dictionary 114 of the learning image (learning video) is constructed with the extracted behavior characteristics, and the background area dictionary 115 of the learning image (learning video) is constructed with the extracted background features (step 208 ).

As described above, a series of processes for recognizing human behavior from a behavioral video (test video) using a behavioral domain separation dictionary constructed by dividing a learning image into a background region and a behavioral region will be described.

3 is a flowchart illustrating a main process of recognizing human behavior based on background features and behavior characteristics of a learning image or recognition of human behavior based on behavior characteristics of a learning image according to an embodiment of the present invention.

3, the input feature extracting unit 122 in the behavior recognizing module 120 extracts features from the input image (test video) (step 302) and transmits the extracted features to the feature adapting unit 124 (Step 304).

Next, the feature adaptation unit 124 detects the background availability degree by adapting the characteristics of the test video (input image) transmitted to the background feature of the learning video (learning image) constructed in the background area dictionary 115 306). That is, it is detected whether background information is useful, and then it is determined whether or not the detected background availability satisfies a preset reference threshold value (step 308). Here, the background usefulness can be detected based on, for example, the background region of the behavior classification and the maximum concentration of the sparse coefficients associated with the behavioral region.

If it is determined in step 308 that the detected background usability satisfies the preset reference threshold value, the behavior recognizing unit 126 recognizes the behavior characteristics of the behavior area dictionary 114 and the background area dictionary 115 (Step 310), and outputs a behavior recognition level value (step 314) that is generated as a result of the behavior recognition based on the background feature of the input image (test video).

If it is determined in step 308 that the detected background usability does not satisfy the preset reference threshold value, the behavior recognizing unit 126 recognizes that the input usability of the input image (Test video) (step 312), and outputs a behavior recognition level value that is generated as a result of the behavior recognition (step 314).

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. It is easy to see that this is possible. That is, the embodiments disclosed in the present invention are not intended to limit the scope of the present invention but to limit the scope of the present invention.

Therefore, the scope of protection of the present invention should be construed in accordance with the following claims, and all technical ideas within the scope of equivalents should be interpreted as being included in the scope of the present invention.

Claims (12)

Constructing a behavior feature of the learning image as a behavior area dictionary and constructing a background feature of the learning image as a background area dictionary,
Extracting features from an input image,
Detecting a background availability degree by adapting the extracted feature to a background feature of the background region dictionary;
Recognizing a behavior of the input image based on a background feature and a behavior feature of the learning image when the detected background usefulness satisfies a reference threshold value;
Recognizing a behavior of the input image based on a behavior characteristic of the learning image when the calculated background usefulness does not satisfy the reference threshold value
Based on the behavior of the user.
The method according to claim 1,
The learning image includes:
Background subtraction, or motion-based object separation techniques.
A Rare Representation Based Behavior Recognition Method.
The method according to claim 1,
In the building process,
Dividing the learning image into a behavior area and a background area;
Extracting a behavior feature and a background feature from the separated behavioral region and background region,
Constructing the extracted behavior feature into the behavior area dictionary, and constructing the extracted background feature as the background area dictionary
Based on the behavior of the user.
The method according to claim 1,
The action feature may include:
Contains global and local features.
A Rare Representation Based Behavior Recognition Method.
5. The method of claim 4,
The global feature may include:
Extracted using a motion history image (MHI)
A Rare Representation Based Behavior Recognition Method.
The method according to claim 1,
The above-
Based on the maximum concentration of sparse coefficients associated with the background and behavioral areas of the behavioral classification
A Rare Representation Based Behavior Recognition Method.
Dictionary building means for building a behavior region dictionary having behavior characteristics extracted from the learning image and a background region dictionary having a background feature extracted from the learning image,
An input feature extraction unit for extracting a feature from an input image;
A feature adaptation unit adapted to adapt the feature extracted to the background feature of the background region dictionary to detect the background usability,
Recognizing a behavior of the input image based on a behavior feature of the behavior area dictionary and a background feature of the background area dictionary when the detected background usefulness satisfies a reference threshold value, A behavior recognition unit that recognizes a behavior of the input image based on a behavior characteristic of the behavior area dictionary when the reference threshold value is not satisfied,
Based on the behavior of the user.
8. The method of claim 7,
Wherein the pre-
An area separator for separating the learning image into a behavior area and a background area,
A behavior feature extraction unit for extracting the behavior feature from the separated behavioral region,
A background feature extraction unit for extracting the background feature from the separated background area;
The behavior area dictionary constructed with the extracted behavior feature,
The background region dictionary constructed with the extracted background feature
Based on the recognition result.
9. The method of claim 8,
Wherein the region separator comprises:
A background subtraction or motion-based object separation technique separates the behavior area from the background area
A behavior recognition device based on rare expressions.
9. The method of claim 8,
The action feature may include:
Contains global and local features.
A behavior recognition device based on rare expressions.
11. The method of claim 10,
Wherein the behavior feature extraction unit comprises:
Extracting the global feature using a motion history image (MHI)
A behavior recognition device based on rare expressions.
8. The method of claim 7,
The feature adaptation unit,
Detecting the background usefulness based on the background region of the behavior classification and the maximum concentration of the sparse coefficients associated with the behavioral region
A behavior recognition device based on rare expressions.

Images