KR101563297B1 - Method and apparatus for recognizing action in video - Google Patents

Abstract

The present invention relates to a method and an apparatus for recognizing an action in a video. The apparatus for recognizing an action: extracts one or more RGB images and joint data from an input video by using Kinect; stratifies the RGB images and the joint data to extract attributes for action recognition; extracts and combines a first attribute value for the RGB images and a second attribute value for the joint data according to the stratification; extracts at least one third attribute value; groups the third attribute value into a predetermined number of groups by using K-means clustering; converts the groups into a histogram exhibiting an action pattern; and recognizes the action pattern from the histogram.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for recognizing a behavior in a video,

The present invention relates to a method of recognizing a behavior in an image, and more particularly, to a method of extracting data necessary for behavior recognition using a Kinect and learning behavior by layering the extracted data to learn a behavior.

Features designed from existing image or attitude data are well applied in a specific field, but there are disadvantages that it is difficult to apply if the data is changed or the kind of behavior is changed. Therefore, a method of using the features designed by experts with time and effort has been proposed in the method of recognizing the behavior with the designed features. However, there is still a problem that it is difficult to apply to other sensors and data as well as time and effort to improve the performance.

Meanwhile, prior art documents cited below suggest a system and method for real-time recognition of human behavior contained in a video image captured by a video camera. However, the prior art is designed to recognize an action by designing an image-based feature. Thus, it takes a lot of time to design the aforementioned feature, and there is a fatal disadvantage that the designed feature can be used only for a specific sensor.

From this point of view, it can be seen that there is a need for technical means to efficiently reduce the time required to design features necessary for behavior recognition, and to easily extend non-experts.

Patent Document 1: Japanese Patent Application Laid-Open No. 10-2010-0104272 (Sep. 29, 2010)

Therefore, the first problem to be solved by the present invention is to extract RGB image and joint data from an input image using Kinect, to classify the extracted data to learn characteristics, to combine the learned features, To provide a method of recognizing behavior in a video.

Accordingly, a second problem to be solved by the present invention is to extract RGB image and joint data from an input image using a Kinect, to classify the extracted data to learn characteristics, to combine the learned features, To thereby provide a device for recognizing a behavior in a video.

According to an aspect of the present invention, there is provided a behavior recognition apparatus for extracting at least one RGB image and joint data from a input image using a Kinect, Layering the behavior recognition device so as to extract features for behavior recognition with respect to the RGB image and joint data; Deriving one or more third feature values by extracting and combining at least one first feature value for the RGB image and at least one second feature value for the joint data according to the respective layering, ; Grouping the third feature values by a predetermined number using K-means clustering, and converting the group into a histogram representing a behavior pattern; And a step of recognizing a behavior pattern from the histogram by the behavior recognition apparatus.

The joint data including the relative 3D position of the body joints are sequentially stored with respect to time through the Kinect according to the embodiment described above so that the locus of the joints with time is derived, Lt; / RTI >

The extracted RGB image and joint data according to the embodiment may be extracted from the same frame of the image, respectively.

The layer according to an embodiment of the present invention may be a method of recognizing an action in a video, in which the input layer is the RGB image and the joint layer, and the output layer is the first feature value and the second feature value.

The histogram according to the embodiment may be a method of recognizing a behavior in a video, which indicates the frequency of a specific behavior pattern in the input image.

The third feature value is a vector, and the centroid is learned by the predetermined number of groups. The method of recognizing the behavior in the image may be a method of recognizing the behavior in the image.

The step of recognizing the behavior pattern according to the embodiment may include a step of recognizing a behavior pattern according to the behavior type by searching the behavior type mapped with the histogram by the behavior recognition apparatus It can be a way to recognize.

In order to achieve the second object, an extraction unit extracts one or more RGB images and joint data using an input image according to an exemplary embodiment of the present invention. Wherein each of the first and second feature values for the RGB image and the at least one second feature value for the joint image are obtained by extracting features for behavior recognition with respect to the RGB image and the joint data, Extracting and combining the first feature value and the second feature value, deriving one or more third feature values, and grouping the third feature values into a histogram representing a behavior pattern by grouping the third feature values by a predetermined number using KM's clustering; And a recognition unit for recognizing a behavior pattern from the histogram.

The extracting unit may extract the RGB image and the joint data from the same frame of the input image, respectively.

The processing unit according to an embodiment of the present invention is characterized in that the input layer is the RGB image and the joint data, and the output layer is a layer having the first feature value and the second feature value. Lt; / RTI >

The third feature value is a vector, and the centroid is learned by the predetermined number of groups. The device may recognize the behavior in the image.

The recognition unit according to the above-described embodiment may recognize a behavior pattern in a video, by detecting a behavior pattern according to the behavior type by searching for a behavior type mapped with the histogram.

According to the present invention, the RGB image and the joint data are extracted from the input image using the kinect, the extracted data is layered to learn the characteristics, the learned features are combined, It is possible to efficiently reduce the time required for designing the necessary features and to easily expand the non-experts.

FIG. 1 is a flow chart illustrating a method of recognizing a behavior in an image adopted by embodiments of the present invention.
FIG. 2A is a diagram illustrating a hierarchical structure of independent component analysis according to an embodiment of the present invention.
FIG. 2B is a diagram showing a layered result according to an embodiment of the present invention.
FIG. 2C is a diagram illustrating model parameters that are learned and connected according to the layering according to an embodiment of the present invention.
3 is a diagram illustrating a method of applying deep learning to modified posture information according to an embodiment of the present invention.
4 is another diagram illustrating a method of recognizing a behavior in a video according to an embodiment of the present invention.
FIG. 5 is a block diagram illustrating an apparatus for recognizing a behavior in an image adopted by an embodiment of the present invention.
FIG. 6 is a diagram illustrating a result of the present invention in comparison with a conventional technique according to an embodiment of the present invention. Referring to FIG.

Prior to describing the embodiments of the present invention, the technical means adopted by the embodiments of the present invention will be introduced to solve the problems occurring in the conventional behavior recognition system.

Several methods can be used to learn features to overcome the limitations of features designed to recognize behavior in the aforementioned images. In recent years, the Deep Learning approach has been successful, and it has demonstrated high efficiency performance by learning image-based features through deep learning in many areas such as numeric handwriting recognition, object recognition, and scene recognition. In particular, image - based behavior recognition is far superior to using features learned in deep learning than those using existing designed features in Hollywood data sets, which are known to be difficult.

On the other hand, the attitude-based behavior recognition method, which is another approach for behavior recognition, mainly uses accurate joint data. Motion capture data can be used to acquire accurate joint data and convert into attitude-based features to recognize behavior. However, accurate joint data is difficult to acquire without motion capture equipment, and there is a disadvantage that it is expensive to be consumed. Recently, there are many noises using Kinect, but the joint data can be obtained easily and cheaply. However, it is also a disadvantage that too much time and effort are required to design the feature which is useful when the joint data is converted into attitude information exist.

Thus, embodiments of the present invention not only learn posture-based features from joint data obtained from Kinect, but also incorporate image-based features to complement the advantages and disadvantages of image-based and posture-based behavior recognition methods I would like to propose a technical means to

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description and the accompanying drawings, detailed description of well-known functions or constructions that may obscure the subject matter of the present invention will be omitted. It should be noted that the same constituent elements are denoted by the same reference numerals as possible throughout the drawings.

FIG. 1 is a flowchart illustrating a method of recognizing a behavior in an image adopted by an embodiment of the present invention. The behavior recognition apparatus extracts one or more RGB images and joint data from a input image using a Kinect Extracting features for behavior recognition from the RGB image and joint data, extracting features for behavior recognition from the RGB image and joint data, extracting at least one first feature value for the RGB image according to the respective layering and at least one second feature value for the joint data, Extracting and combining the feature values, deriving one or more third feature values, grouping the third feature values by a predetermined number using K-means clustering, By converting the histogram into a histogram, a behavior pattern is recognized from the histogram.

More specifically, in step S110, the behavior recognition apparatus extracts one or more RGB images and joint data from the input image using a Kinect. In other words, it is possible to acquire RGB image, depth image, and skeleton pose data using Kinect device for behavior recognition. Here, the OpenNI library of the Kinect can be used to acquire the RGB image, the depth image, and the joint data.

In addition, the joint data including the relative 3D position of the body joint through the Kinect is sequentially stored with respect to time, so that the locus of the joint can be derived with time. That is, by using the Kinect OpenNI library, it is possible to acquire the joint data. In the acquired joint data, relative X-axis, Y-axis, and Z-axis, which are relative 3D positions from the Kinect, have. Therefore, if you accumulate this information continuously, you can know the skeleton trajectory of how each joint moved with time.

Also, the extracted RGB image and joint data can be extracted from the same frame of the image, respectively. In other words, each of the RGB image, the depth image, and the joint data can be extracted using the Kinect OpenNI library, and the extracted data is synchronized with each other on a frame-by-frame basis. For example, the first frame of the RGB image, the first frame of the depth image, and the first joint data may be data extracted at the same time.

In step S120, the behavior recognition apparatus layered so as to extract features for behavior recognition with respect to the RGB image and the joint data.

 Here, steps S120 and S130 will be described in more detail with reference to FIG. 2A through FIG. 2C.

FIG. 2A illustrates a hierarchical structure of independent component analysis according to an exemplary embodiment of the present invention, and FIG. 2B illustrates a layered result according to an exemplary embodiment of the present invention.

More specifically, FIGS. 2A and 2B may be Independent Subspace Analysis (ICA), which is one of the models used by the present invention to learn features for behavior recognition in the embodiment. Here, in the independent component analysis model of the layer, the input layer may be the RGB image and the joint data, and the output layer may be the first feature value and the second feature value.

In other words, the input layer is the data extracted in step S110, and the output layer may be a feature or a component layer. If the input layer is X, then X can be a observed random variable, and if the output layer is S, S can be a random variable that can not be observed. Here, the input layer X is a transformed two-dimensional image into a one-dimensional vector, and the input layer X and the output layer S may be connected to each other in a fully connected manner. Each connected line is _denoted w _i and is a model parameter, which is an object to be learned. The number of w _i can be (dimension of S) * (dimension of X) because the above network is pulley-connected between S and X.

Therefore, in learning, only the input value X is known, and since S and w _i are unknown, most of the learning in the machine learning uses the optimization method, and the optimization process is as follows. First, an objective function to be optimized is searched by learning a set of model parameters w _i which minimizes or maximizes the function using a design optimization algorithm. . Once the model parameter is learned, it may be as shown in Fig. 2 (c), which is an image of the model parameters, i.e., the vector w _i , connected to s _i , one of S's.

In FIG. 2C, a total of twelve w _i vectors are imaged, and each w _i vector can serve as a feature detector for detecting a specific pattern in an image. Then, when the end of the study the form of a pattern similar to the shape of the w _i characteristic detector present in the image in the recognition process will be the output value of the s _i increases, the opposite case is that the output value of the s _i can be made small. Therefore, in the learning process, the independent component analysis model can be modified by learning w _i , and the output value of the model in the recognition process after the learning is S vector. By analyzing the S vector, it is possible to know what patterns are present in the image.

Returning now to FIG. 1, step S120 and subsequent steps will be described.

In step S130, the behavior recognition device extracts and combines one or more first feature values for the RGB image according to the respective layering and one or more second feature values for the joint data, .

More specifically, in order to learn a feature through a deep network layered to learn a feature for behavior recognition in step S120, an output value S vector can be obtained through the learned independent component analysis (ICA). Thereafter, the obtained output value S vector may be used as an input value X for ICA learning at a higher level, and the above process may be repeated to accumulate the level higher. That is, using the learning process, w _i of a higher-level ICA can be learned to detect a more complex feature than a lower-level feature detector. Here, the shape of the feature detector may be slightly different depending on whether the input value X is the RGB image or joint data extracted in step S110, but there is no difference in the learning process.

Therefore, we study the deep network for appearance-based features separately, learn the deep network for pose-based features separately, and input the re-learned data to each network. The S vector (first feature value) which is the final output value of the main image-based DIP network is obtained, and the S vector (second feature value) which is the final output value of the posture-based DIP network is obtained. At this time, if each S vector is 10 dimensions, a final output value S vector (third feature value) of 20 dimensions in total can be obtained.

3 is a diagram illustrating a method of applying deep learning to modified posture information according to an embodiment of the present invention.

More specifically, in the above figure of FIG. 3, the entire posture information is divided into five parts by the body, the left arm, the right arm, the left leg, and the right leg, Can be learned by sampling for t ₁ time. Then, in the lower figure of FIG. 3, each of the learned first layer networks is copied as necessary and the partial samples are passed through. First, the input of the second layer network is t ₂ and the sampling is performed longer than the time of the first layer network input data. This is equivalent to t ₂ > t _1, and the data having the length of t ₂ can be partially sampled according to the input size t 1 of the first layer network and passed through the copied network of each part. At this time, the equal interval part sampling is performed, and the interval can be arbitrarily set by the user. If all the data in each part passes through the network, the output values can be collected and used as inputs to the second layer network to learn the second layer network. At this time, space-time abstraction can be performed because the whole body data is learned for a longer time than the one-layer network.

Returning now to FIG. 1, step S130 and subsequent steps will be described.

In step S140, the behavior recognition apparatus groups the third feature values by a predetermined number using K-means clustering, and converts the group into a histogram representing a behavior pattern. Here, the histogram represents a frequency of a specific behavior pattern in the input image, and the third characteristic value is a vector, and the centroid can be learned by the predetermined number of groups.

More specifically, the object of the cadence clustering may be the final output value S vector (third characteristic value) of the DIP network learned in step S130, and it is assumed here that the dimension is M. For example, one S vector is generated per sample data, and a M dimension vector of a full bloom sample size can be generated. Here, when the Kminz's clustering is performed, the full sample can be grouped, and the grouping can learn a centroid that is the center of each group. That is, if grouped into five through the clustering, each of the five center points may be an M dimension vector. Here, the number of the groups may be predetermined by a user, and a vector value of each center point, which is an initial value, may be set to a random value.

The clustered cluster is a word when compared to a document, and can be expressed as a histogram representing how often a word appears in a document (such as a political article or a sports article) depending on the type of document. In other words, behavioral awareness can be expressed as a histogram of how often a certain pattern is displayed in a moving picture, and therefore, clusters clustered through a bag of words expression method is called a histogram . ≪ / RTI >

In step S150, the behavior recognition apparatus recognizes a behavior pattern from the histogram. In other words, the behavior recognition apparatus recognizes the behavior pattern according to the behavior type by searching for the behavior type mapped with the histogram.

More specifically, if the histogram obtained in the step S140 and the information on the behavior type about the behavior obtained by observing the behavior are used, the SVM learning process determines what kind of behavior is indicated when the histogram appears, and another type of histogram By acquiring mapping information about what action is to be taken, behavior can be recognized.

(x2-kernel Support Vector Machine) 학습 및 분류과정을 포함할 수 있다. 도 4는 앞서 기술한 도 1의 각 과정에 대응하는 구성을 포함하는 바 도 4의 상세한 설명은 도 1의 상세한 설명으로 대신한다.FIG. 4 is another diagram illustrating a method for recognizing a behavior in an image according to an exemplary embodiment of the present invention. FIG. 4 is a flowchart illustrating an image-based feature learning process, a posture-based feature learning process, Vector quantization using KM's clustering (Vector quanization), labeling of input data

(x2-kernel Support Vector Machine) learning and classification process. FIG. 4 includes a configuration corresponding to each step of FIG. 1 described above, and the detailed description of FIG. 4 replaces the detailed description of FIG.

FIG. 5 is a block diagram illustrating an apparatus for recognizing a behavior in an image adopted in an embodiment of the present invention. In FIG. 5, an apparatus 50 for recognizing a behavior in an image includes a configuration corresponding to each process of FIG. 1 . Therefore, in order to avoid duplication of explanation, the function is outlined mainly in the detailed configuration of the system.

The extracting unit 51 extracts one or more RGB images and joint data from the input image using the key knot.

The processing unit (52) is configured to layer each of the RGB images and the joint data to extract features for behavior recognition, and generate at least one first feature value for the RGB image according to each layering and one or more Extracts and combines the second feature values, derives one or more third feature values, and groups the third feature values by a predetermined number using the Kmuze clustering, thereby converting the group into a histogram representing a behavior pattern .

The recognition unit 53 recognizes the behavior pattern from the histogram.

Further, the extracting unit 51 extracts the RGB image and the joint data from the same frame of the input image.

In addition, the processing unit 52 constitutes a hierarchy in which the input layer is the RGB image and the joint data, and the output layer is the first feature value and the second feature value.

The third feature value is a vector, and the center point is learned by the predetermined number of groups.

Further, the recognition unit recognizes the behavior pattern according to the behavior type by searching for the behavior type mapped with the histogram.

FIG. 6 is a diagram illustrating a result of the present invention in comparison with a conventional technique according to an embodiment of the present invention. Referring to FIG.

More specifically, as a confusion matrix derived from the experimental result according to the embodiment of the present invention, the value inside the matrix means a percentage, and the sum of each row is 100. The index on the left side of the matrix indicates the label of the input data, and the index below the matrix indicates the output label. Therefore, the higher the value of the diagonal line item with the same input and output, the better the hit rate.

The upper picture (61) is a confusion matrix resulting from the image-based behavior recognition, and the average of the diagonal components is 67.5%. In addition, the middle figure (62) is a confusion matrix when only posture-based features are used, and the average of the diagonal components is 56.3%. In the lower part (63), the behavior-aware confusion matrix is characterized by combining the posture-based feature and the image-based feature proposed by the present invention, and the average of the diagonal components is 76.8%. That is, the method of recognizing the behavior combining features of the posture-based feature and the image-based feature proposed by the present invention can confirm the tendency of complementing each feature.

Table 1 below shows the recognition performance using a benchmark data cell called HudaAct dataset.

In Table 1, it can be seen that RGB-STIP, RGBHOGHOF, RGB-SITP, and DepthDescriptor are state-of-the-art technologies, which are only 0.9% different from the method proposed by the present invention. Accordingly, the method proposed by the present invention has little difference in performance between the global recognition technology and performance, and the superiority of the proposed technology is seen from the viewpoint of time investment and cost expenditure to develop it.

Almost all existing behavioral recognition technologies use features designed for behavior recognition. Designing these features requires a deeper understanding of the characteristics of computer vision and joint data, making it almost impossible to create them without expert knowledge of the source. In addition, if the types or types of behaviors that each person recognizes and executes are different, such as mainly hand movements, leg movements, or distant actions, etc., In this study, In other words, developers who want to develop behavior awareness systems must design their own features that can perform well within the domain they are using, which is very difficult due to the aforementioned issues.

Therefore, a technical means which shows similar performance to the system using the features designed by the experts through the learning based system proposed by the present invention described above is proposed. This proposed learning framework can be easily implemented by combining existing open source, etc., and it is possible to implement only the learning data corresponding to the action to be recognized, such as the action on the hand, the action on the bridge, the action taken from afar, It has the effect of automatically learning without having to design related features.

Furthermore, it can easily be applied to unmanned surveillance or service robots because it can recognize complicated behaviors with only one Kinect camera, and it can be applied to other sensors that add other algorithms or extract attitude information according to the purpose.

Meanwhile, the embodiments of the present invention can be embodied as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device and the like, and also a carrier wave (for example, transmission via the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily deduced by programmers skilled in the art to which the present invention belongs.

The present invention has been described above with reference to various embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

50: Apparatus for recognizing an action in a video
51:
52:
53:

Claims (12)

Extracting at least one RGB image and joint data using a Kinect from an input image of the behavior recognition apparatus;
Layering the behavior recognition device so as to extract features for behavior recognition with respect to the RGB image and joint data;
Deriving one or more third feature values by extracting and combining at least one first feature value for the RGB image and at least one second feature value for the joint data according to the respective layering, ;
Grouping the third feature values by a predetermined number using K-means clustering, and converting the group into a histogram representing a behavior pattern; And
Wherein the behavior recognition apparatus recognizes a behavior pattern from the histogram. The method according to claim 1,
Wherein the joint data including the relative 3D position of the body joints is sequentially stored with respect to time through the Kinect to thereby derive trajectories of the joints with time. The method according to claim 1,
And extracting the extracted RGB image and joint data from the same frame of the image, respectively. The method according to claim 1,
Wherein the input layer is the RGB image and the joint layer, and the output layer is the first feature value and the second feature value. The method according to claim 1,
Wherein the histogram represents a frequency of a specific behavior pattern in the input image. The method according to claim 1,
Wherein the third feature value is a vector and the centroid is learned by the predetermined number of groups. The method according to claim 1,
Wherein the step of recognizing the behavior pattern comprises:
And recognizing a behavior pattern according to the behavior type by searching for a behavior type mapped with the histogram by the behavior recognition apparatus. An extracting unit for extracting at least one RGB image and joint data from the input image using a key knot;
Wherein each of the first and second feature values for the RGB image and the at least one second feature value for the joint image are obtained by extracting features for behavior recognition with respect to the RGB image and the joint data, Extracting and combining the first feature value and the second feature value, deriving one or more third feature values, grouping the third feature values by a predetermined number using the cadence clustering, and converting the group into a histogram representing a behavior pattern; And
And a recognition unit for recognizing a behavior pattern from the histogram. 9. The method of claim 8,
The extracting unit extracts,
And extracting the RGB image and the joint data from the same frame of the input image, respectively. 9. The method of claim 8,
Wherein,
Wherein the input layer is the RGB image and the joint data, and the output layer is a layer having the first feature value and the second feature value. 9. The method of claim 8,
Wherein the third feature value is a vector and the centroid is learned by the predetermined number of groups. 9. The method of claim 8,
Wherein,
And a behavior pattern is recognized according to the behavior type by searching for a behavior type mapped with the histogram.

Images