JPH0546583A - Confirmation device for moving body action - Google Patents

Abstract

PURPOSE:To select an action with a high likelihood by acquiring timeseries models in action as determinative state transition models corresponding to respective recognition categories by training based upon learning data and calculating the probability that those models generate actions to be recognized. CONSTITUTION:In learning mode, parameters of state transition models for recognition are estimated from data for learning and stored by recognition categories in a state transition model storage memory 30 for recognition. In recognition mode, the likelihoods of the models which are stored in the state transition model storage memory 30 for recognition by the learning and corresponds to the respective categories are calculated to perform maximum likelihood estimation which employs the category corresponding to the model having the maximum likelihood as a recognition result. Processing up to quantization are the same between the learning and recognition. Consequently, the action of a moving body such as a person in a moving picture can be recognized. Namely, stable processing becomes possible since model fitting is not required.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving object recognition device for recognizing a motion pattern of a human or other moving body from a moving image.

[0002]

2. Description of the Related Art Pattern recognition technology for moving images is
Although many studies have been conducted in recent years, the following two approaches can be broadly divided into those aiming to recognize motions and behaviors.

(1) There is a model-based approach based on a model. This expresses each part of the human body such as the body and arm as a geometric model such as an ellipsoid and a generalized cylinder, and describes the human posture by parameters such as their joint angles.

(2) There is something called a feature-based heuristic approach. For this purpose, human physical distribution is counted for an actual scene image. In this case, for example, a method of counting the number of area regions having a certain size or more in the threshold-processed image is used.

[0005]

In the conventional model-based approach technique of the above (1), it is necessary to fit the model to the image. Therefore, when a noisy real image is targeted, its pattern recognition is not performed. There was a problem of instability.

The conventional heuristic of the above (2)
In the approach technology, it is necessary for a human to construct an ad-hoc heuristic method for processing contents and parameters for each target and each scene.
Only low-level recognition such as counting was difficult, and advanced processing such as behavior recognition was difficult.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a recognition technique for realizing advanced behavior recognition without relying on unstable model fitting. It is to provide the recognition system construction technology.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

[0009]

In order to achieve the above-mentioned object, in the present invention, in an animal body action recognition device for recognizing the action of an animal body such as a human being in a scene, each action of the animal body action is , A method of expressing with a vector of feature quantities such as mesh features extracted from images and direction distribution of optical flow, and a time series model of behavior as a probabilistic state transition model corresponding to each recognition category is acquired by training with learning data. Means and means for calculating the probabilities that these models generate recognition-targeted behaviors, respectively, are characterized by selecting the behavior with the highest likelihood.

[0010]

According to the above-mentioned means, each motion of the behavior of the moving body such as a human being in the scene is extracted by the mesh feature extracted from the image,
Expressed by a vector of feature quantities such as the direction distribution of the optical flow, a time series model of behavior is acquired by training with learning data as a probabilistic state transition model corresponding to each recognition category, and those models generate recognition target behavior. By calculating the respective probabilities, the action with the highest likelihood can be selected, and the action of the moving object such as a person in the moving image can be recognized by learning from the case.

That is, the present invention, in particular, differs from the prior art in that model fitting is not required and stable processing is possible, and since learning is performed from cases, a human can set parameters of the recognition system according to individual situations. The difference is that there is no need to decide.

[0012]

Embodiments of the present invention will now be described in detail with reference to the drawings. In all the drawings for explaining the embodiments, parts having the same function are designated by the same reference numerals, and repeated description thereof will be omitted.

[First Embodiment] FIG. 1 is a block diagram showing a schematic configuration of a first embodiment of a moving object behavior recognition apparatus of the present invention, FIG.
FIG. 3 is a block diagram showing a functional configuration of the moving object behavior recognition device according to the first embodiment. In FIGS. 1 and 2, 11 is an image input device, 12 is a computer, 13 is an external memory device, 21 is an image input unit, 22 is an image memory, 23 is a feature extraction unit, 24 is a feature storage memory, and 25 is a quantum. Avatar, 2
6 is a symbol storage memory, 27 is a likelihood calculation unit, 28 is a recognition result memory, 29 is a model parameter estimation unit, 30
Is a recognition state transition model storage memory. As the recognition state transition model storage memory 30, for example, an external memory device 13 is used.

The basic operation of the first embodiment has two stages of learning and recognition. At the time of learning, the parameters of the recognition state transition model are estimated from the learning data and stored in the recognition state transition model storage memory 30 for each recognition category. At the time of recognition, the likelihood of the model corresponding to each category stored in the recognition state transition model storage memory 30 is calculated by learning, and the maximum likelihood estimation in which the category corresponding to the model having the maximum likelihood is set as the recognition result is performed. To do. The processing up to quantization is the same during learning and recognition. The common part will be described along the flow shown in FIG.

First, a moving image including a person in action is captured from the image input unit 21 of the image input device 11 such as a TV camera and stored in the image memory 22.

Next, the feature extraction unit 23 obtains a plurality of feature quantities from the moving image.

An example of the feature quantity used here is shown below.
First, consider the mesh features shown in FIG. That is, first, the image memory 22 is set to have N × m pixels.
It is divided into xM sub-blocks, and an image is binarized in each of these sub-blocks. Next, it is a method of obtaining the occupancy rate of black pixels in this sub-block and using this as an N × M dimensional feature vector. That is, let a _{ij be} the occupation rate of black pixels of the mesh (i, j), and arrange this vector,

[0018]

## EQU1 ## This is a method in which fm = (a ₀₀ , a ₀₁ , ..., A _ij , ... a _MN ) is used as the feature vector.

Alternatively, there are the following three examples as characteristics using the optical flow as shown in FIG. First, an image is divided into N × M sub-blocks having n × m pixels by using optical flows obtained from images of a plurality of time frames, and a flow direction in each sub-block is divided. Is a feature vector. That is,
Let θ _{ij be} the average direction of the flow vector of the mesh (i, j) (angle formed with the x axis), and arrange this vector,

[0020]

[Equation 2]

Is a feature vector. Second
Similarly, the size of the flow is used as the feature vector. That is, let r _{ij be} the average size of the flow vector of the mesh (i, j), and arrange this,

[0022]

## EQU3 ## This is a method in which f _r = (r ₀₀ , r ₀₁ , ..., r _ij , ... r _MN ) is used as the feature vector.

Thirdly, there is also a method in which the power spectrum of the two-dimensional Fourier transform is divided into appropriate meshes, and the feature vectors having the average power and phase of each mesh as components are similarly used.

After the feature vector is obtained, the quantizer 25
The vector sequence is converted into a symbol sequence by and is recorded in the symbol storage memory 26. This is due to vector quantization. That is, each feature vector is converted into a symbol corresponding to a representative point vector having the shortest distance among them, based on a list of representative points for quantization prepared in advance. This representative point group is called a codebook.
The k-mean method ([1] Hidden Markov Model for Speech Recognitio is used to create the codebook.
n XD Huang, Y.Ariki, MAJack Edinburg Univ.Press p
117), LBG method ([2] An Algorithm for Vector
Quantizer design, Y.Linde, A.Buzo, RMGray IEEE
Trans.Commin. Vol.COM-28, pp, 84-95,1980). In the case of the present embodiment, it is necessary to create a codebook when learning the recognition model, but any method is applicable. Further, the distance measure to be used includes Euclidean distance, Mahalanobis distance in consideration of variance of each dimension, and the like. As will be described later, recognition based on a continuous model that does not require quantization is also possible.

By the processing up to this point, the image series is converted into a symbol string. The operations up to this point are the same both during recognition and during learning. With respect to the flow of the processing thereafter, first, at the time of recognition will be described.

At the time of recognition, these feature vector sequences are
It is recorded in the feature storage memory 24. Then, the likelihood calculating unit 27 calculates the probability of generating this feature vector sequence from each of the models stored in the recognition state transition model storage memory 30 prepared for the number of categories to be recognized. For the following description, the model parameters are defined as follows.

[0027]

[Number 4] T: the observed symbol sequence O = O _1,
Length of O ₂ , ..., O _T N: Number of states in model L: Number of symbols in model S = {s}: Set of states. s _t is the t-th state (not observable)

[0028]

[Equation 5] υ = {υ ₁ , υ ₂ , ..., υ _L }: A set of observable symbols A = {a _ij | a _ij = Pr (s _{t + 1} = j | s _t = i) }: State transition probability. a
_ij is the probability of transition from state i to state j B = {b _j (O _t ) | b _j (O _t ) = Pr (O _t | s _t = j)}: symbol output probability b _j (k) is the state Probability of outputting the symbol υ _k at j π = {π _i | π _i = Pr (s ₁ = i)}: initial state probability The probability that a certain model will generate an observed symbol sequence is the forward algorithm ([1] Hidden Markov Model
for Speech Recognition XDHuang, Y.Ariki, MAJack
Edinburg Univ. Press p148) can be obtained as follows.

Probability Pr (O |) that a model λ = {A, B, π} outputs a symbol sequence O = O ₁ , O ₂ , ..., O _T.
λ) is

[0030]

[Equation 6]

Here, α _T (i) is defined by α _T (i) ≡Pr (O ₁ , O ₂ , ..., O _t , s _t = i | λ). (2), and Specifically,

[0032]

[Equation 7]

It is obtained by the recurrence formula of

The model having the maximum likelihood thus obtained is selected as a recognition result and stored in the recognition result memory 28. The above is the processing flow at the time of recognition.

Next, a processing flow for learning will be described. The model parameter estimation unit 29 estimates, for a symbol sequence obtained from a plurality of learning data given for each category, parameters of a state transition model that generate the symbol sequence, and recognizes the state transition model for recognition. It is stored in the storage memory 30. This is a sequence of symbols,

[0036]

[Equation 8]: When O = O ₁ , O ₂ , ..., O _T is given, the Baum-Welch algorithm ([1] Hi
dden Markov Model for Speech Recognition XDHuan
g, Y.Ariki, MAJack Edinburg Univ.Press p152, [3]
Speech recognition by probabilistic model Seiichi Nakagawa (See p. 55 of The Institute of Electronics, Information and Communication Engineers). Where Baum-Welc
The h algorithm will be described. This is a procedure in which a model parameter having a higher likelihood than that is repeatedly obtained based on a certain model parameter. It is possible to confirm the convergence by confirming the value of the likelihood by the forward algorithm described above every iteration.

[0037]

[Equation 9]

[0038]

[Equation 10]

[0039]

[Equation 11]

[0040]

[Equation 12]

However, here,

[0042]

[Equation 13]

[0043]

[Equation 14]

The parameters of the recognition state transition model corresponding to the learning data can be obtained by the above procedure. The model thus obtained for each category is used for recognition.

As an example of the experimental result of the processing flow described in the first embodiment, the one shown in FIG. 5 will be described. In this example, four behaviors (the right hand is raised and then lowered, the left hand is raised and lowered, and the right and left feet are the same) are shown as the behaviors to be recognized. In FIG. 5, a series of five pieces arranged side by side shows an example of each operation. The operation is to raise the right hand, the left hand, the right foot, and the left foot from the above. 3 for each of these categories
Two trials were performed, one of which was used as learning data and the other of which was used as recognition experiment data. A mesh feature was used as the feature vector. In the quantization, the vectors of the 20 images shown in FIG. 5 were used as the representative points. That is, among the 20 feature vectors f _q (i), (i = 1, ..., 20) obtained from these images, the feature vector f _{m is determined} by the one having the closest Euclidean distance.
(J) is quantized into a symbol Oj, which is represented by O _j = argmin _i {f _m (j), f _q (i)}. These 20 representative points will form a codebook. In this experimental example, in order to simplify the creation of the codebook, k-
An appropriate image during the operation was selected regardless of the codebook creation algorithm such as the mean method.

Data for recognition experiments were applied to the four state transition models generated by learning, and the one having the maximum likelihood among the four models was selected as the recognition result.
The symbols (here, numbers) corresponding to each image are shown in FIG.
Shown in. Next, the results of the experiment are shown. An example of a symbol string corresponding to the operation (raising the left foot) shown in FIG. 7 is shown. FIG. 8 shows the result when this symbol string is used as the recognition data. This is the likelihood of the four models with respect to the recognition data (motion = raise left foot). It can be seen that models belonging to the same category have a high likelihood and are correctly recognized. When the recognition rate of each combination was examined by sequentially using each of the three trials as the learning data in the same test, the average recognition rate was 88%. 88 for 3 people used in the experiment
%, 88%, 96%, and an average result of 90% was obtained.

[Second Embodiment] FIG. 9 is a block diagram showing the functional arrangement of a second embodiment of the present invention. The second embodiment is an example in which conversion into symbols by quantization is not required, and as shown in FIG. 9, the continuous model likelihood calculating unit 61, the continuous model parameter estimation without passing through the quantization unit from the feature memory. In the unit 63, recognition and learning processes are performed, respectively. In this case, each state of the state transition model is formulated as a probability vector for outputting a feature vector. That is, it is assumed that the probability density function in which the feature vector is output from each state is expressed in the form of a mixture of normal distributions. Therefore, the parameters of the model use the mean and variance of the normal distribution mixed in place of the symbol output probabilities in the above-mentioned example, and the weighting coefficient of those mixtures, but the basic operation is the This is the same as the first embodiment. Reference numeral 62 is a recognition continuous state transition model storage memory.

Although the present invention has been specifically described based on the embodiments, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Nor.

[0049]

As described above, according to the present invention, a behavior recognition system based on moving images can be constructed by learning from a case. Adaptable and highly recognizable. Also, since it does not include model fitting on the image,
Robust processing can be realized even for real images. Therefore, the present invention can be widely applied to suspicious behavior monitoring in banks and shops, cutting out desired motion parts from moving images such as sports, and the like.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of a first embodiment of a moving object behavior recognition device of the present invention,

FIG. 2 is a block diagram showing a functional configuration of a moving object behavior recognition device according to the first embodiment,

FIG. 3 is a diagram for explaining mesh features of the first embodiment,

FIG. 4 is a diagram for explaining a feature amount using the optical flow according to the first embodiment,

FIG. 5 is a diagram showing a representative image of an experiment target operation of the first embodiment,

FIG. 6 is a diagram showing symbols corresponding to a representative image of the first embodiment,

FIG. 7 is a diagram showing an example of a symbol string corresponding to a certain operation of the first embodiment,

FIG. 8 is a diagram showing an experimental result of the first embodiment,

FIG. 9 is a block diagram showing a functional configuration of a second embodiment of the present invention.

[Explanation of symbols]

11 ... Image input device, 12 ... Computer, 13 ... External memory device, 21 ... Image input device, 22 ... Image memory, 23 ... Feature extraction unit, 24 ... Feature storage memory, 25 ...
Quantization unit, 26 ... Symbol string storage memory, 27 ... Likelihood calculation unit, 28 ... Recognition result memory, 29 ... Model parameter estimation unit, 30 ... Recognition state transition model storage memory,
41 ... Example of mesh feature extraction image, 51 ... Example of optical flow, 61 ... Continuous model likelihood calculation unit, 62 ... Recognition continuous state transition model storage memory, 63 ... Continuous model parameter estimation unit.

Claims (1)

[Claims] 1. A moving object recognition apparatus for recognizing a moving object such as a human being in a scene, wherein each motion of the moving object is characterized by a mesh feature extracted from an image, a directional distribution of optical flow, and the like. , A method of acquiring a time-series model of behavior as a probabilistic state transition model corresponding to each recognition category by training with learning data, and the probabilities that those models generate recognition target behaviors. An apparatus for recognizing a behavior of a moving object, comprising: