JP2012088881A - Person motion detection device and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a person motion detection device that detects a motion of a person from video photographed by a camera.SOLUTION: A person motion detection device 1 includes: feature point track information generation means 10 of generating a track of feature points as feature point track information for each frame image of video; feature quantity extraction means 20 of generating a track feature quantity by accumulating a direction and a magnitude of a movement vector at a feature point for each range width obtained by dividing a possible range thereof by a predetermined number; learning data storage means 40 of clustering a plurality of track feature quantities into a predetermined number of clusters, and finding and storing, as learning data, a distribution obtained by accumulating a plurality of track feature quantities constituting a known motion by the clusters; and operation identification means 30 of generating a distribution obtained by accumulating clusters that the track feature quantities belong to from a plurality of track feature quantities within a predetermined time section, and comparing the distribution with the learning data to identify the motion of the person.

Description

  The present invention relates to a human motion detection device that detects a human motion from video captured by a camera and a program thereof.

In recent years, researches for automatically recognizing human movements have been actively conducted. For example, a method has been proposed in which a contact-type measuring instrument (sensor) is attached to the body, and a person's movement is recognized from speed and acceleration information measured by the measuring instrument (see Patent Document 1).
However, such a method of attaching a measuring instrument to the body and recognizing the movement is not preferable in consideration of the installation cost and the influence (load) on the human body. In recent years, therefore, many studies have been conducted on recognizing a person's movement by analyzing a video image of the person. For example, a method for recognizing a person's movement from the locus of the person in the video has been proposed (see Patent Documents 2 and 3).

In addition, when obtaining the trajectory of a person from an image, three-dimensional (horizontal and vertical) obtained by tracking the horizontal and vertical coordinates of the feature point for each frame in the time direction for the feature points for each frame in the image. A method for recognizing a person's movement using a time) feature has also been proposed (see Non-Patent Document 1).
The method described in Non-Patent Document 1 limits the tracking time to a predetermined time, and sets the three-dimensional feature as a feature amount (trajectory feature amount) of a fixed-dimension (fixed-length) trajectory. By using the “Bag-of-words (BOW)” method in which the quantity is regarded as one word for classification, the movement of the person is classified into movements obtained in advance by learning.

In addition, as another method for detecting the motion from the trajectory feature amount using the “Bag-of-words” method, motion detection is performed using the angle of the movement vector when the feature point moves for each frame. A technique has also been proposed (see Non-Patent Document 2).
The method described in Non-Patent Document 2 uses a bin width θ (bin number 2π / θ) with a predetermined motion vector angle, and [0, θ), [θ, 2θ), ..., [2π-θ. , 2π) (note that [a, b) indicates a range from a to less than b), thereby generating a histogram as fixed dimension trajectory features, and using the “Bag-of-words” method. Human motion detection is possible.

JP-A-10-113343 JP 200387777 A Japanese Patent Laid-Open No. 2002-8042

Matikainen, P., Hebert, M. and Sukthankar, R. 2009. Trajectons: Action recognition through the motion analysis of tracked features.Workshop on Video-Oriented Object and Event Classification (ICCV). (Sep. 2009). V Mezaris, A Dimou, I Kompatsiaris, "Local invariant feature tracks for high-level video feature extraction", Proc. 11th International Workshop on Image Analysis for Multimedia Interactive Services, (WIAMIS 2010), April 2010.

  However, in the methods described in Patent Documents 2 and 3, it is necessary to accurately cut out a person region in a video for each frame. Therefore, in the methods described in Patent Documents 2 and 3, in order to make it easy to cut out a person region, the background is limited to a flat (predetermined color, etc.), or the person to be extracted is limited to one person. Conditions will be required. That is, the methods described in Patent Documents 2 and 3 have a problem that the motion of a person cannot be detected with high accuracy in a complicated video in which an unspecified number of persons appear.

In the methods described in Non-Patent Documents 1 and 2, the feature points in the video are tracked in the time direction, and a plurality of operations can be clustered by using the “Bag-of-words” method. Even if there are a plurality of persons in the video, the motion can be detected to some extent robustly.
However, the methods described in Non-Patent Documents 1 and 2 include the following problems.
In the method described in Non-Patent Document 2, although the speed at which the feature point moves (the length of the movement vector) is an important factor in addition to the angle of the movement vector, the speed is determined as an action determination element. Is not taken into account. For this reason, the method described in Non-Patent Document 2 includes a problem that, even though the motion speed is unnatural, the motion is erroneously detected when the angle of the movement vector approximates the result learned in advance. It is out.

  On the other hand, the method described in Non-Patent Document 1 uses a feature value obtained by tracking a feature point in the time direction as a trajectory feature value, and thus seems to consider a feature value based on the speed in the time direction. It is done. However, since the method described in Non-Patent Document 1 uses the “Bag-of-words” method with a fixed dimension (fixed length) trajectory feature, the tracking time must be limited to a predetermined time, The trajectory feature value is cut off during the operation. For this reason, the technique described in Non-Patent Document 1 has a problem in that it cannot accurately detect an operation depending on the time length of the operation.

  The present invention has been made in view of the above problems. Even when angle and speed information is included, a fixed dimension (fixed length) trajectory feature amount is used regardless of the operation time. It is an object of the present invention to provide a human motion detection device and a program thereof capable of accurately detecting a human motion.

  The present invention has been made to solve the above-described problems. First, the human motion detection device according to claim 1 is a human motion detection device that detects the motion of the person from an image of a person photographed. The feature point trajectory information generation unit, the time feature amount generation unit, the learning data storage unit, and the action identification unit are provided.

  In this configuration, the human motion detection device detects feature points that are features in the image for each frame image by the feature point locus information generation unit, and performs feature value matching for each frame image. Thus, a trajectory obtained by tracking the position of the feature point in the time direction is generated as the feature point trajectory information. For this feature point, a general feature point detection method such as Harris operator, SIFT, SURF, or the like can be used. In this way, by tracking the feature points, the motion in the video is extracted as a set of trajectories of the feature points.

In addition, the human motion detection device may include a movement vector for each feature point frame image based on the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation unit by the temporal feature amount generation unit. A trajectory that is a feature amount of a trajectory of a feature point is generated by accumulating the direction and size of each of the range and the range that can be taken by the size for each range width divided into a predetermined number. The feature value.
The direction of the movement vector represents the direction in which the feature point moves, and the magnitude of the movement vector represents the speed at which the feature point moves, and is a feature amount that characterizes the movement of the person. In addition, the time feature value generation unit accumulates the possible range of the direction and size of the movement vector for each range width divided into a predetermined number, thereby depending on the length of the trajectory, that is, the operation time length. Instead, a fixed-length feature value is extracted.

  In addition, the human motion detection device clusters a plurality of trajectory feature quantities into a predetermined number of clusters, and calculates a distribution obtained by accumulating a plurality of trajectory feature quantities constituting a known motion for each known motion. Are stored in advance as learning data in the learning data storage means. With this learning data, a plurality of trajectory feature amounts that constitute a person's motion is modeled by a predetermined number of clusters.

Then, the human motion detection device generates a distribution by accumulating the clusters to which the trajectory feature amount belongs from a plurality of trajectory feature amounts in which the end point of the trajectory exists in the time interval for each predetermined time interval by the motion identification unit. Then, the person's action is identified based on whether or not it is similar to the cluster distribution for each action stored in the learning data storage means. The reason why the trajectory ends within the predetermined time interval is used as a reference because it can be considered that one operation is completed at that stage.
In this way, a plurality of trajectory feature quantities that constitute a person's motion is specified by the distribution of the cluster, and the motion identifying means detects the motion of the person by comparing the distribution with the distribution of the learning data. Can do.

  According to a second aspect of the present invention, there is provided the human motion detection device according to the first aspect, further comprising a spatial feature generating unit.

In such a configuration, the human motion detection device generates, as a spatial feature amount, a brightness gradient of the frame image at the position of the feature point included in the feature point locus information generated by the feature point locus information generation unit by the spatial feature amount generation unit. And added to the trajectory feature value. The spatial feature amount may be a luminance gradient of the feature point of the frame image at the start point, end point, or intermediate point of the trajectory, or a luminance gradient obtained by averaging the luminance gradient of the frame image for each trajectory of the feature point. .
As described above, the human motion detection device generates a trajectory feature amount by adding a feature amount in the spatial direction to the feature amount in the time direction. As a result, the motion identifying means identifies the motion by taking into account not only the feature of the person's movement but also the feature of the appearance.

  Further, the human motion detection device according to claim 3 is the human motion detection device according to claim 1 or 2, wherein the time feature quantity generation means includes a direction feature quantity generation means, a speed feature quantity generation means, It was set as the structure provided with.

  In such a configuration, the human motion detection device accumulates the direction of the movement vector for each different range width obtained by dividing the range that the direction of the movement vector can take by a plurality of predetermined numbers by the direction feature amount generation unit. Thus, a directional feature amount that is a feature amount constituting the temporal feature amount is generated.

In addition, the human motion detection device accumulates the size of the movement vector for each different range width obtained by dividing the range of the size of the movement vector by a plurality of predetermined numbers by the speed feature amount generation unit. Thus, a speed feature quantity that is a feature quantity constituting the temporal feature quantity is generated.
As a result, the direction feature quantity includes a plurality of features from a distribution in which the direction of movement of the person is roughly classified to a distribution in which the direction is finely classified. Also, the speed feature amount includes a plurality of features from a distribution in which the speed of movement of a person is roughly classified to a distribution in which the person is finely classified.

  According to a fourth aspect of the present invention, there is provided the human motion detection apparatus according to the third aspect, wherein the time feature quantity generating means further comprises a smoothing means.

In this configuration, the human motion detection device generates a plurality of trajectories obtained by smoothing the trajectories of the feature points in the feature point trajectory information by the smoothing unit. In addition, the direction feature quantity generation unit and the speed feature quantity generation unit generate a direction feature quantity and a speed feature quantity for each of the trajectories smoothed by the smoothing unit.
As a result, the direction feature quantity and the speed feature quantity include a plurality of features from a strictly reproduced trajectory to a roughly reproduced trajectory.

  The human motion detection device according to claim 5 is the human motion detection device according to any one of claims 1 to 4, wherein the motion identification means includes a weighted distribution generation means, a classification means, It was set as the structure provided with.

In such a configuration, the human motion detection device regards each trajectory feature amount having a trajectory end point within a predetermined time interval as a word by the weighted distribution generation unit, and documents a plurality of words existing within the time length. Therefore, the importance of the trajectory feature amount generated by the feature amount extraction unit is calculated by the tf-idf method, and the cluster distribution is generated by weighting the frequency of the cluster to which the trajectory feature amount belongs. .
Further, the human motion detection device is based on the distance between the cluster distribution generated by the weighted distribution generation unit by the classification unit and the cluster distribution for each operation stored as learning data in the learning data storage unit. Similarity is determined, and a person's action is classified. For this distance, for example, the Euclidean distance is used.

  Thus, since the trajectory feature amount is a fixed-length feature amount, the human motion detection device calculates the importance of the trajectory feature amount using the tf-idf method used in the “Bag-of-words” method. can do. As a result, the importance of the trajectory feature amount on the background area that frequently occurs in the video can be reduced, and the importance of the trajectory feature amount of the person occurring at a specific time can be increased.

  Furthermore, the person motion detection program according to claim 6 is a computer program comprising: a feature point trajectory information generating means, a time feature amount generating means, and an action identifying means for detecting a motion of the person from an image of a person photographed. It was set as the structure made to function as.

  In such a configuration, the human motion detection program detects the feature points for each frame image of the video by the feature point trajectory information generation unit, and performs the feature point matching for each frame image to thereby determine the position of the feature point. Is generated as feature point trajectory information. In addition, the human motion detection program includes a movement vector for each feature point frame image based on the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation unit by the temporal feature amount generation unit. A trajectory that is a feature amount of a trajectory of a feature point is generated by accumulating the direction and size of each of the range and the range that can be taken by the size for each range width divided into a predetermined number. The feature value.

  Then, the human motion detection program generates a distribution by accumulating the clusters to which the trajectory feature amount belongs from a plurality of trajectory feature amounts in which the end point of the trajectory exists within a predetermined time interval by the motion identifying unit, and learning data storage unit A person's action is identified based on whether or not it is similar to the cluster distribution for each action stored in. In the learning data storage means, a plurality of trajectory feature quantities are clustered into a predetermined number of clusters, and a distribution obtained by accumulating a plurality of trajectory feature quantities constituting a known action for each cluster is used as a known action. The data is stored in advance as learning data in association with each other.

The present invention has the following excellent effects.
According to the first and sixth aspects of the present invention, since the feature amount related to the motion of the person can be expressed from the video by the fixed-length trajectory feature amount regardless of the motion time, the feature amount is cut off during the motion. Therefore, it is possible to accurately extract the feature amount of the operation without any problem. Thus, according to the present invention, it is possible to detect a person's movement with high accuracy from an image by using a highly accurate feature amount.

  Further, according to the first and sixth aspects of the present invention, since the trajectory feature amount can be expressed by a fixed length, the motion detection of the person by the “Bag-of-words” method in which the trajectory feature amount is regarded as a word is performed. It becomes possible. As a result, the present invention performs clustering for each motion in which a trajectory end point exists within a predetermined time interval, so that even when there are a plurality of persons in the video, the motion is completed at the timing. The movement of a person can be detected individually.

  According to the second aspect of the present invention, it is possible to consider the spatial feature amount in addition to the time feature amount when detecting the motion of the person. As a result, the present invention can add not only movement but also appearance characteristics as determination factors to human motion detection. For example, whether a motion is caused by noise or caused by motion of a person's hand. It is possible to distinguish and determine whether it is a thing.

  According to the third aspect of the present invention, it is not the case where the operation of the learning data is accurately reproduced by classifying a plurality of directional feature quantities and velocity feature quantities from coarsely classified distributions to finely classified distributions. Even if it is a rough movement, the movement can be discriminated.

  According to the fourth aspect of the present invention, by smoothing the trajectory, even if the movement of the person is different due to individual differences for each person, the difference is absorbed and the same movement is obtained. It can be determined as a movement, and the movement of a person can be detected robustly.

  According to the fifth aspect of the present invention, it is possible to use the tf-idf method by setting the trajectory feature amount to a fixed length, increasing the importance of the trajectory of the person's motion in the video, and the background region. The importance of the trajectory can be reduced. As a result, the present invention can robustly detect the movement of a person.

It is a block block diagram which shows the whole structure of the person motion detection apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the feature point locus | trajectory information which the feature point locus | trajectory information generation means of the person motion detection apparatus which concerns on embodiment of this invention produces | generates. It is a schematic diagram for demonstrating the smoothing of the locus | trajectory of the feature point which the smoothing means of the human motion detection apparatus which concerns on embodiment of this invention performs, Comprising: (a) is a figure which applied the Haar filter in two steps, (b) ) Is a diagram showing how the trajectory of the feature point is smoothed. It is a figure which shows the direction feature-value (direction feature-value histogram) which the direction feature-value production | generation means of the person motion detection apparatus which concerns on embodiment of this invention produces | generates. It is a figure which shows the speed feature-value (speed feature-value histogram) which the speed feature-value production | generation means of the human motion detection apparatus which concerns on embodiment of this invention produces | generates. It is explanatory drawing for demonstrating the production | generation method of the code book in the code book production | generation means of the human motion detection apparatus which concerns on embodiment of this invention. It is explanatory drawing for producing | generating the production | generation method of the histogram in the histogram production | generation means of the human motion detection apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the learning phase (codebook production | generation) of the person motion detection apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the learning phase (histogram generation) of the human motion detection apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the motion detection phase of the human motion detection apparatus which concerns on embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
[Configuration of human motion detection device]
Initially, with reference to FIG. 1, the structure of the human motion detection apparatus which concerns on embodiment of this invention is demonstrated. The person motion detection device 1 detects a motion of a person shown in the video from a video shot by a camera (not shown). Here, the human motion detection device 1 includes a feature point trajectory information generation unit 10, a feature amount extraction unit 20, a motion identification unit 30, and a learning data storage unit 40.

The feature point trajectory information generation unit 10 detects a point (feature point) that is a feature of the frame image for each frame (frame image) of the input video, and tracks the feature point in the time direction to thereby detect the feature point. The feature point trajectory information is generated by connecting the position information (coordinates) in the time direction.
Here, the feature point trajectory information generation unit 10 includes a foreground region extraction unit 11, a feature point detection unit 12, and a feature point tracking unit 13.

  The foreground area extraction unit 11 extracts a moving area as a foreground area for each frame of an input video. The foreground area extraction unit 11 outputs information (for example, a binary image) obtained by dividing the extracted foreground area and the background area other than the extracted foreground area to the feature point detection unit 12.

  Note that the foreground area extracting unit 11 can extract the foreground area by a general background difference process. For example, if the video is a video shot with a fixed camera, an image in which no person is shown is shot in advance as a background image, and there is a difference by taking the difference from the input frame (frame image). Extract the region as a foreground region.

In addition, the foreground area extraction unit 11 calculates an average or variance of pixel values (or luminance values) with a predetermined number of frames for each pixel of the frame image, for example, and the variation in pixel values is larger than a predetermined threshold value. The pixel may be a pixel in the foreground area.
As described above, the foreground area extraction unit 11 can extract an area where a person has moved mainly by extracting an area having movement as a foreground area.

  The feature point detection means 12 detects a point (feature point) that is a feature of the frame image for each frame of the input video. For example, the feature point detection unit 12 detects a feature point based on a change in pixel value or luminance value with respect to an adjacent pixel. The feature point detection unit 12 outputs the position (coordinates) of the feature point detected for each frame image to the feature point tracking unit 13. Here, it is assumed that the feature point detection unit 12 does not output to the feature point tracking unit 13 when the detected feature point is not included in the foreground region extracted by the foreground region extraction unit 11. Accordingly, in the feature point tracking calculation processing in the feature point tracking unit 13, it is possible to prevent the feature point tracking for the background feature points that are not related to the movement of the person.

A general method can be used as the feature point detection method in the feature point detection means 12. For example, the feature point detection means 12 detects a feature point by performing a corner detection process represented by a Harris operator on the input frame image.
The Harris operator is a method for detecting feature points based on the correlation of image signals, and is an operator having a feature that a correlation output value becomes large at feature points such as edges and corners in an image.

The Harris operator first smoothes the input image (frame image) by the Gaussian operator. Then, the Harris operator in the square window W of a predetermined size on the image, for each coordinate (x, y), the gradient I _u (x, y), I _v ( The matrix A shown in the following equation (1) is calculated using x, y). Here, the gradients I _u (x, y) and I _v (x, y) are a partial differential value related to x and a partial differential value related to y of the luminance value I (x, y), respectively.

Then, as shown in the following equation (2), the Harris operator obtains the minimum values of the eigenvalues λ ₁ and λ ₂ of the matrix A calculated by the equation (1) as the feature amount H _xy .

  In addition, since accurate calculation of the eigenvalue requires a large amount of calculation, the calculation is performed using the determinant (detA) and trace (trA) of the matrix A as shown in the following expression (3) instead of the expression (2). It is good to do. Note that κ is a predetermined constant, for example, a constant in a range of “0.04” to “0.15” recommended by Harris et al. In a reference paper (reference paper: Harris, C., Stephens). , M .: A Combined Corner and Edge Detector. Proceedings of the 4th Alvey Vision Conference. Manchester, UK (1988) 147-151.

The feature amount H _xy calculated in this way indicates features such as edges and corners as the value increases. Therefore, the feature point detection unit 12 determines the pixel at the coordinates (x, y) as the feature point when the feature amount H _xy is larger than a predetermined threshold value.
As described above, the feature point detection unit 12 detects the feature point for each frame image, and outputs only the feature points in the foreground region extracted by the foreground region extraction unit 11 to the feature point tracking unit 13.
The feature point detection means 12 may use a general feature amount detection method such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), etc., in addition to the Harris operator.

  The feature point tracking unit 13 tracks the feature points detected by the feature point detection unit 12 for each frame. The feature point tracking means 13 tracks feature points in the time direction by matching feature points with similar feature quantities for each frame.

  That is, for each frame image, the feature point tracking unit 13 determines that the feature amount of a feature point in a certain frame image matches the feature amount of the feature point in the previous frame image as the same feature point. If the feature amount is not matched, tracking of the feature point is ended. As a result, while the feature points match in the time direction, the feature points are tracked.

A general method can be used as the feature point tracking method in the feature point tracking means 13. For example, the Lucas-Kanade method can be used.
The Lucas-Kanade method is one of spatial local optimization methods that assume that the optical flows are the same in the local region of the same object. An optical flow is a velocity vector that represents how much a feature point moves in which direction between successive images.

  Here, the luminance value of coordinates (x, y) in a certain square window W at time t of the frame image is I (x, y, t), and coordinates (x, y) in the square window W at time (t + δt). The optical flow (u, v) is expressed by the following equation (4) where I (x, y, t + δt) is the luminance value of.

As described above, the feature point tracking unit 13 performs feature point matching between the frame images, and tracks feature points having similar optical flows (u, v) as traces of the same feature points. Note that whether or not the optical flows are similar can be determined by the distance between the optical flows (for example, the Euclidean distance).
Here, the feature point tracking unit 13 generates feature point trajectory information by connecting the coordinate positions of the feature points in the frame image in association with time information (for example, frame number) for each trajectory of the feature points. To do. The feature point trajectory information generated by the feature point tracking unit 13 is output to the feature amount extracting unit 20.

The feature point trajectory information generating unit 10 is, for example, as shown in FIG. 2, the time _{t i, ..., t j,} ..., each frame image of the input video in _{t k (a), (b} ), ( in c), the operation (here there is a person, in the case of performing the operation) to bring the cellular phone to the ear, the time _{t i, ..., t j,} ..., at t _k, sequentially plurality detect feature points in the frame image To do. Then, at time t _k the locus of the feature points is completed, as shown in (d), each frame image (a), (b), by connecting the detected feature point (c), wherein Generate a point trajectory.

In FIG. 2, p _i represents the position of the feature point at the time t _i , p _j represents the position of the feature point at the time t _j , and p _k represents the position of the feature point at the time t _k. Yes. In FIG. 2, the number of feature points is reduced in order to easily explain the trajectory of feature points.
Thus, the feature point trajectory information generating unit 10, tracking trajectory _{p i, ..., p j,} ..., and generates a feature point trajectory information by connecting the coordinate position of p _k.
Returning to FIG. 1, the description of the configuration of the human motion detection device 1 will be continued.

  The feature amount extraction unit 20 generates a feature amount (trajectory feature amount) for each feature point trajectory based on the feature point trajectory information generated by the feature point trajectory information generation unit 10. Note that the feature quantity extraction unit 20 performs, for each feature point trajectory, a multidimensional feature quantity in the time direction (temporal feature quantity) and a multidimensional feature quantity in the spatial direction in the frame image (spatial feature quantity). Is generated as a trajectory feature amount of a fixed length (fixed dimension). Here, the feature quantity extraction unit 20 includes a temporal feature quantity generation unit 21 and a spatial feature quantity generation unit 22.

Based on the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation unit 10, the time feature amount generation unit 21 generates a trajectory of the feature point (a feature point movement vector for each frame image). From this, a multi-dimensional feature quantity (temporal feature quantity) in the time direction is generated. That is, the time feature quantity generation means 21 determines the movement direction (direction of movement vector [angle]) and the movement speed (size of movement vector [length]) of a feature point that is a feature in the time direction of the human motion. Based on this, a feature quantity in the time direction is generated. This time feature amount is a trajectory feature amount indicating a feature in the time direction of the trajectory of the feature point.
Here, the time feature quantity generation means 21 includes a smoothing means 211, a direction feature quantity generation means 212, and a speed feature quantity generation means 213.

  The smoothing unit 211 performs a plurality of levels (smoothing levels) of smoothing processing on the trajectory of the feature points. The smoothing unit 211 generates a plurality of trajectories by smoothing a complex trajectory of feature points at a plurality of level smoothing levels. This smoothing process can be realized by a general low-pass filter typified by a Haar filter.

  In this way, by representing the trajectory of the feature point with a plurality of smoothing levels, the trajectory of the person's motion is expressed as a trajectory that approximates the general motion of the person regardless of the personality of the person. However, the unsmoothed trajectory accurately represents the trajectory of the person's movement. Therefore, the smoothing unit 211 generates a trajectory smoothed at a plurality of levels including a trajectory that has not been smoothed, and outputs the trajectory to the direction feature value generating unit 212 and the velocity feature value generating unit 213. .

Here, the process in which the smoothing unit 211 smoothes the trajectory at a plurality of smoothing levels by the Haar filter will be specifically described with reference to the mathematical formula and FIG.
The Haar filter is a filter represented by a transfer function of the following equation (5) in discrete time (z space).

  Here, the x-coordinate and y-coordinate of the feature point k when the trajectory of the feature point k is smoothed to q steps (q: an integer of 0 or more) with the Haar filter shown in the equation (5) are as follows: 6) Formula.

Further, assuming that the trajectory of the feature point k exists in the frame numbers t ₁ to t ₂ , the x coordinate p ^x _{k, q} of the feature point k can be expressed by the following equation (7), and the equation (5) The Haar filter shown in (5) can be expressed by the following equation (8). Since the y coordinate p ^y _{k, q} is the same as the x coordinate, the mathematical formula is omitted.

  Here, with reference to FIG. 3, how the trajectory of the feature point is smoothed by the Haar filter will be schematically described. Here, as shown in FIG. 3A, an example is shown in which the Haar filter of the equation (5) is applied in two stages. That is, the smoothing unit 211 applies a Haar filter to the trajectory of the feature point k at the smoothing level 0 (Level 0: q = 0) to generate a trajectory at the smoothing level 1 (Level 1: q = 1). Further, a smoothing level 2 (Level 2: q = 2) trajectory is generated by applying the Haar filter to the smoothing level 1 trajectory.

Thereby, as shown in FIG. 3 (b), the smoothing means 211 has the trajectory of the level _{0 Pk, 0} feature point of the feature point _k (solid line in the figure) and the level 1 feature of _{Pk, 1} . A trajectory of points (broken line in the figure) and a trajectory of _{Pk, 2} feature points of Level 2 (dashed line in the figure) are respectively generated, and the coordinate positions of the trajectories are feature point trajectories with different smoothing levels. As information, it outputs to the direction feature-value production | generation means 212 and the speed feature-value production | generation means 213.
Returning to FIG. 1, the description of the configuration of the human motion detection device 1 will be continued.

The direction feature quantity generation unit 212 has a fixed dimension (fixed length) feature in the direction in which the feature point moves based on the position of the feature point included in the feature point trajectory information smoothed in multiple stages by the smoothing unit 211. A quantity (direction feature quantity) is generated. Note that this direction feature amount is a feature amount constituting a time feature amount.
This directional feature quantity generation means 212 accumulates the angles (directions of the movement vectors) at which the feature points move on the frame image with respect to each smoothing level trajectory generated by the smoothing means 211 for each fixed angular width ( By generating a histogram, a direction feature amount is generated.
That is, the direction feature quantity generation unit 212 has a feature point for each [0, θ), [θ, 2θ),..., [2π−θ, 2π), where θ is the bin width (angle width) of the histogram. Accumulate the angle that moves. Here, [a, b) represents not less than a and less than b.
At this time, the direction feature quantity generation unit 212 generates a plurality of histograms having different bin widths (angle widths) of the histograms.

  Specifically, the direction feature value generation unit 212 sets the angles from “0” to “2π” to bin widths that are divided into four, eight, and sixteen, and generates a histogram of the locus of each smoothing level. . For example, when generating a histogram of bin width “π / 2” obtained by dividing the angle “0” to “2π” into four, [0, π / 2), [π / 2, π), [π, 3π / 2) The angle is accumulated every [3π / 2, 2π).

  For example, FIG. 4 shows an example in which the angle at which the feature point moves with three different bin widths is histogrammed with respect to the trajectory of the feature point smoothed at the three smoothing levels described in FIG. As shown in FIG. 4, the direction feature value generation unit 212 has bin widths “π / 2” (bin number “4”), “π / 4” (bin number “8”), “π / 8” (bin The number of smoothing levels (here, “3”) are generated for each of the histograms of the number “16”), so that a fixed feature of 84 (number of bins (4 + 8 + 16) × number of smoothing levels (3)) dimensions is obtained. A quantity (direction feature quantity: direction feature quantity histogram) is generated.

  The speed feature quantity generation means 213 has a fixed dimension (fixed length) feature for the speed at which the feature point moves based on the position of the feature point included in the feature point trajectory information smoothed in multiple stages by the smoothing means 211. A quantity (speed feature quantity) is generated. Since the trajectory of this feature point is tracked for each frame image, the speed of the feature point may be the length of the movement vector of the feature point on the frame image. Here, velocity feature quantities are generated from the horizontal length and the vertical length of the movement vector, respectively. Note that this speed feature amount is a feature amount constituting a time feature amount.

  The speed feature quantity generation means 213 uses the speed at which the feature point moves on the frame image (the size of the movement vector [the length in the horizontal direction, the length in the vertical direction) for each smoothing level trajectory generated by the smoothing means 211. The speed feature amount is generated by accumulating (histogram-izing) the length of each]] for each constant speed width.

Note that the speed feature quantity generation unit 213 generates a plurality of histograms having different bin widths, similarly to the direction feature quantity generation unit 212.
Specifically, for example, when generating the feature value for the horizontal speed, the speed feature value generation unit 213 has the slowest horizontal speed based on the feature point trajectory information, that is, the length of the movement vector in the horizontal direction. The shortest speed (length) is defined as the minimum value v _{s of the} histogram. Further, the speed (length) having the fastest horizontal speed, that is, the longest horizontal length of the movement vector is defined as the maximum value v _{f of the} histogram.

Then, the speed feature quantity generation unit 213 sets the speeds of v _{s to} v _f to bin widths that are divided into 4, 8, and 16, and generates a histogram of the trajectory of each smoothing level. For example, when generating a histogram of bin width “{v _f −v _s } / 4” obtained by dividing the speed of v _{s to} v _f into four, [v _s , v _s + {v _f −v _s } / 4) , [V _s + {v _f −v _s } / 4, v _s + {v _f −v _s } / 2), [v _s + {v _f −v _s } / 2, v _s + 3 × {v _f −v _s } / 4) and [v _s + 3 × {v _f −v _s } / 4, v _f ] are accumulated. Here, [a, b) represents a range from a to b, and [a, b] represents a range from a to b.
Also, the speed feature quantity generation unit 213 generates a histogram for the speed in the vertical direction as in the horizontal direction.

For example, FIG. 5 shows an example in which the speed at which the feature point moves with three different bin widths is histogrammed with respect to the trajectory of the feature point smoothed at the three smoothing levels described in FIG.
As shown in FIG. 5, the speed feature quantity generation unit 213 generates 84-dimensional fixed feature quantities as the speed feature quantity in the horizontal direction and the vertical direction, in the same manner as the direction feature quantity generation unit 212. That is, the speed feature quantity generation unit 213 generates a 168-dimensional (84 × 2) fixed dimension feature quantity (speed feature quantity: speed feature quantity histogram) as the speed feature quantity in the horizontal direction and the vertical direction.
As described above, the speed feature quantity generation unit 213 can generate a speed feature quantity of a fixed dimension (fixed length) without depending on the time length of the trajectory of the feature point.

  The spatial feature quantity generation unit 22 uses a feature point trajectory based on the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation unit 10 to generate a multidimensional feature quantity (space) in the spatial direction. Feature amount). That is, the spatial feature value generation means 22 generates feature values of feature points on the frame image as appearance (appearance) features. This spatial feature amount is a trajectory feature amount indicating a feature in the spatial direction of the trajectory of the feature point.

  The spatial feature value generation means 22 generates feature values of feature points on the frame image, and can generate fixed-length feature values by general feature value expression. For example, a SURF (Speeded Up Robust Features) feature amount, a SIFT (Scale-Invariant Feature Transform) feature amount, or the like can be used as the feature amount.

When the SURF feature value is used as this feature value, the spatial feature value generating unit 22 obtains the most dominant luminance inclination direction (luminance gradient: dominant rotation) by the Haar weblet at the feature point. Then, the spatial feature value generation means 22 is based on the most dominant direction as a reference, and the sum (horizontal Σdx, vertical Σdy) and magnitude of the direction of the luminance gradient in 16 predetermined blocks near the feature point, respectively. The four values of the sum total (horizontal Σ | dx |, vertical Σ | dy |) are calculated as feature amounts.
That is, the spatial feature value generation unit 22 calculates a 64-dimensional (16 × 4) feature value as the SURF feature value for each feature point.
If a SIFT feature value is used as the feature value, the spatial feature value generating unit 22 calculates a 128-dimensional feature value for each feature point.

  Here, the spatial feature value generating means 22 extracts the SURF feature value (or SIFT feature value) from the corresponding frame image at all feature points on the trajectory, and averages each trajectory, thereby obtaining the feature point. Generate spatial features in. Since the SURF feature value (SIFT feature value) is an appearance feature, it is not always necessary to calculate the feature value for all the trajectories corresponding to the time direction of the feature points. For example, the spatial feature value generation unit 22 may generate a feature value as a representative of the start point, end point, or intermediate point of the trajectory of the trajectory of the feature point.

  The spatial feature generating unit 22 adds a spatial feature to the temporal feature (direction feature and velocity feature) generated by the temporal feature generating unit 21, thereby generating a trajectory feature and identifying an action. The data is output to the means 30.

As described above, the feature quantity extracting unit 20 is configured to generate the fixed dimension temporal feature quantity (direction feature quantity (84 dimensions in this embodiment)) and speed feature quantity (in this embodiment, 168 dimensions) generated by the temporal feature quantity generation means 21. )) And a fixed dimension spatial feature quantity generated by the spatial feature quantity generation means 22 (in this embodiment, 64 dimensions [in the case of a SURF feature quantity]), even if the motion time of the person is variable, it is fixed. A dimension (fixed length) trajectory feature amount is generated (extracted) for each trajectory of feature points.
Here, the feature quantity extraction means 20 calculates the fixed dimension of the trajectory feature quantity (time feature quantity and spatial feature quantity) for each trajectory of the feature point, that is, the end time of the trajectory, that is, the time when the action of the person is completed ( For example, the final frame number of the trajectory) is output to the action identifying means 30.

  The motion identification unit 30 refers to learning data stored in the learning data storage unit 40 to be described later, and is multidimensional (fixed dimension) in which the end point of the locus exists within the predetermined time interval extracted by the feature amount extraction unit 20. ) To identify the action of the person. This motion identification means 30 is a general method such as a method based on an If-Then rule for sequentially determining whether or not to approximate a trajectory feature amount obtained in advance for each motion, and a method based on a machine learning support vector machine (SVM). Can be used. Here, the motion identification unit 30 regards the multidimensional trajectory feature quantity as one word (hereinafter also referred to as a trajectory word), and identifies the motion using the “Bag-of-words” method.

  The motion identification unit 30 includes a learning unit 31 and a motion determination unit 32. In addition, the action identifying unit 30 is configured to set an operation mode via an input unit (not shown), so that a “learning phase” in which learning data is learned and a “motion detection phase” in which a person's action is detected from a video. The learning unit 31 operates in the “learning phase”, and the operation determination unit 32 operates in the “operation detection phase”.

  The learning unit 31 learns the distribution of the trajectory feature amount for each operation from the trajectory feature amount in the video when the person moves in advance extracted by the feature amount extraction unit 20. Here, the learning unit 31 includes a code book generating unit 311 and a histogram generating unit 312.

The code book generating unit 311 inputs the trajectory feature amount (trajectory word) extracted by the feature amount extracting unit 20 from the video obtained by photographing various actions, and sets a predetermined number (k) of trajectory words. The code book is generated by clustering into clusters.
This code book is a word dictionary in which a plurality of trajectory words are classified into k pieces (for example, 1000 pieces) determined in advance based on their features (multidimensional feature amounts).
The clustering in the code book generating unit 311 can be performed using, for example, a K-means method (K-means method).
The code book generating unit 311 writes and stores a code book composed of a plurality of trajectory words classified into k clusters in the learning data storage unit 40.

The video used by the code book generating unit 311 to generate the code book is not particularly limited. For example, when the human motion detection device 1 detects a human motion with a fixed camera, it is determined in advance. This is a video taken for several days with a camera installed at a certain position.
Further, here, the code book generating means 311 applies the sequence 1 for a plurality of trajectory words whose trajectories have ended in a sequence of a predetermined time length (predetermined time interval) (for example, 1 second [corresponding to 25 frames]). As a document (document), a trajectory word and its cluster included in the document are written and stored in the learning data storage means 40 for each document. This document is used when the importance of the locus word is calculated in the weighted histogram generation means 321 of the action determination means 32 described later.

The histogram generation means 312 receives a plurality of trajectory feature quantities (trajectory words) extracted by the feature quantity extraction means 20 from a video obtained by photographing a predetermined action, and a distribution of appearance frequencies of trajectory words in the action (histogram). ).
This histogram generation means 312 is a cluster in which the distance (Euclidean distance) is the closest among the k clusters of the codebook generated by the codebook generation means 311 in a known operation in advance. And a histogram composed of k bins is generated.

Note that the histogram generation unit 312 normalizes the histogram. That is, the histogram generation unit 312 normalizes the frequencies of the respective clusters so that the total value of the frequencies accumulated for each cluster becomes “1.0”. This makes it possible to express one motion on the same basis regardless of the number of trajectories, and to easily and robustly detect the motion.
As described above, the histogram generation unit 312 writes and stores the histogram created in the known operation in the learning data storage unit 40 in association with the operation.

Here, with reference to FIG. 6 and FIG. 7 (refer to FIG. 1 as appropriate), a learning method performed by the learning unit 31 in the “learning phase” will be schematically described. Note that the trajectory word is actually a multidimensional feature quantity, but is schematically shown as a trajectory shape in FIGS. 6 and 7.
First, as shown in FIG. 6A, the learning unit 31 inputs a plurality of multidimensional trajectory feature amounts (trajectory words W ₁ , W ₂ ,..., W _n ) extracted by the feature amount extraction unit 20. Then, it is written in the learning data storage means 40. Then, the learning unit 31, based on the codebook generator 311, as shown in FIG. 6 (b), a plurality of loci words _{W 1,} W 2, ..., the _{W n,} for example, the feature amount by the K-means method Into k clusters (C ₁ , C ₂ ,..., C _k ). In this way, the code book generating unit 311 generates a code book CB that is a dictionary of trajectory words classified into k clusters.

Then, as shown in FIG. 7A, the learning unit 31 includes a plurality of multidimensional trajectory feature amounts (trajectory words w ₁ , w ₂ ,..., W extracted by the feature amount extracting unit 20 by known operations. _n ). Then, the learning unit 31, the histogram generation unit 312, the locus words _{w 1,} w 2, ..., each _{w n} is the codebook CB throat clusters shown in FIG. _{6 (b) (C 1,} C 2, .., C _k ), and the number (frequency) belonging to each cluster is obtained, and a histogram H is generated as shown in FIG. In this histogram H, the frequencies of the respective clusters are normalized so that the total value of the frequencies becomes “1.0”.
As described above, the learning unit 31 generates learning data by generating the histogram H for each known operation by the histogram generation unit 312.
Returning to FIG. 1, the description of the configuration of the human motion detection device 1 will be continued.

The motion determination unit 32 refers to the learning data stored in the learning data storage unit 40 and uses the multidimensional (fixed dimension) trajectory feature amount (trajectory word) extracted by the feature amount extraction unit 20 to determine the person's character. The operation is determined.
The motion determination unit 32 stores, in the learning data storage unit 40, a plurality of trajectory words whose trajectory end points exist in a sequence of a predetermined time length (predetermined time interval) (for example, 1 second [corresponding to 25 frames]). The operation is determined with reference to the learned data. In this way, a plurality of trajectory words whose trajectories have ended within a predetermined time interval indicate characteristics of a series of motions in which the motion has been completed. The time length of this sequence can be arbitrarily determined.
Here, the operation determination unit 32 includes a weighted histogram generation unit 321 and a classification unit 322.

  A weighted histogram generation unit (weighted distribution generation unit) 321 inputs a trajectory feature amount (trajectory word) in one sequence extracted by the feature amount extraction unit 20, and a distribution of the appearance frequency of the trajectory word in the sequence ( Histogram). The weighted histogram generation means 321 weights the appearance frequency of the histogram based on the importance of the trajectory word.

That is, the weighted histogram generation unit 321 determines the distance (Euclidean distance) of each of a plurality of trajectory words in one sequence among the k clusters of the code book stored in the learning data storage unit 40. Classify into close clusters and generate a histogram consisting of k bins.
Further, the weighted histogram generation means 321 regards the trajectory word in one sequence as one document (document), calculates the importance of the trajectory word in all documents using the tf-idf method, and calculates the trajectory word. By multiplying the frequency of appearance of the cluster to which the number belongs by the importance, a weight is added to the histogram (cluster distribution). Here, the whole document refers to a document collected from a plurality of videos obtained by photographing various operations by the learning unit 31 in advance in the learning phase.

The weighted histogram generation unit 321 normalizes the frequency of each cluster so that the total value of the frequency accumulated for each cluster becomes “1.0”. Thereby, the comparison with the learning data memorize | stored in the learning data memory | storage means 40 can be performed on the same reference | standard.
The distribution (histogram) of the appearance frequency of the trajectory word generated in this way is output to the classification means 322.

Here, a method in which the weighted histogram generation unit 321 calculates importance by the tf-idf method will be specifically described using mathematical expressions.
Here, the weighted histogram generation means 321 calculates importance for each cluster to which the trajectory word belongs for a plurality of trajectory words in which the end point of the trajectory exists in a sequence having a predetermined time length (for example, 1 second). To do.
That is, the weighted histogram generation unit 321 calculates the importance w _xd in the document d of the cluster x to which the locus word belongs by the product of the tf _xd value and the idf _x value shown in the following equation (9).

The idf _x value of the equation (9) is a logarithm of the reciprocal of the frequency of documents including the cluster x in all documents, and is expressed by the following equation (10).

Here, N, the total number of documents, n _x, in all documents, a number of documents including cluster x. Thus, the idf _x value is inversely proportional to the frequency of documents containing cluster x.
Further, the tf _xd value in the equation (9) is the frequency of the cluster x in a certain document d and is expressed by the following equation (11).

Here, OC _xd is the number of clusters x in a document d, and W is a set of trajectory words in the document d. OC _id is the number of trajectory words i (clusters) in the set of trajectory words.
In this way, the weighted histogram generation means 321 calculates the importance of the cluster to which the locus word belongs by the tf-idf method and generates a histogram. Therefore, the importance of the locus word on the background region that frequently occurs is calculated. The importance of trajectory words that frequently occur in a specific sequence can be increased. The feature point trajectory information generation unit 10 extracts the trajectory from the foreground feature points. However, the feature points may be traced in the background due to lighting or noise. In this case, the weighted histogram generation means 321 can generate a histogram that more appropriately represents the action of the person by reducing the importance of the trajectory word on the background region.
Returning to FIG. 1, the description of the configuration of the human motion detection device 1 will be continued.

  The classifying unit 322 includes a distribution (histogram) of appearance frequencies of trajectory words in a sequence generated by the weighted histogram generation unit 321 and a distribution (histogram) for each operation of learning data stored in the learning data storage unit 40. Similarity is determined on the basis of the distance between and the movement of the person in the sequence is classified into a predetermined movement.

In other words, the classification unit 322 determines that the distance between the histogram (cluster distribution) in the input sequence and the histogram (cluster distribution) of the learning data, for example, the closest Euclidean distance, is a similar operation. The operation corresponding to the histogram of the similar learning data is classified as the human operation in the sequence.
This classification result is output as a human motion detection result in the human motion detection device 1.

  The learning data storage means 40 stores learning data that associates the distribution of appearance frequencies (histograms) of trajectory feature quantities clustered into a predetermined number of clusters and a person's action by prior learning. is there. The learning data storage means 40 can be composed of a general storage medium such as a hard disk or a semiconductor memory.

The learning data storage means 40 includes a codebook obtained by clustering a plurality of trajectory words in which a multidimensional trajectory feature amount is regarded as one word (trajectory word) into a predetermined number of clusters, and a trajectory generated in a certain operation. A histogram in which the distribution of each word cluster is associated with the operation is stored as learning data.
Further, the learning data storage means 40 includes, for a plurality of trajectory words having a trajectory end point in a sequence having a predetermined time length (predetermined time interval), the sequence as one document (document). The trajectory word and its cluster are stored for each document.

By configuring the human motion detection device 1 in this manner, the human motion detection device 1 treats the variable length feature quantity in the time direction as a fixed length (fixed dimension) trajectory feature quantity, thereby enabling “Bag-of. -Words "technique can be used to detect human motion.
The person motion detection device 1 can be operated by a program (person motion detection program) that causes a general computer to function as each of the above-described means.

[Operation of human motion detection device]
Next, the operation of the human motion detection device according to the embodiment of the present invention will be described with reference to FIGS. Here, the operation of the human motion detection device 1 will be described separately for the “learning phase” and the “motion detection phase”.

[Learning phase (first stage)]
First, the operation in the learning phase (first stage) of the human motion detection device 1 will be described with reference to FIG. In the learning phase (first stage) in FIG. 8, trajectory feature amounts (trajectory words) are extracted from a plurality of videos obtained by photographing various actions, and a predetermined number (k) of trajectory words are extracted. This is an operation of generating a code book used for classifying the trajectory by clustering into clusters.

First, in the human motion detection device 1, the feature point trajectory information generation unit 10 generates feature point trajectory information indicating the trajectory of the feature point from the input video.
That is, the human motion detection apparatus 1 extracts a moving area as a foreground area by background difference processing for each frame image of the input video by the foreground area extraction unit 11 (step S1).

In the human motion detection device 1, the feature point detection means 12 detects a point (feature point) that is a feature of the frame image by a feature point detection method such as a Harris operator for each frame of the input video ( Step S2). At this time, the feature point detection unit 12 discards the feature points other than the region determined as the foreground region in step S1.
Then, the human motion detection device 1 tracks the feature points having similar feature quantities (for example, luminance gradient) in the feature points detected in step S2 by the feature point tracking unit 13, for each frame (time direction). Feature point trajectory information is generated (step S3).

Then, the human motion detection device 1 uses the time feature quantity generation means 21 of the feature quantity extraction means 20 based on the feature point trajectory information generated in step S3, so that the time direction multidimensional feature quantity (time feature quantity) is obtained. Is generated.
That is, the human motion detection apparatus 1 performs multi-level smoothing processing on the trajectory (coordinates) of the feature point described in the feature point trajectory information generated in step S3 by the smoothing unit 211 ( Step S4). At this time, for example, the smoothing unit 211 applies the Haar filter in two stages, and generates feature point trajectory information having a smoothing level of three stages.

Thereafter, the human motion detection device 1 uses the direction feature quantity generation unit 212 to move the feature point on the frame image based on the feature point trajectory information smoothed in multiple stages in step S4 (the angle of the movement vector). , 0 to 2π) are accumulated for every fixed angle width (histogram), to generate a directional feature amount (step S5).
At this time, the direction feature value generation unit 212 generates a histogram by accumulating the movement vectors of the respective angles with different angular widths (for example, π / 2, π / 4, π / 8) as bin widths.

Furthermore, the human motion detection device 1 determines the moving speed of the feature points on the frame image for each constant speed width based on the feature point trajectory information smoothed in multiple stages in step S4 by the speed feature quantity generation unit 213. The speed feature quantity is generated by accumulating (histogram) (step S6).
At this time, the speed feature quantity generation unit 213 uses the horizontal length and the vertical length of the movement vector of the feature point for each frame on the frame image as the speed of the feature point. Further, the speed feature quantity generation unit 213 generates a histogram by accumulating the movement vectors of the respective speeds using different speed widths as bin widths.

Furthermore, the human motion detection device 1 uses the spatial feature quantity generation means 22 to perform multidimensional features in the spatial direction with respect to the trajectory (coordinates) of the feature points described in the feature point trajectory information generated in step S3. A quantity (spatial feature quantity; for example, SURF feature quantity, SIFT feature quantity, etc.) is generated (step S7).
At this time, the spatial feature quantity generation means 22 extracts the feature quantity (SURF feature quantity or SIFT feature quantity) from the corresponding frame image at all feature points on the trace, and averages it for each trace.
Note that the generation of the feature amounts in steps S5 to S7 is not necessarily performed in this order, and the feature amounts may be generated by parallel processing.

  In this way, the feature quantity extraction means 20 is a trajectory feature comprising a temporal feature quantity (direction feature quantity, velocity feature quantity) having a fixed length in the time direction and a spatial feature quantity having a fixed length in the spatial direction for each feature point. Generate quantity. Accordingly, the human motion detection device 1 can express a trajectory with a fixed-length multidimensional trajectory feature amount even if the trajectory length is variable in the time direction. Thus, the human motion detection device 1 can handle the word as one multidimensional trajectory feature quantity (trajectory word).

Then, the person motion detection device 1 learns the distribution of trajectory feature values for each motion from the trajectory feature values in the video when the person moves in advance by the learning unit 31.
That is, the human motion detection device 1 uses a trajectory feature amount (trajectory word) extracted from a plurality of videos in which various motions are captured by the feature amount extraction unit 20 by the code book generation unit 311 to use a plurality of trajectory words. Is clustered into a predetermined number (k) of clusters to generate a codebook to be a word dictionary (step S8). Then, the code book generation unit 311 writes and stores the generated code book in the learning data storage unit 40 (step S9). Note that the codebook generation means 311 uses the tf-idf method in the motion detection phase described later, and the trajectory word and its cluster for each document when the sequence of a predetermined time length of the input video is one document. The learning data storage means 40 is written and stored.

  Through the above operations, the human motion detection apparatus 1 can collect various trajectories as trajectory words having a fixed-length multidimensional feature and generate a word dictionary (codebook) clustered into k pieces.

[Learning phase (second stage)]
Next, the operation in the learning phase (second stage) of the human motion detection device 1 will be described with reference to FIG. 9 (refer to FIG. 1 as appropriate for the configuration). In the learning phase (second stage) in FIG. 9, a trajectory feature amount (trajectory word) is extracted from a video obtained by photographing a predetermined action, and the code book generated in the learning phase (first stage) is referred to. This is an operation of generating a feature quantity of the operation as a histogram by forming a histogram in cluster units.
The operations from step S11 to S17 are the same as the operations from step S1 to S7 described with reference to FIG.

After step S <b> 17, the human motion detection apparatus 1 uses the histogram generation unit 312 of the learning unit 31 to extract a plurality of trajectory feature amounts (trajectory words) extracted from a video obtained by capturing a predetermined motion in the feature amount extraction unit 20. Using this, the distribution (histogram) of the appearance frequency of the trajectory word in the operation is generated (step S18). The histogram generation unit 312 then writes and stores the generated histogram in the learning data storage unit 40 in association with each operation (step S19). Note that the histogram generation unit 312 generates a histogram for each operation, and normalizes the histogram so that the total value of frequencies becomes “1.0” in advance.
With the above operation, the human motion detection device 1 can generate a distribution (histogram) of appearance frequency of trajectory words in a certain motion as a feature amount for each motion.

[Motion detection phase]
Next, referring to FIG. 10 (refer to FIG. 1 as appropriate for the configuration), the operation in the motion detection phase of the human motion detection device 1 will be described.
The operations from step S21 to S27 are the same as the operations from step S1 to S7 described with reference to FIG.

  After step S27, the human motion detection apparatus 1 uses the weighted histogram generation unit 321 of the motion determination unit 32 to obtain a plurality of trajectory words having trajectory end points in a sequence having a predetermined time length as the learning data storage unit 40. Are classified into clusters having the shortest distance (Euclidean distance) among the k clusters of the codebook stored in, and a histogram composed of k bins is generated (step S28).

  At this time, the weighted histogram generation means 321 regards the trajectory word in one sequence as one document (document), and the trajectory words in all documents (here, all the documents stored in the learning data storage means 40). Is calculated using the tf-idf method, and a weight is added to the histogram by multiplying the appearance frequency of the cluster to which the locus word belongs by the importance. As a result, the weighted histogram generation means 321 can generate a histogram that more appropriately represents the motion of the person by reducing the importance of the trajectory word on the background region. Note that the weighted histogram generation means 321 normalizes the histogram so that the total value of the frequencies is “1.0” in advance.

Then, the human motion detection device 1 compares the histogram (weighted histogram) generated in step S28 by the classification unit 322 with the histogram for each operation of the learning data stored in the learning data storage unit 40, and The movement of the person in the sequence is classified into a predetermined movement (step S29).
The movement classified in this way is output to the outside as a human movement detection result of the human movement detection apparatus 1.

As described above, the human motion detection device 1 can represent a trajectory of a variable length person in the time direction as a trajectory feature amount of a fixed length (fixed dimension), and faithfully represent a series of motion trajectories as feature amounts. Therefore, it is possible to accurately detect the movement of the person from the video.
Furthermore, since the human motion detection apparatus 1 uses a fixed-length (fixed dimension) trajectory feature amount as the feature amount of the trajectory of the feature point, the trajectory feature amount is regarded as a word (trajectory word) and “Bag-of -Words "technique can be used to detect human motion. As a result, the human motion detection device 1 can reduce the importance of frequently occurring feature quantities on the background and perform human motion more robustly.

  As described above, since the human motion detection device 1 according to the present invention can detect a human motion robustly, it detects an abnormal behavior of a person by video monitoring, a specific motion detection, or a man-machine interface triggered by a gesture. It can be widely applied.

The configuration and operation of the human motion detection device 1 according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.
For example, here, the feature quantity extraction unit 20 generates both the temporal feature quantity and the spatial feature quantity as the trajectory feature quantity, but only the temporal feature quantity may be used. In this case, the spatial feature generation means 22 may be omitted from the configuration of FIG. At this time, the trajectory feature amount includes a directional feature amount and a speed feature amount that are temporal feature amounts.

  In addition, although the learning means 31 is provided here, it is not necessary to provide the learning means 31 in every person motion detection device 1. That is, in a certain person motion detection device 1, after learning and storing learning data in the learning data storage means 40, the motion detection phase can be executed as long as at least the learning data storage means 40 is provided. In this case, the learning means 31 may be omitted from the human motion detection device 1 that does not perform learning.

[Evaluation result of human motion detection device]
Finally, in the human motion detection device 1 according to the embodiment of the present invention, a human motion detection result when using a time-direction feature amount that could not be considered in the past will be described. Here, the measurement of the reproduction rate [Recall] (%) at which each motion can be detected from the image for the motion of a person (Pointing) and the motion of placing an object (ObjectPut) is performed. did.

In [Table 1], when the motion is detected only with the conventional SURF feature value (SURF), when the motion is detected by adding the angle feature value (direction feature value) to the SURF feature value (SURF + angle), In the present invention, the measurement results of the recall are shown for the case where the motion is detected by adding the speed, which is the characteristic amount in the time direction (SURF + angle + speed). As shown in [Table 1], the reproducibility could be improved by detecting the motion by adding the speed, which is the time-direction feature amount in the present invention.
As described above, the present invention can detect the motion of a person more robustly than the conventional motion detection method by treating the time-direction feature value, which is a variable-length feature value, as a fixed-length feature value. it can.

DESCRIPTION OF SYMBOLS 1 Human motion detection apparatus 10 Feature point locus information generation means 11 Foreground area extraction means 12 Feature point detection means 13 Feature point tracking means 20 Feature quantity extraction means 21 Time feature quantity generation means 211 Smoothing means 212 Directional feature quantity generation means 213 Speed Feature quantity generation means 22 Spatial feature quantity generation means 30 Action identification means 31 Learning means 311 Codebook generation means 312 Histogram generation means 32 Action determination means 321 Weighted histogram generation means (weighted distribution generation means)
322 Classification means 40 Learning data storage means

Claims (6)

A human motion detection device that detects a motion of the person from a video of the person,
Feature points are detected for each frame image of the video, and feature points are matched for each frame image, thereby generating a trajectory tracking the position of the feature points in the time direction as feature point trajectory information. Feature point trajectory information generating means,
Based on the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation unit, the direction and the size of the movement vector for each frame image of the feature point are determined as the direction and the size. A time feature amount generating means for generating a time feature amount by accumulating the range that can be taken for each range width divided into a predetermined number, and making a trajectory feature amount that is a feature amount of the trajectory of the feature point;
A plurality of trajectory feature quantities are clustered into a predetermined number of clusters, and a distribution obtained by accumulating a plurality of trajectory feature quantities constituting a known motion for each cluster is previously learned in association with each known motion. Learning data storage means for storing as data,
For each predetermined time interval, a distribution obtained by accumulating the clusters to which the trajectory feature amount belongs is generated from a plurality of trajectory feature amounts whose trajectory end points exist in the time interval, and stored in the learning data storage unit. Action identification means for identifying the action of the person depending on whether the distribution is similar to the cluster distribution for each action;
A human motion detection device comprising:   Spatial feature value generation means for generating a luminance gradient of a frame image at the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation means as a spatial feature quantity and adding the spatial gradient to the trajectory feature quantity; The human motion detection device according to claim 1, further comprising: The time feature amount generation means includes:
Direction features that are feature amounts constituting the temporal feature amount by accumulating the directions of the movement vectors for each different range width obtained by dividing the possible range of the direction of the movement vector by a plurality of predetermined numbers. Direction feature amount generating means for generating a quantity;
A feature amount constituting the temporal feature amount by accumulating the size of the movement vector for each different range width obtained by dividing the range that the movement vector can take by a plurality of predetermined numbers. A speed feature quantity generating means for generating a speed feature quantity;
The human motion detection device according to claim 1, further comprising: The time feature amount generating means further comprises a smoothing means for generating a plurality of trajectories obtained by smoothing the trajectories of the feature points in the feature point trajectory information,
The direction feature quantity generation unit and the speed feature quantity generation unit generate the direction feature quantity and the speed feature quantity for a plurality of trajectories smoothed by the smoothing unit, respectively. Item 4. The human motion detection device according to Item 3. The operation identification means includes
By treating each trajectory feature amount having a trajectory end point in the time interval as a word and considering a plurality of words existing in the time interval as a document, the feature amount extraction unit performs the tf-idf method. A weighted distribution generating means for calculating the importance of the generated trajectory feature quantity and generating a cluster distribution by weighting the frequency of the cluster to which the trajectory feature quantity belongs;
Similarity is determined based on the distance between the distribution of clusters generated by the weighted distribution generation unit and the distribution of clusters for each operation stored as learning data in the learning data storage unit, and the motion of the person is determined. A classification means for classifying;
The human motion detection device according to any one of claims 1 to 4, further comprising: In order to detect the movement of the person from the video of the person,
Feature points are detected for each frame image of the video, and feature points are matched for each frame image, thereby generating a trajectory tracking the position of the feature points in the time direction as feature point trajectory information. Feature point trajectory information generating means,
Based on the position of the feature point included in the feature point trajectory information generated by the feature point trajectory information generation unit, the direction and the size of the movement vector for each frame image of the feature point are determined as the direction and the size. A time feature amount generating means for generating a time feature amount by accumulating the range that can be taken for each range width divided into a predetermined number, and making a trajectory feature amount that is a feature amount of the trajectory of the feature point;
A plurality of trajectory feature quantities are clustered into a predetermined number of clusters, and a distribution obtained by accumulating a plurality of trajectory feature quantities constituting a known motion for each cluster is previously learned in association with each known motion. A distribution obtained by accumulating the clusters to which the trajectory feature amount belongs from a plurality of trajectory feature amounts having an end point of the trajectory within the predetermined time interval for each predetermined time interval with reference to the learning data storage means stored as data. Action identifying means for identifying the action of the person according to whether or not it is similar to the cluster distribution for each action stored in the learning data storage means,
It is made to function as a person motion detection program characterized by things.

Images