JP2011100175A - Device and program for deciding personal action - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a personal action decision device, capable of deciding the action of a person included in a crowded scene by means of video processing only. <P>SOLUTION: A personal action decision device 1 includes: a personal area detection means 10 for detecting one or more personal areas included in a video image at predetermined frame intervals by machine learning; a personal track generation means 20 for calculating a feature amount for each personal area, for deciding a personal area, having a resembling feature amount of the personal area in a plurality of frame images, to be the personal area of an identical person, and for generating a personal track by connecting the positions of the centers of gravity in the personal areas of the identical person; and a personal action decision means 30 for deciding that the person acts in conformity to an action condition, if the feature amount of the personal action satisfies the action condition by calculating the feature amount for each personal track and by deciding whether the feature amount of the personal track satisfies the action condition. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for determining an action of a person included in a video.

  Human motion recognition by video analysis is a useful technique that can be utilized for, for example, automatic recognition of human behavior, personal authentication, or video search using motion as a key. In particular, there is a great demand for a technology for automatically detecting a specific action of a person from a fixed surveillance camera image for the purpose of crime prevention. For example, a system has been developed that automatically analyzes video from a fixed surveillance camera installed in an elevator or bookstore and issues an alarm for abnormal behavior of a person.

Here, human motion recognition by video analysis can be broadly classified into technology based on models and technology based on features.
As a technique based on this model, for example, there is a technique in which each part such as a human torso and arm is expressed as a geometric model, and the posture of the person is described by parameters such as their joint angles (Patent Document 1). reference). The technology based on this model focuses on analyzing the detailed movements of parts such as human arms and legs.

On the other hand, as a technique based on features, for example, there is a technique for determining a person region from an image feature amount for processing a two-dimensional moving image, and performing gesture recognition, counting the number of people, and calculating a moving direction (Patent Document). 2).
In addition, as this technique, for example, there is a technique that obtains a movement vector from time-series data of a person position and recognizes the action of the person from the speed and the number of times of stationary (see Patent Document 3).

JP 2000-149025 A JP-A-5-46583 JP 2006-221379 A

  Here, human motion recognition by video analysis is based on the premise that a person region is cut out from video and the outline of the person is accurately acquired. At this time, the conventional technique is intended for a scene with a flat background and dotted with about one to several persons, and is not intended for a crowded scene in which it is difficult to cut out an area for each person. . For example, it is considered that the inventions described in Patent Documents 1 and 2 described above are intended for videos including only one person.

  The invention described in Patent Document 3 is intended for relatively crowded scenes, but uses RFID (Radio Frequency IDentification) to measure a person's position, and does not necessarily perform human action determination only by video processing. However, there is a problem that the scale of the apparatus is increased and the cost is increased.

  Therefore, an object of the present invention is to provide a human behavior determination device and a program thereof that can accurately determine a human behavior even in a crowded scene by only video processing.

  The human behavior determination device according to the first invention of the present application is a human behavior determination device that receives a video in which a plurality of frame images are continuous and determines the behavior of one or more persons included in the video, and includes a human region detection. Means, a person trajectory generation means, and a person action determination means.

  According to this configuration, the human behavior determination device detects one or more human regions included in the video by machine learning at predetermined frame intervals by the human region detection unit. In addition, the human behavior determination device calculates a feature amount for each person region detected by the person region detection unit by the person trajectory generation unit, and assigns a person region having a similar person region feature amount in a plurality of frame images to the same person. And the center of gravity of the person areas of the same person are connected to generate a person trajectory. That is, for example, the human behavior determination apparatus generates a human trajectory over a longer period (for example, 30 frames) than a motion vector detected from two consecutive frame images.

  Further, the human behavior determination device calculates a feature amount for each human trajectory generated by the human trajectory generation unit by the human behavior determination unit, and determines whether or not the human trajectory feature amount satisfies a preset behavior condition. When it is determined that the feature amount of the person trajectory satisfies the action condition, it is determined that the person is performing an action corresponding to the action condition. In other words, the human behavior determination device determines the human behavior based on the human trajectory obtained over a longer period than the conventional motion vector.

  Further, in the human behavior determination device according to the second invention of the present application, the human area detecting means may select any one of a distance between each other, a color histogram distribution distance, or a size difference for a plurality of human areas detected from the same frame image. Is calculated as a representative feature quantity, and it is determined that the representative feature quantity is similar when the representative feature quantity is equal to or less than a preset threshold, and the representative feature quantity is located at the center side in a similar person area. And a representative person area selecting unit that selects one representative person area as a person area.

Here, in the same frame image, since similar image features exist around the same person, a plurality of person regions are detected around a certain person. In addition, in the same frame image, a plurality of person regions are detected from the periphery of another person who is located at a distant position.
According to such a configuration, the person behavior determination device can detect one person area located on the center side among the person areas even when a plurality of person areas are detected from around the same person in the same frame image. select.

In addition, the human behavior determination device according to the third invention of the present application further includes a human prediction unit that predicts a prediction region of a person using a prediction filter from the human trajectory generated by the human trajectory generation unit, and the human trajectory generation unit includes the prediction region. A human trajectory is generated when the center of gravity of the human area is included in the human trajectory.
According to this configuration, the human behavior determination device can prevent a situation in which a human trajectory is generated from an erroneously detected person area even when the human area is erroneously detected by occlusion.

Further, in the human behavior determination device according to the fourth invention of the present application, the human behavior determination means uses a behavior condition in which a correspondence relationship between a feature amount of a human trajectory and a threshold value preset for each human behavior is set in advance. It is characterized by determining the behavior of
According to such a configuration, the human behavior determination device can determine the human behavior by simple threshold processing.

Further, in the human behavior determination device according to the fifth invention of the present application, when the human behavior determination means determines that the person is performing the preset verification target behavior, reverse playback is a command for playing back the video in the reverse direction. A reverse playback command is output from the human behavior determination unit, and a reverse video playback unit is further provided that plays back the video in the reverse direction and outputs the video to the human region detection unit based on the reverse playback command. Features.
According to such a configuration, the human behavior determination device verifies the determination result using the reversely reproduced video, so that the accuracy of the determination result can be further increased.

  In addition, the human behavior determination program according to the sixth aspect of the present invention is configured to input a video in which a plurality of frame images are continuous, and to determine a behavior of one or more persons included in the video, It is made to function as a detection means, a person locus generation means, and a person action determination means.

  According to this configuration, the human behavior determination program detects one or more human regions included in the video by machine learning at a predetermined frame interval by the human region detection unit. In addition, the human behavior determination program calculates a feature amount for each person area detected by the person region detection unit by the person trajectory generation unit, and assigns a person region having a similar person region feature amount in a plurality of frame images to the same person. And the center of gravity of the person areas of the same person are connected to generate a person trajectory. That is, for example, the human behavior determination program generates a human trajectory over a longer period than a motion vector detected from two consecutive frame images.

  In addition, the human behavior determination program calculates a feature amount for each human trajectory generated by the human trajectory generation unit by the human behavior determination unit, and determines whether or not the human trajectory feature amount satisfies a preset behavior condition. When it is determined that the feature amount of the person trajectory satisfies the action condition, it is determined that the person is performing an action corresponding to the action condition. In other words, the human behavior determination program determines the human behavior based on the human trajectory obtained over a longer period than the conventional motion vector.

According to the present invention, the following excellent effects can be obtained.
According to the first and sixth inventions of the present application, since a person's action can be determined based on a person trajectory obtained over a longer period than the conventional motion vector, the person's action can be accurately determined even in a crowded scene by only video processing. Can be judged. Therefore, according to the first and fifth inventions of the present application, it is possible to provide an accurate human behavior determination technique at low cost.

  According to the second aspect of the present invention, even when a plurality of person areas are detected from the periphery of the same person in the same frame image, one person area located on the center side can be selected from these person areas. Therefore, according to the second invention of the present application, it is possible to generate an accurate person trajectory and more accurately determine the action of the person.

  According to the third invention of the present application, even when a person area is erroneously detected, it is possible to prevent a situation in which a person locus is generated from the erroneously detected person area. Therefore, according to the third invention of the present application, it is possible to generate an accurate person trajectory and more accurately determine the action of the person.

According to the fourth aspect of the present invention, since the action of a person can be determined by simple threshold processing, the processing speed can be increased.
According to the fifth aspect of the present invention, since the determination result is verified using the reverse reproduction video, the accuracy of the determination result can be further increased.

It is a block diagram which shows the whole structure of the person action determination apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the person area | region detection means of FIG. It is a figure which shows the training data stored in the person image database of FIG. 2, (a) is an example of a positive example image, (b) is an example of a negative example image. It is a figure explaining the HOG feature-value in this invention, (a) is an example of an image, (b) is a figure explaining a cell and a block, (c) is a figure which shows a brightness | luminance histogram. It is a figure explaining the difference image in this invention, (a) is a figure which shows an image | video, (b) is a figure which shows the background difference image produced | generated from the image | video. It is a figure explaining clustering of the representative person area | region by the representative person area | region selection part of FIG. It is a block diagram which shows the structure of the person locus | trajectory production | generation means of FIG. It is a figure explaining the similarity of the color histogram in this invention, (a) shows the example in which a color histogram is not similar, (b) shows the example in which a color histogram is similar. It is a figure explaining the person locus | trajectory in this invention, (a) shows the gravity center position of a person area | region, (b) shows the person locus | trajectory produced | generated by connecting a gravity center position. FIG. 8 is a diagram for explaining prediction by the person prediction unit in FIG. 7, where (a) shows a case where no occlusion occurs and (b) shows a case where occlusion occurs. It is a block diagram which shows the structure of the person action determination means of FIG. It is a figure explaining the person locus | trajectory feature-value normalization part of FIG. 11, (a) shows the average motion vector for every block, (b) shows the normalization coefficient for every block. It is a flowchart which shows the whole operation | movement of the person action determination apparatus of FIG. It is a flowchart which shows operation | movement of the person area detection means of FIG. It is a flowchart which shows operation | movement of the person locus | trajectory production | generation means of FIG. It is a flowchart which shows operation | movement of the person action determination means of FIG. In 2nd Embodiment of this invention, it is a block diagram which shows the structure of a person action determination means. In 3rd Embodiment of this invention, it is a block diagram which shows the structure of a person action determination means. It is a figure explaining feature space in a 3rd embodiment of the present invention. It is a block diagram which shows the whole structure of the person action determination apparatus which concerns on 4th Embodiment of this invention.

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

[Overall configuration of person action determination device]
With reference to FIG. 1, the whole structure of the human action determination apparatus 1 which concerns on 1st Embodiment of this invention is demonstrated. As shown in FIG. 1, the human behavior determination device 1 determines the behavior of one or more persons included in a video, and includes a human region detection unit 10, a human trajectory generation unit 20, and a personal behavior determination unit 30. With.

  For example, the human behavior determination apparatus 1 receives an image captured by one imaging camera (not shown). Here, the photographing camera is fixed at a crowded place such as an event venue, an elevator, a bookstore, a station, or an airport. That is, the video shot by the camera is often a crowded scene including two or more people.

  The person area detection means 10 calculates HOG (Histograms of Oriented Gradients) feature quantities in a change area where differences such as inter-frame differences and background differences occur, and the change area is determined by machine learning such as a support vector machine. It is determined whether the area. Further, when a plurality of person areas are detected for the same person, the person area detecting means 10 performs clustering and assigns one person area to one person.

  The person trajectory generation means 20 matches the person area detected by the person area detection means 10 in consideration of the distance and the color histogram. Then, the person trajectory generation unit 20 generates a person trajectory by tracking the person region over a predetermined number of frames. At this time, the person trajectory generation means 20 predicts the person position using a prediction filter in order to robustly track even a crowded scene. As a result, the person trajectory generation means 20 can resume tracking when the person area reappears even if a tracking error occurs due to occlusion. Occlusion refers to a state in which, for example, a person located on the back side is hidden by an obstacle such as another person located on the near side when viewed from the photographing camera.

  The human behavior determination unit 30 calculates the feature amount of the human trajectory such as the average speed, the tracking time, and the moving direction from the human trajectory generated by the human trajectory generation unit 20. Then, the person action determination unit 30 determines the action of the person such as “run”, “meet a person”, and “place an object” from the feature amount of the person trajectory. For example, when “running”, the amount of movement between frames is large, and when “meeting a person”, the acceleration increases on the opposite side (negative direction) of the moving direction.

[Configuration of Person Area Detection Means]
Hereinafter, the configuration of the person region detecting means 10 will be described with reference to FIG. As shown in FIG. 2, the person area detection means 10 includes a person image database 11, a training data feature amount calculation unit 12, a change region detection unit 13, a change region feature amount calculation unit 14, and a person region determination unit 15. And a representative person area selection unit 16.

The person image database 11 stores in advance, as training data, a positive example image obtained by cutting out a person in advance and a negative example image other than the person.
Here, the detection accuracy of the person region depends on the training data in the person image database 11. For this reason, it is preferable that the positive image stored in the person image database 11 is cut out from the video of the photographing camera that actually performs photographing. On the other hand, the person image database 11 can also make the person action determination device 1 versatile by storing, as training data, positive example images and negative example images cut out from various videos. .

  As shown in FIG. 3A, the positive image is an image obtained by cutting out persons of various races, postures, clothes, ages and genders. Further, as shown in FIG. 3B, negative examples include, for example, electronic devices such as facsimiles, personal computers, and mobile phone terminals, moving means such as automobiles and motorcycles, clothing such as shoes, and animals. It is the image which cut out things other than various persons, such as.

  The training data feature amount calculation unit 12 refers to the person image database 11, calculates the feature amounts of the training data stored in the person image database 11, and outputs them to the person region determination unit 15. Here, the training data feature amount calculation unit 12 calculates a HOG feature amount as the feature amount of the training data.

<HOG features>
Hereinafter, the HOG feature amount will be described with reference to FIG. 4 (see FIG. 2 as appropriate). This HOG feature amount is a histogram of the gradient direction of luminance in a local region (cell) of an image. In this example, as shown in FIG. 4B, one cell includes, for example, 5 pixels in the vertical direction and 5 pixels in the horizontal direction. In addition, one block includes, for example, 3 cells in the vertical direction and 3 cells in the horizontal direction.

  First, the training data feature amount calculation unit 12 obtains the luminance gradient strength and the luminance gradient direction from all the pixels included in the image of FIG. 4A, for example. In the cell of FIG. 4B, the brightness gradient strength and the brightness gradient strength at each pixel are indicated by arrows. That is, in the cell of FIG. 4B, the length of the arrow indicates the luminance gradient strength, and the direction of the arrow indicates the luminance gradient direction.

  Next, for example, as shown in FIG. 4C, the training data feature quantity calculation unit 12 divides the luminance gradient direction into 9 directions at intervals of 20 ° between 0 ° and 180 ° for each cell. Thus, a luminance histogram is generated. That is, in this histogram, the vertical axis represents the luminance gradient intensity, and the horizontal axis represents the luminance gradient direction.

  Furthermore, the training data feature amount calculation unit 12 normalizes, for example, a histogram generated for each cell in units of blocks. Here, for example, the training data feature amount calculation unit 12 performs normalization on all blocks while moving the block to be normalized one cell at a time. In other words, each cell is repeatedly normalized with different blocks.

  In addition, although the training data feature-value calculation part 12 demonstrated as using a HOG feature-value, it is not limited to this. For example, the training data feature quantity calculation unit 12 can use an arbitrary feature quantity such as a SIFT (Scale Invariant Feature Transform) feature quantity or an edge feature quantity.

Hereinafter, returning to FIG. 2, the description of the configuration of the person region detecting means 10 will be continued.
The change area detection unit 13 obtains a change area in which a video is input from the photographing camera and changes in the video. Specifically, the change area detection unit 13 generates a difference image such as an inter-frame difference image or a background difference image at intervals of three frames from the video, for example.

  For example, as shown in FIG. 5A, it is assumed that the input video includes moving persons H1 to H4, a rolling ball Ob1, and a background. In this case, the change area detection unit 13 generates a background difference image as shown in FIG. In this background difference image, a portion without change (motion) (background portion) is black, and a portion with change (motion) (persons H1 to H4 and ball Ob1) is white or gray. . And the change area | region detection part 13 detects the rectangular area | region of the persons H1-H4 and the rectangular area | region of ball | bowl Ob1 from a background difference image as a part which has a change. Further, as shown in FIG. 5A, the change area detection unit 13 obtains the same area as the part having the change from the input video, and outputs it to the change area feature quantity calculation unit 14 as the change areas R1 to R5. To do.

  Although the change area detection unit 13 has been described as generating difference images at intervals of three frames, the present invention is not limited to this. For example, the change area detection unit 13 can generate a difference image at intervals of one frame, two frames, or four or more frame intervals.

Hereinafter, returning to FIG. 2, the description of the configuration of the person region detecting means 10 will be continued.
The change area feature quantity calculation unit 14 receives the change area from the change area detection unit 13 and calculates the feature quantity of the change area. At this time, it is preferable that the change area feature quantity calculation unit 14 uses the same type of feature quantity (for example, HOG feature quantity) as the training data feature quantity calculation unit 12.

  As described above, since the change region feature value calculation unit 14 calculates the HOG feature value only from the change region, compared to the case where the HOG feature value is calculated from the entire frame image, the processing speed can be increased.

  In addition, although the change area | region feature-value calculation part 14 demonstrated as using a HOG feature-value, it is not limited to this. For example, the change area feature value calculation unit 14 can use an arbitrary feature value such as a SIFT feature value or an edge feature value.

  The person region determination unit 15 receives the HOG feature amount of the training data from the training data feature amount calculation unit 12 and performs machine learning using the HOG feature amount. Here, the person region determination unit 15 uses a support vector machine (SVM) as supervised machine learning.

  The support vector machine is machine learning capable of recognizing two classes (for example, a person class and a class other than a person). Specifically, the support vector machine uses a discriminant function that outputs 1 if the inner product of the training data and the synaptic load is equal to or greater than a threshold, and outputs -1 if it is less than the threshold. At this time, the support vector machine learns the learning parameters as having a margin from each training data, not just between different classes of training data, in order to minimize the recognition error of the training data. That is, the support vector machine obtains a margin between training data belonging to different classes, and learns learning parameters so that this margin is maximized. Note that the person region determination unit 15 preferably performs machine learning by the support vector machine before the change region feature amount calculation unit 14 receives the change region feature amount.

  Thereafter, the person region determination unit 15 receives the HOG feature amount of the change region from the change region feature amount calculation unit 14. Further, the person area determination unit 15 determines whether or not the HOG feature amount of the change area is a person class using the machine-learned learning parameter. Then, the person area determination unit 15 determines that the change area is a person area when the HOG feature amount of the change area is a person class. In this case, the person area determination unit 15 outputs the change area as a person area to the representative person area selection unit 16.

  On the other hand, when the HOG feature amount of the change area is a class other than a person, the person area determination unit 15 determines that the change area is not a person area and discards the change area. The person area is a rectangular area including a person in the video.

  In addition, although demonstrated that the person area | region determination part 15 performs machine learning by a support vector machine, it is not limited to this. Here, the person region determination unit 15 can use machine learning capable of recognizing two classes, for example, a decision tree or a neural network. Further, although the person region determination unit 15 has been described as performing supervised machine learning, unsupervised machine learning may be performed.

Hereinafter, returning to FIG. 2, the description of the configuration of the person region detecting means 10 will be continued.
Here, it is preferable that the person action determination device 1 detects only one person region for the same person. However, since similar image features exist around the same person in the same frame image, a plurality of person regions are detected around a certain person. In addition, a plurality of person regions are also detected for another person at a distant position in the same frame image. As described above, if a plurality of person regions are detected from the periphery of the same person in the same frame image, the person action determination device 1 causes inconvenience in determining the person's action.

For this reason, the representative person area selection unit 16 receives a person area from the person area determination unit 15 and selects one person area for the same person in one frame image.
Note that the process of aggregating person areas of the same person from a plurality of person areas and selecting one person area from the aggregated person areas may be called clustering.

  Hereinafter, with reference to FIG. 6, clustering by the representative person region selection unit 16 will be described in order from the first example to the third example. At this time, in each example, as shown in FIG. 6, four person areas R1 to R4 are detected for the person H1, three person areas R5 to R7 are detected for the person H2, and three person areas R3 to R7 are detected for the person H3. It is assumed that the person areas R8 to R10 are detected.

<First example of selection: color histogram>
In this first example, the representative person area selection unit 16 uses a color histogram as a representative feature quantity serving as a reference for selecting a representative person area.

  The representative person area selection unit 16 calculates a color histogram distribution for each of the person areas R1 to R10. Further, the representative person area selection unit 16 obtains the distance between the color histogram distributions of the person areas R1 to R10. At this time, if the same person, the color histogram distribution is considered to be similar. Therefore, the representative person area selection unit 16 collects the person areas of the same person whose color histogram distribution distance is equal to or less than a preset threshold. Furthermore, the representative person area selecting unit 16 selects a representative person area located on the center side as a person area from the person areas aggregated as the person area of the same person.

  In the example of FIG. 6, the representative person area selecting unit 16 includes person areas R1 to R4 corresponding to the person H1, person areas R5 to R7 corresponding to the person H2, and person areas R8 to R10 corresponding to the person H3. Summarize. Further, the representative person area selecting unit 16 selects the representative person area R3 located on the center side from the person areas R1 to R4 as the person area of the person H1. Further, the representative person area selecting unit 16 selects the representative person area R6 located on the center side from the person areas R5 to R7 as the person area of the person H2. Further, the representative person area selecting unit 16 selects the representative person area R9 located on the center side from the person areas R8 to R10 as the person area of the person H3. After that, the representative person area selection unit 16 outputs the selected person areas R3, R6, and R9 to the person trajectory generation unit 20 in FIG. In FIG. 6, the representative person regions R3, R6, and R9 are shown by bold lines.

<Second example: distance of person area>
In this second example, the representative person area selection unit 16 uses the distance of the person area as a representative feature quantity that serves as a reference for selecting the representative person area.

  For example, the representative person area selection unit 16 calculates the distance of the center of gravity position as the distance of the person area in the input person areas R1 to R10. At this time, it is considered that the person areas R1 to R10 of the same person are detected at positions close to each other. Therefore, the representative person area selection unit 16 aggregates the person areas R1 to R10 whose distance is equal to or less than a preset threshold as person areas of the same person. Then, the representative person area selecting unit 16 selects a representative person area located on the center side from the person areas aggregated as the same person as the person area, and outputs the selected person area.

  In the example of FIG. 6, the person areas R1 to R4 are located nearby, the person areas R5 to R7 are located nearby, and the person areas R8 to R10 are located nearby. Therefore, the representative person area selecting unit 16 aggregates the person areas R1 to R4 corresponding to the person H1, the person areas R5 to R7 corresponding to the person H2, and the person areas R8 to R10 corresponding to the person H3. After that, the representative person area selection unit 16 selects the person areas R3, R6, and R9 as in the first example, and outputs them to the person trajectory generation means 20 in FIG.

<Third example: size of person area>
In the third example, the representative person area selecting unit 16 uses the size of the person area as a representative feature amount serving as a reference for selecting the representative person area.

  The representative person area selection unit 16 calculates the size (for example, the height or width of the person area) of the input person areas R1 to R10. At this time, it is considered that the person areas R1 to R10 of the same person are detected with sizes close to each other. Therefore, the representative person area selection unit 16 aggregates the person areas R1 to R10 whose size difference is equal to or less than a preset threshold as person areas of the same person. Then, the representative person area selecting unit 16 selects a representative person area located on the center side from the person areas aggregated as the same person as the person area, and outputs the selected person area.

  In the example of FIG. 6, the person areas R1 to R4 are close in size, the person areas R5 to R7 are close in size, and the person areas R8 to R10 are close in size. Therefore, the representative person area selecting unit 16 aggregates the person areas R1 to R4 corresponding to the person H1, the person areas R5 to R7 corresponding to the person H2, and the person areas R8 to R10 corresponding to the person H3. After that, the representative person area selection unit 16 selects the person areas R3, R6, and R9 as in the first example, and outputs them to the person trajectory generation means 20 in FIG.

  As described above, the representative person area selection unit 16 can assign one person area to one person in the same frame image using any one of the first to third examples. For example, the operator sets in advance whether to use one of the first example to the third example. Note that, when only one person area is input, the representative person area selection unit 16 preferably outputs the person area to the person trajectory generation unit 20 without performing clustering.

[Configuration of person trajectory generation means]
Hereinafter, the configuration of the human trajectory generation means 20 will be described with reference to FIG.
As shown in FIG. 7, the person trajectory generation unit 20 includes a person region feature amount calculation unit 21, a feature amount database 22, a person region feature amount collation unit 23, a person trajectory generation unit 24, and a person prediction unit 25. Is provided.

  The person area feature amount calculation unit 21 receives a person area from the person area detection means 10 (see FIG. 2) and calculates the feature amount of the person area. Then, the person area feature value calculation unit 21 outputs the calculated feature value of the person area to the person area feature value matching unit 23 as the feature value of the person area in the processing target frame image (current frame image). At this time, the person area feature quantity calculation unit 21 registers the feature quantity of the person area in the feature quantity database 22 described later.

Here, the person region feature value calculation unit 21 may calculate any one of the HOG feature value, the color histogram, and the person position as the feature value of the person region. Here, when using the person position as the feature quantity of the person area, the person area feature quantity calculation unit 21 sets the center position of the person area as the person position, for example.
In addition, the person region feature amount calculation unit 21 determines whether the combination of the HOG feature amount and the color histogram, the combination of the HOG feature amount and the person position, or the combination of the color histogram and the person position is the feature of the person region. It may be calculated as a quantity.
Furthermore, the person area feature quantity calculation unit 21 may calculate a combination of the HOG feature quantity, the color histogram, and the person position as the feature quantity of the person area.
It should be noted that the operator previously sets which character action determination device 1 uses as the feature value of the person area.

  The feature amount database 22 stores the feature amount of the person area calculated by the person region feature amount calculation unit 21. In other words, the feature amount database 22 stores the feature amount of the person area detected in the past frame image.

  The person region feature amount matching unit 23 refers to the feature amount database 22 and reads the feature amount of the person region in the past frame image. In addition, the person area feature quantity matching unit 23 receives the feature quantity of the person area in the frame image to be processed from the person area feature quantity calculation unit 21. Further, the person region feature amount matching unit 23 receives the predicted position and prediction region of the person from the person prediction unit 25 described later.

  Then, the person region feature amount matching unit 23 compares the feature amount of the person region in the frame image to be processed with the feature amount of the person region in the past frame image, and whether the feature amounts of these person regions are similar. Determine whether or not. Hereinafter, the first to fifth examples will be sequentially described with respect to the matching of the human region feature amount by the human region feature amount matching unit 23.

<First example: color histogram>
Hereinafter, a first example of human area feature amount matching performed by the person region feature amount matching unit 23 will be described with reference to FIG. 8 (see FIG. 7 as appropriate). In this first example, the person region feature amount matching unit 23 uses a color histogram as the feature amount of the person region.

  In FIG. 8, the color histogram on the left side is a color histogram of the person region in the frame image to be processed. In FIG. 8, the color histogram on the right side is the color histogram of the person region in the past frame image. In FIG. 8, the red-blue region is illustrated as the color histogram, but in the present invention, other colors may be included in the color histogram.

  In the case of another person, as shown in FIG. 8A, the color histogram of the person area in the frame image to be processed is greatly different from the color histogram of the person area in the past frame image. On the other hand, in the case of the same person, as shown in FIG. 8B, the color histogram of the person area in the frame image to be processed is similar to the color histogram of the person area in the past frame image.

  Accordingly, the person area feature amount matching unit 23 calculates the distribution h of the color histogram of the person area in the frame image to be processed and the distribution q of the color histogram of the person area in the past frame image by the following equation (1). A distance BC (p, q) is obtained. The person region feature amount matching unit 23 determines that the color histograms are similar when the distance BC (p, q) calculated by the equation (1) is equal to or less than a preset threshold value. On the other hand, the person region feature amount matching unit 23 determines that the color histograms are not similar if the distance BC (p, q) exceeds a preset threshold value. Here, the person region feature amount matching unit 23 uses the Bhattacharya distance as the distance BC (p, q).

  The person area feature amount matching unit 23 has a person area ID list that stores person area IDs that uniquely identify the person area. Then, the person area feature amount matching unit 23 refers to the person area ID list, and when the color histograms are similar (in the case of the same person), the person area ID assigned to the person area in the past frame image is processed. To the person area detected in the frame image. That is, the person area of the same person has the same person area ID between the past frame image and the frame image to be processed.

  On the other hand, if the color histograms are not similar (in the case of another person), the person area feature amount matching unit 23 has detected a new person area ID that is not registered in the person area ID list in the processing target frame image. It is given to the person area. In this case, the person area feature amount matching unit 23 registers the newly assigned person area ID in the person area ID list. Thereafter, the person area feature amount matching unit 23 outputs the person area to which the person area ID is assigned to the person locus generating unit 24.

<Second example: person position>
Hereinafter, a second example of human region feature amount matching performed by the person region feature amount matching unit 23 will be described. In the second example, the person area feature amount matching unit 23 uses the person position as the feature amount of the person area.

  The person region feature amount matching unit 23 obtains the distance between the person position in the frame image to be processed and the person position predicted using the past frame image. Here, the person position in the frame image to be processed is the person position input from the person region feature amount calculation unit 21. The person position predicted using the past frame image is a predicted position input from the person prediction unit 25.

  Then, the person region feature amount matching unit 23 determines that the person positions are similar if the distance is within a preset threshold. On the other hand, the person area feature amount matching unit 23 determines that the person positions are not similar if the distance exceeds a preset threshold value. Thereafter, as in the first example described above, the person area feature amount matching unit 23 refers to the person area ID list, assigns a person area ID to the person area, and outputs the person area ID to the person locus generation unit 24.

  At this time, it is preferable that the person area feature amount matching unit 23 outputs the person area to the person locus generation unit 24 only when the person position in the processing target frame image is included in the prediction area. In other words, when the person position in the processing target frame image is not included in the prediction area, the person area feature amount matching unit 23 skips the matching process as a false detection of the person area. As a result, the human behavior determination apparatus 1 can prevent a situation in which a human trajectory is generated from an erroneously detected human region.

<Third example: HOG feature amount>
Hereinafter, a third example of human area feature amount matching performed by the person region feature amount matching unit 23 will be described. In the third example, the person region feature amount matching unit 23 uses a HOG feature amount as the feature amount of the person region.

  The person region feature amount matching unit 23 calculates the distance between the HOG feature amount in the processing target frame image and the HOG feature amount in the past frame image for each dimension of the HOG feature amount. Then, the person region feature amount matching unit 23 calculates the sum of distances for each dimension, and determines that the HOG feature amounts are similar if the calculated sum is equal to or less than a preset threshold value. On the other hand, the person area feature amount matching unit 23 determines that the HOG feature amounts are not similar if the sum exceeds a preset threshold value. Thereafter, as in the first example described above, the person area feature amount matching unit 23 refers to the person area ID list, assigns a person area ID to each person area, and outputs the person area ID to the person locus generation unit 24.

<Fourth example: Similarity>
Hereinafter, a fourth example of collation of human region feature amounts by the human region feature amount collation unit 23 will be described. In the fourth example, the person region feature amount matching unit 23 uses a similarity obtained by combining a color histogram, a person position, and a HOG feature amount as the feature amount of the person region.

  In this case, the person region feature quantity matching unit 23 multiplies the person position, the color histogram, and the HOG feature quantity by weights, and adds these to calculate the similarity. Then, the person region feature amount matching unit 23 determines that the similarity is similar if the calculated similarity is equal to or less than a preset threshold value. On the other hand, the person region feature amount matching unit 23 determines that they are not similar if the similarity exceeds a preset threshold. Thereafter, as in the first example described above, the person area feature amount matching unit 23 refers to the person area ID list, assigns a person area ID to each person area, and outputs the person area ID to the person locus generation unit 24.

  Specifically, the person region feature amount matching unit 23 calculates the similarity using the following equation (2). Since the person position, color histogram, and HOG feature amount can be calculated in the same manner as in the first to third examples described above, description thereof is omitted.

  In Equation (2), Sim is the similarity, α is the weight of the person position, and Loc is the person position. In Equation (2), β is the weight of the color histogram, and His is the color histogram. Furthermore, in Expression (2), γ is the weight of the HOG feature value, and Hog is the HOG feature value.

  In the formula (2), the three weights α, β, and γ are set in advance. Further, in the expression (2), the weight of the person position, the color histogram, and the HOG feature value that is not used for calculating the similarity is set to zero.

  Here, the person position, the color histogram, and the HOG feature amount are standardized because the units are different. Specifically, the person area feature amount matching unit 23 uses the following equation (3) so that the average is zero and the variance is one for each feature amount (person position, color histogram, and HOG feature amount). , Standardize. Accordingly, the person region feature amount matching unit 23 can eliminate the influence due to the difference in units. In Expression (3), X represents the original feature value, μ represents the average of X, σ represents the variance of X, and Z represents the standardized feature value.

<Fifth example: Combination of threshold determination>
In the following, a fifth example of person area feature amount matching by the person area feature amount matching unit 23 will be described. In this fifth example, the person region feature amount matching unit 23 calculates the color histogram of the first example, the person position of the second example, and the HOG feature amount of the third example. Then, the person region feature amount matching unit 23 determines that they are not similar if any one of the color histogram, the person position, and the HOG feature amount exceeds the respective threshold values. On the other hand, the person region feature amount matching unit 23 determines that the color histogram, the person position, and the HOG feature amount are similar if they are all equal to or less than the respective threshold values. Thereafter, as in the first example described above, the person area feature amount matching unit 23 refers to the person area ID list, assigns a person area ID to each person area, and outputs the person area ID to the person locus generation unit 24.

  As described above, the person area feature amount matching unit 23 can select the representative person area using any one of the first to fifth examples. In this case, for example, the operator sets in advance whether to use any of the first to fifth examples.

Returning to FIG. 7, the description of the configuration of the human trajectory generation unit 20 will be continued.
The person trajectory generation unit 24 receives the person area to which the person area ID is assigned from the person area feature amount matching unit 23. Then, the person trajectory generation unit 24 obtains the gravity center position of the person area between the past frame image and the frame image to be processed, and connects the gravity center positions of the person areas assigned with the same person area ID. Generate a human trajectory.

  Hereinafter, the details of the person trajectory will be described with reference to FIG. 9 (see FIG. 7 as appropriate). In FIG. 9, # 1 to # 9 indicate the frame image numbers in order from the oldest, # 1 corresponds to the oldest frame image, and # 9 corresponds to the frame image to be processed (current frame image). . Reference numerals R1 to R9 indicate person regions of the same person detected in the respective frame images. Furthermore, the symbols P1 to P9 indicate the gravity center positions of the person regions R1 to R9. In FIG. 9, only a part of the reference numerals are shown for the sake of simplicity. The broken line in FIG. 9B will be described later.

  As shown in FIG. 9A, the person trajectory generation unit 24 obtains respective gravity center positions P1 to P8 for the person regions R1 to R8 detected in the past frame images (# 1 to # 8). Further, the person trajectory generation unit 24 obtains the gravity center position P9 from the person region R9 detected in the processing target frame image (# 9). Then, the person trajectory generation unit 24 connects these barycentric positions P1 to P9 to generate a person trajectory as shown in FIG. 9B. Thereafter, the person trajectory generation unit 24 outputs the person trajectory to the person prediction unit 25 and the person action determination unit 30 (see FIG. 11).

  Here, in the human trajectory, each of the line segments connecting adjacent gravity center positions P1 to P2... P8 to P9 corresponds to the motion vector of the person. In the example of FIG. 9B, the human trajectory includes eight human trajectory motion vectors V1 to V8. Hereinafter, the motion vector included in the person trajectory is abbreviated as a person trajectory motion vector.

Although FIG. 9 illustrates a person trajectory for one person, when a plurality of persons are included in the video, the person trajectory generation unit 24 preferably generates a person trajectory for each person.
In addition, the person trajectory generation unit 24 ends the generation of the person trajectory for the person when the number of frames exceeds a preset number (for example, 30 frames) after the person area of the person is first detected. May be.
Further, when the person trajectory generation unit 24 first detects a person area of a certain person, the person trajectory generation unit 24 outputs the position of the center of gravity of the person area to the person prediction unit 25 as the current position of the person instead of the person trajectory.

Hereinafter, the details of the person prediction unit 25 will be described with reference to FIG. 10 (see FIG. 7 as appropriate). In FIG. 10, the predicted position of the person is indicated by a symbol Yb.
The person prediction unit 25 receives the person locus from the person locus generation unit 24 and predicts the person's prediction area Ya and the prediction position Yb from the person locus using a prediction filter. Here, the person prediction unit 25 uses a prediction filter such as a Kalman filter or a particle filter.

  The Kalman filter predicts the prediction region Ya and the prediction position Yb from a plurality of observation positions (centroid positions P1 to P9 in FIG. 9). Specifically, the Kalman filter calculates the Kalman gain from the error between the predicted position Yb and the observation position, and adjusts the correction amount according to the magnitude. In the Kalman filter, the error covariance changes in accordance with the detection state of the person area in each frame image in addition to the error amount.

  As shown in FIG. 10A, the Kalman filter reduces the prediction area Ya when the person area is successfully detected. On the other hand, as shown in FIG. 10B, the Kalman filter enlarges the prediction area Ya when the amount of error is large or when the detection of the person area is unstable. Then, the person prediction unit 25 outputs the prediction region Ya and the prediction position Yb predicted by the Kalman filter to the person region feature amount matching unit 23.

  As described above, the Kalman filter can update the predicted position and the predicted region according to the motion model specified in advance even when the detection of the human region fails due to occlusion. For this reason, the person action determination device 1 can re-detect the person area of the same person when the occlusion is exceeded, and the reliability is improved.

  Note that the person prediction unit 25 cannot predict the prediction region Ya and the prediction position Yb when the current position of the person is input from the person locus generation unit 24. Therefore, the person predicting unit 25 outputs the current position as the predicted position Yb to the person region feature amount matching unit 23.

[Configuration of person action determination means]
Hereinafter, the configuration of the person action determination unit 30 will be described with reference to FIG.
As shown in FIG. 11, the human behavior determination unit 30 includes a human trajectory feature amount calculation unit 31, a human trajectory feature amount normalization unit 32, and a behavior determination unit 33.

  The person trajectory feature quantity calculation unit 31 receives a person trajectory from the person trajectory generation means 20 and calculates a feature quantity for each person trajectory. Here, the human trajectory feature quantity calculation unit 31 uses a tracking time, an initial detection position, a current detection position, a movement direction, a movement distance, an average speed, an average acceleration, and a multidimensional feature quantity including linearity as a person. The trajectory feature amount is calculated.

Returning to FIG. 9, a specific example of the calculation of the feature amount of the human trajectory will be described (see FIG. 11 as appropriate). At this time, it is assumed that the person trajectory shown in FIG.

<First example: tracking time>
The person trajectory feature quantity calculation unit 31 can calculate the tracking time from the following equation (4).

  Tracking time = 1 / frame rate x frame interval x number of barycentric positions Equation (4)

  For example, consider a case where nine gravity center positions P1 to P9 are included in a person trajectory, the frame rate of the video is 30, and a person region is detected at intervals of three frames. In this case, the human trajectory feature quantity calculation unit 31 calculates a tracking time of 0.9 seconds as shown in Expression (5).

  0.9 = 1/30 × 3 × 9 Expression (5)

<Second example: first detection position>
The person trajectory feature quantity calculation unit 31 calculates the gravity center position P1 in the oldest frame image # 1 as the initial detection position.

<Third example: current detection position>
The person trajectory feature quantity calculation unit 31 calculates the gravity center position P9 in the processing target frame image # 9 as the current detection position.

<Fourth example: moving direction>
The human trajectory feature quantity calculation unit 31 calculates the moving direction by averaging the directions of all the human trajectory motion vectors V1 to V8.

<Fifth example: moving distance>
The person trajectory feature quantity calculation unit 31 calculates the movement distance by adding up the lengths of the lines connecting the first detection position (center of gravity position P1) to the current detection position (P9) in order.

<Sixth example: average speed>
The human trajectory feature quantity calculation unit 31 averages the magnitudes of all the human trajectory motion vectors V1 to V8, and calculates an average speed.

<Seventh example: average acceleration>
The human trajectory feature quantity calculation unit 31 obtains the difference between adjacent human trajectory motion vectors V1 to V8 for each of the frames # 1 to # 9. For example, the human trajectory feature quantity calculation unit 31 obtains the difference between the human trajectory motion vector V5 and the human trajectory motion vector V4 as the acceleration of frame # 4. Then, the human trajectory feature quantity calculation unit 31 averages the accelerations of all the frames # 1 to # 9.

<Eighth example: linearity>
The person trajectory feature quantity calculation unit 31 obtains an approximate straight line connecting the first detection position (center of gravity position P1) to the current detection position (P9). Then, the human trajectory feature quantity calculation unit 31 obtains a normal line from each detection position (centroid positions P2 to P8) other than the initial detection position and the current detection position to the approximate line. Furthermore, the person trajectory feature quantity calculation unit 31 calculates an average value of the lengths of all the normals, and sets this average value as linearity. In FIG. 9B, the approximate straight line and the normal line from the gravity center position P2 to the approximate straight line are shown by broken lines.

  Thereafter, the person trajectory feature quantity calculation unit 31 outputs the calculated feature quantity of the person trajectory to the person trajectory feature quantity normalization unit 32. The human trajectory feature quantity calculation unit 31 calculates all of the tracking time, the initial detection position, the current detection position, the moving direction, the moving distance, the average speed, the average acceleration, and the linearity as the human trajectory feature quantity. Instead, only what is used in the action determination unit 33 described later may be calculated.

  Hereinafter, with reference to FIG. 12, the normalization by the human trajectory feature amount normalization unit 32 will be described separately in a preparation stage and a normalization stage (see FIG. 11 as appropriate).

  Depending on the detection position in the two-dimensional image coordinates, the moving distance between frame images in units of pixels differs. For example, let us consider a case where a photographing camera is installed so as to look down at a person from above. In this case, even if the person is moving at a constant speed, in the video shot by the photographing camera, the person trajectory motion vector is large on the near side and the person trajectory motion vector is small on the far side. Accordingly, the human trajectory feature quantity normalization unit 32 corrects the human trajectory motion vector error (that is, the speed error) due to the difference in the detection position.

<Preparation stage>
First, a normalization video for a predetermined time (for example, about 2 hours) is captured in advance by the imaging camera, and this normalization video is input to the person trajectory feature amount normalization unit 32. Then, as shown in FIG. 12A, the person trajectory feature amount normalization unit 32 divides the normalization video into a predetermined number of blocks (for example, 3 vertical blocks and 4 horizontal blocks). In addition, the person trajectory feature amount normalization unit 32 detects a motion vector from this block for each block from the start to the end of the normalization video. Then, the human trajectory feature quantity normalization unit 32 averages all detected motion vectors for each block, and calculates an average motion vector.

  In FIG. 12A, the average motion vector is indicated by an arrow for each block. That is, the size of this arrow is the size of the average motion vector, and the direction of this arrow is the direction of the average motion vector. As shown in FIG. 12A, the average motion vector increases on the front side of the photographing camera (lower side in FIG. 12), and the average motion vector decreases on the far side of the photographing camera (upper side in FIG. 12).

  Then, the human trajectory feature quantity normalization unit 32 calculates a normalization coefficient for each block so that the average motion vectors are equal between these blocks. For example, as shown in FIG. 12B, since the average motion vector increases on the front side of the photographing camera, the human trajectory feature quantity normalization unit 32 is, for example, 0.6-0. A normalization factor of less than 1 such as 7 is calculated. On the other hand, since the average motion vector becomes smaller on the far side of the photographing camera, the human trajectory feature normalization unit 32 calculates a normalization coefficient exceeding 1 such as 1.2 to 1.5 in the upper block, for example. To do. Further, in the middle block, the human trajectory feature amount normalization unit 32 calculates a normalization coefficient near one.

  In order to perform normalization accurately, it is preferable that the normalization video and the video for detecting the person area are shot at the same place using the same shooting camera.

<Normalization stage>
The human trajectory feature quantity normalization unit 32 receives the human trajectory feature quantity from the human trajectory feature quantity calculation unit 31 and normalizes the human trajectory feature quantity. First, the human trajectory feature quantity normalization unit 32 obtains which block the motion vector is included from the position where the human trajectory motion vector is detected (for example, the start point or the end point of the human trajectory motion vector). Then, the human trajectory feature amount normalizing unit 32 multiplies the magnitude of the human trajectory motion vector by the normalization coefficient of the block corresponding to the human trajectory motion vector. Furthermore, the person trajectory feature amount normalization unit 32 outputs the normalized feature amount of the person trajectory to the behavior determination unit 33.

  This normalization is to correct the size of the human locus motion vector due to the difference in detection position, that is, the feature quantity related to the speed. For this reason, the human trajectory feature amount normalization unit 32 does not have to normalize the feature amounts that are not related to the speed, for example, the initial detection position, the current detection position, and the movement direction.

Returning to FIG. 11, the description of the configuration of the human behavior determination unit 30 will be continued.
The behavior determination unit 33 receives the human trajectory feature amount from the human trajectory feature amount normalization unit 32 and determines the human behavior based on the human trajectory feature amount. Note that although the behavior determination unit 33 is described as performing rule-based determination based on simple threshold processing, determination based on machine learning may be performed.

  In addition, the behavior determination unit 33 includes a determination unit for each type of behavior in order to individually determine the behavior of the person from the feature amount of the person trajectory. In the example of FIG. 11, the behavior determination unit 33 includes a determination unit 33 a that runs, a determination unit 33 b that meets a person, and a determination unit 33 c that walks in the opposite direction. Hereinafter, a first example to a third example of determination of person behavior will be described in order.

<First example: “Run”>
The running determination unit 33a determines whether or not the feature amount of the person trajectory satisfies the behavior condition of “running”. The running determination unit 33a determines that the person's action is “run” when the action condition is satisfied. Thereafter, the running determination unit 33a outputs “run” as the determination result. On the other hand, the running determination unit 33a does not determine that the person's action is “run” and does not output anything when the action condition is not satisfied. Here, the action condition of “run” is set as follows, for example.

<< Action conditions for "Running">>
Average speed> Average speed threshold AND
Linearity> Linearity threshold AND
Moving distance> Moving distance threshold

  In this “run” action condition, the average speed, the moving distance, and the linearity use values calculated as the feature amount of the person trajectory. Further, the threshold value of the average speed, the threshold value of the moving distance, and the threshold value of the linearity are set in advance or determined by machine learning.

  The action condition of “run” includes AND, but at least one of them may be OR. Moreover, although the exceeding (>) is used for the action condition of “running”, the above (≧), the following (≦), or the less (<) may be used.

<Second example: “Meeting people”>
The determination unit 33b that meets the person determines whether or not the feature amount of the person trajectory satisfies the action condition “Meet the person”. Then, the determination unit 33b that meets the person determines that the action of the person is “meet the person” when the action condition is satisfied. Thereafter, the determination unit 33b that meets the person outputs “Meet the person” as the determination result. On the other hand, the determination unit 33b that meets the person does not determine that “meet the person” and outputs nothing when the action condition is not satisfied. Here, the action condition of “meet people” is set as follows, for example.

<<< Action Conditions for “Meeting People” >>>
Upper limit threshold for first detection position> First detection position> Lower threshold for first detection position AND
Current detection position upper limit threshold> Current detection position> Current detection position lower limit threshold AND
Movement distance> Movement distance threshold AND
Average speed> Average speed threshold AND
Average acceleration> Average acceleration threshold AND
Linearity> Linearity threshold

  In the action condition of “meet people”, the first detection position, the current detection position, the moving distance, the average speed, the average acceleration, and the linearity use values calculated as the feature amount of the person trajectory. Further, the upper limit threshold and lower limit threshold for the first detection position, the upper limit threshold and lower limit threshold for the current detection position, the movement distance threshold, the average speed threshold, the average acceleration threshold, and the linearity threshold are set in advance or Determined by learning.

  In addition, although the action condition of “meet people” includes AND, at least one of them may be OR. Moreover, although the exceeding (>) is used as the action condition of “meet with a person”, the above (≧), the following (≦), or the less (<) may be used.

<Third example: “Walk in the opposite direction”>
The determination unit 33c walking in the opposite direction determines whether or not the feature amount of the person trajectory satisfies the action condition “walk in the opposite direction”. Then, the determination unit 33c walking in the opposite direction determines that the person's action is “walk in the opposite direction” when the action condition is satisfied. Thereafter, the determination unit 33c walking in the opposite direction outputs “walk in the opposite direction” as the determination result. On the other hand, if the determination unit 33c walking in the opposite direction does not satisfy this action condition, the determination unit 33c does not determine “walk in the opposite direction” and outputs nothing. Here, the action condition of “walking in the opposite direction” is set as follows, for example.

<< Action conditions for "walking in the opposite direction">>
Movement distance> Movement distance threshold AND
Upper limit threshold for moving direction> Moving direction> Lower limit threshold for moving direction

  In the action condition of “walking in the opposite direction”, the value calculated as the feature amount of the person trajectory is used as the movement distance and the movement direction. Further, the threshold for the movement distance and the upper limit threshold and the lower limit threshold for the movement direction are set in advance or determined by machine learning.

  The action condition “walk in the opposite direction” uses AND, but OR may be used. In addition, the action condition “walking in the opposite direction” exceeds (>), but above (≧), below (≦), or less (<) may be used.

  Needless to say, the behavior of the person determined by the behavior determination unit 33 is not limited to the first to third examples. For example, the behavior determination unit 33 may include a determination unit (not shown) for placing an object and determine “place an object”. In this case, the determination unit for placing an object may determine “place an object” when the current detection position hardly changes and there is a downward movement.

  Further, the behavior determination unit 33 is not limited to a determination method for manually setting a feature amount or a threshold value of a person trajectory used for behavior determination, and may use a determination method based on principal component analysis. Details of the determination method based on the principal component analysis will be described later as a third embodiment.

  Here, for example, it is conceivable that “meet a person” and “place an object” are determined at the same time. For this reason, the behavior determination unit 33 is configured such that each determination unit such as a determination unit 33b that meets a person or a determination unit that places an object is independent. In this configuration, the behavior determination unit 33 may output contradictory determination results. However, for example, by adding a condition that the average speed is low to the action condition of “place object”, the action determination unit 33 simultaneously determines “place object” and “run” that contradict each other. Can be prevented.

[Overall operation of human action determination device]
With reference to FIG. 13, the whole structure of the person action determination apparatus 1 is demonstrated (refer FIG. 1 suitably). As shown in FIG. 13, in the human behavior determination apparatus 1, a video is input from the photographing camera by the human region detection unit 10, and one or more human regions included in the video are detected by machine learning (Step S <b> 1). ).

  In the human behavior determination apparatus 1, the human trajectory generation unit 20 receives a human region from the human region detection unit 10 and calculates a feature amount for each human region. Then, the human behavior determination apparatus 1 determines whether or not the feature amount of the human region is similar by the human trajectory generation unit 20. Furthermore, the human behavior determination apparatus 1 generates a human trajectory by connecting the barycentric positions of the human regions of the same person by the human trajectory generating unit 20 (step S2).

  In the human behavior determination apparatus 1, the human behavior determination unit 30 receives a human trajectory from the human trajectory generation unit 20 and calculates a feature amount for each human trajectory. Further, the person action determination unit 30 determines whether or not the feature amount of the person locus satisfies the action condition. Then, when the feature amount of the person trajectory satisfies the action condition, the person action determination unit 30 determines that the person is performing an action corresponding to the action condition (step S3).

[Operation of Person Area Detection Means]
Hereinafter, the operation of the person region detection means 10 will be described with reference to FIG. 14 (see FIG. 2 as appropriate). The person area detection means 10 calculates the feature amount of the training data by the training data feature amount calculation unit 12 (step S11).

  Further, the person area detecting means 10 performs machine learning on the training data by the person area determining unit 15 to obtain learning parameters (step S12). Then, the person area detection means 10 obtains a change area from the video by the change area detection unit 13 (step S13).

  In addition, the person area detection unit 10 calculates the feature quantity of the change area by the change area feature quantity calculation unit 14 (step S14). Then, the person area detection unit 10 determines whether or not the change area is a person area by using the learning parameter by the person area determination unit 15 (step S15).

  In addition, when a plurality of person areas are input from the person area determination unit 15 by the representative person area selection unit 16, the person area detection unit 10 performs clustering and selects one person area (step S16).

[Operation of human trajectory generation means]
Hereinafter, the operation of the human trajectory generation means 20 will be described with reference to FIG. 15 (see FIG. 7 as appropriate). The person trajectory generation means 20 calculates the feature amount of the person region by the person region feature amount calculation unit 21. Then, the person trajectory generation means 20 registers the feature amount of the person region in the feature amount database 22 by the person region feature amount calculation unit 21 (step S21).

  In addition, the person trajectory generation unit 20 uses the person region feature amount matching unit 23 to match the feature amount of the person region in the current frame image with the feature amount of the person region in the past frame image, and the person region of the same person It is determined whether or not (step S22).

  In addition, the person trajectory generation means 20 obtains the center of gravity position of the person area of the same person between the past frame image and the frame image to be processed by the person trajectory generation unit 24 and connects them to obtain the person trajectory. Generate (step S23).

  In addition, the person trajectory generation unit 20 predicts a person position using the person trajectory by the person prediction unit 25 (Kalman filter), and outputs a prediction region and a prediction position (step S24).

[Configuration of person action determination means]
Hereinafter, the operation of the human behavior determination means 30 will be described with reference to FIG. 16 (see FIG. 11 as appropriate). In the human behavior determination unit 30, the human trajectory feature amount calculation unit 31 calculates a feature amount for each human trajectory. Here, the human behavior determination means 30 uses the human trajectory feature quantity calculation unit 31 to include a tracking time, an initial detection position, a current detection position, a movement direction, a movement distance, an average speed, an average acceleration, and a linearity. The feature amount of the human trajectory is calculated using the dimension feature amount (step S31).

  In addition, the person behavior determination unit 30 normalizes the feature amount of the person trajectory using the preset normalization coefficient by the human trajectory feature amount normalization unit 32 (step S32).

  In addition, the person action determination unit 30 determines the action of the person based on the normalized feature amount of the person trajectory by the action determination unit 33 (step S33). And the person action determination means 30 outputs the determination result by the action determination part 33 (step S34).

  As described above, the human behavior determination apparatus 1 according to the first embodiment of the present invention tracks a specific person from a video over a long period (for example, 30 frames) and generates a human trajectory. Then, the person behavior determination device 1 calculates various feature amounts from the person trajectory, and determines a person's behavior based on these feature amounts. As a result, the person action determination device 1 can accurately determine the action of a person even from a crowded scene in which occlusion occurs.

  In addition, even if a plurality of person areas are detected from the periphery of the same person in the same frame image, the person action determination device 1 selects one person area, so that the person's The behavior can be judged more accurately. And since the person locus | trajectory feature-value normalization part 32 normalizes the person action determination apparatus 1, the action of the person can be determined more correctly irrespective of the position of the photographing camera. Furthermore, since the human behavior determination unit 30 performs simple rule-based determination, the personal behavior determination device 1 can increase the processing speed.

(Second Embodiment)
Hereinafter, with reference to FIG. 17, a difference from the first embodiment will be described regarding the human behavior determination device 1 </ b> B according to the second embodiment of the present invention. In the human behavior determination device 1B, the personal behavior determination unit 30B includes a human trajectory feature amount calculation unit 31, a human trajectory feature amount normalization unit 32, a behavior determination unit 33, and an action presentation unit 34.

  The action presentation unit 34 receives a video from the photographing camera and adds the determination result of the behavior determination unit 33 to the video. Further, the action presentation unit 34 may add a person trajectory to the video. Furthermore, the action presentation unit 34 may add both the determination result and the person trajectory to the video. After that, the behavior presentation unit 34 presents the video with the determination result and the person trajectory added to the outside (for example, a supervisor).

  As described above, the person behavior determination device 1B according to the second embodiment of the present invention presents an image to which a person's action and a person trajectory are added to the outside, so that the determination result and the person trajectory are combined with the actual video. This makes it easier to monitor by video.

(Third embodiment)
Hereinafter, with reference to FIG. 18, a difference from the first embodiment will be described regarding a human behavior determination device 1C according to the third embodiment of the present invention. The person action determination device 1C performs principal component analysis on the feature amount of the person trajectory, reduces the number of dimensions to about three dimensions, and creates a feature space. Then, the human behavior determination apparatus 1C creates, for example, a “run” class, a “walk in the opposite direction” class, and a “meet people” class in this feature space, and the feature amount of the person trajectory is stored in the feature space. It is possible to evaluate a person's behavior by evaluating which class is included in the projection.

  As shown in FIG. 18, in the human behavior determination device 1 </ b> C, the personal behavior determination means 30 </ b> C includes a human trajectory feature amount calculation unit 31, a human trajectory feature amount normalization unit 32, an action determination unit 33 </ b> C, and a feature space database 35. And a human trajectory feature amount projection unit 36.

  First, the human behavior determination apparatus 1C generates a human trajectory from a learning video prepared in advance as in the first embodiment. In addition, the human behavior determination device 1C performs principal component analysis on the human trajectory generated from the learning video, and generates a feature space. Then, the human behavior determination device 1 </ b> C stores the generated feature space in the feature space database 35.

  Here, the principal component analysis is a method of obtaining (combining) several factors having correlations into several components and obtaining the total power and characteristics thereof. The feature space includes an action class (average coordinates, variance) for each person's action.

  The human trajectory feature amount projection unit 36 receives the human trajectory feature amount from the human trajectory feature amount normalization unit 32 and multiplies the human trajectory feature amount by an eigenvector to obtain the coordinates of the human trajectory in the feature space. Then, the human trajectory feature amount projection unit 36 outputs the coordinates of the human trajectory to the action determination unit 33C.

  The behavior determination unit 33C receives the coordinates of the human trajectory from the human trajectory feature amount projection unit 36, and calculates the distance between the coordinates of the human trajectory and the behavior class obtained for each behavior. Then, the behavior determination unit 33C determines the behavior of the person based on the distance (action condition).

  Hereinafter, with reference to FIG. 19, the determination of the human action using the feature space will be described (see FIG. 18 as appropriate). As described above, the feature amount of the human trajectory is a multidimensional feature amount including tracking time, initial detection position, current detection position, moving direction, moving distance, average speed, average acceleration, and linearity. In this case, a certain feature amount can be evaluated in a low-dimensional feature space by dimensionally compressing the multi-dimensional feature amount.

  As shown in FIG. 19, the feature amount of the person trajectory is dimensionally compressed into a three-dimensional feature space expressed by the P1, A2, and P3 axes. Thereby, the distance between the feature amounts of the person region can be evaluated as the distance between the points in the three-dimensional feature space. In FIG. 19, each point in the feature space represents a feature amount of the person trajectory.

  Here, in the feature space of FIG. 19, for example, it is assumed that the running class is within the range of the sign CL1, and the class that meets a person is within the range of the sign CL2. In this case, the running determination unit 33a determines “run” if the coordinates of the person trajectory are within the range of the running class CL1. The determination unit 33b that meets a person determines that “meet a person” if the coordinates of the person trajectory are within the range of the class CL2 that meets the person. The codes CL1 and CL2 may be set in advance or may be a variance value for each action class.

  In addition, since the general feature points often gather in the center of the feature space, the behavior determination unit 33C regards feature points far from the center as an abnormal value and determines whether it is “normal behavior” or “abnormal behavior”. May be.

  As described above, the human behavior determination apparatus 1C according to the third embodiment of the present invention can reduce the number of dimensions of the feature amount of the human region, and thus can reduce the processing load. In the third embodiment, three-dimensional dimension compression has been described. However, the present invention is not limited to this, and five-dimensional dimension compression may be performed.

(Fourth embodiment)
Hereinafter, with reference to FIG. 20, a different point from 1st Embodiment is demonstrated about the human action determination apparatus 1D which concerns on 4th Embodiment of this invention. As shown in FIG. 20, the person behavior determination device 1D includes a person area detection unit 10D, a person trajectory generation unit 20, a person behavior determination unit 30D, and a video reverse playback unit 40.

  The person action determination unit 30D determines a person's action for the forward video in the same manner as the person action determination unit 30 in FIG. Then, when determining the action to be verified such as “run”, the person action determination unit 30D outputs a reverse playback command to the video reverse playback unit 40. This verification target action is preset by an operator, for example.

  The video reverse playback means 40 includes a frame memory (not shown) that temporarily stores video input from the photographing camera. Then, the reverse video reproduction means 40 outputs the reverse reproduction video to the person area detection means 10D when the reverse reproduction command is input from the person action determination means 30D. Note that the reversely reproduced video is obtained by reproducing the video stored in the frame memory in the reverse order.

The person area detection means 10D performs the same processing as the person area detection means 10 of FIG. 1 using the reversely reproduced video from the video reverse reproduction means 40.
The person trajectory generation means 20 performs the same processing as the person trajectory generation means 20 of FIG. 1 based on the person region generated from the reverse playback video.

  Thereafter, the person action determining unit 30D determines the person's action based on the person trajectory generated from the reverse reproduction video, similarly to the person action determining unit 30 of FIG. When the same person can determine the same action to be verified for both the forward video and the reverse playback video for the same person, the person action determination unit 30D outputs the determination result. On the other hand, the person action determination unit 30D outputs no determination result when the same person cannot be determined for the same person in both the forward video and the reverse playback video.

  As described above, since the human behavior determination device 1D according to the fourth embodiment of the present invention verifies the determination result using the reversely reproduced video, the accuracy of the determination result can be further increased.

  In each embodiment, the human behavior determination device according to the present invention has been described as an independent device. However, in the present invention, a general computer can be operated by a program that functions as each of the above-described units. This program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

  The human behavior determination device according to the present invention can be used in a security field, sports video analysis, personal authentication, and a video search device using human behavior as keys. In addition, the human behavior determination device according to the present invention can be used in the field of supporting video production for television broadcasting, the Internet, etc., by presenting the obtained information and promoting viewers' understanding. Can do.

1, 1B, 1C, 1D Human action determination device 10, 10D Human region detection means 11 Human image database 12 Training data feature amount calculation unit 13 Change region detection unit 14 Change region feature amount calculation unit 15 Person region determination unit 16 Representative person region Selection unit 20 Human trajectory generation unit 21 Human region feature amount calculation unit 22 Feature amount database 23 Human region feature amount collation unit 24 Human trajectory generation unit 25 Person prediction units 30, 30B, 30C, 30D Human behavior determination unit 31 Human trajectory feature amount Calculation unit 32 Human trajectory feature amount normalization unit 33, 33C Behavior determination unit 33a Running determination unit 33b Determination unit 33c meeting a person Determination unit walking in the opposite direction Action presentation unit 35 Feature space database 36 Human trajectory feature amount projection unit 40 Video Reverse playback means

Claims (6)

A human behavior determination device that receives a continuous video of a plurality of frame images and determines the behavior of one or more persons included in the video,
Human area detection means for detecting one or more human areas included in the video by machine learning at predetermined frame intervals;
A feature amount is calculated for each person region detected by the person region detection means, and a person region having a similar feature amount of the person region in the plurality of frame images is determined as a person region of the same person, and Human trajectory generation means for generating a human trajectory by connecting the center of gravity positions of the human area;
The feature amount is calculated for each person trajectory generated by the person trajectory generation unit, and it is determined whether or not the feature amount of the person trajectory satisfies a preset action condition. When the condition is satisfied, a person action determination unit that determines that the person is performing an action corresponding to the action condition;
A human behavior determination device comprising: The person area detecting means includes
For the plurality of human regions detected from the same frame image, one of a distance between each other, a color histogram distribution distance, or a size difference is calculated as a representative feature amount, and the representative feature amount is preset. A representative person area that determines that the representative feature amount is similar when it is equal to or less than the threshold value and selects one representative person area located on the center side among the person areas with the similar representative feature quantity as the person area The person action determination device according to claim 1, further comprising a selection unit. A person prediction unit that predicts a prediction region of the person by a prediction filter from the person locus generated by the person locus generation unit;
3. The human behavior determination device according to claim 1, wherein the human trajectory generation unit generates the human trajectory when the barycentric position of the human area is included in the prediction area. The person action determination means includes
The action of the person is determined based on the action condition in which a correspondence relationship between a feature amount of the person trajectory and a threshold value set in advance for each action of the person is set in advance. The person action determination device according to claim 3. When the person action determining means determines that the person is performing a preset action to be verified, the person action determining means outputs a reverse playback command that is a command to play the video in the reverse direction,
A reverse playback command is input from the person behavior determination unit, and a video reverse playback unit is further provided that plays back the video in the reverse direction based on the reverse playback command and outputs the video to the person area detection unit. The human behavior determination device according to any one of claims 1 to 4. In order to determine a behavior of one or more persons included in the video, and a video in which a plurality of frame images are continuously input,
Human area detection means for detecting one or more human areas included in the video by machine learning at predetermined frame intervals;
A feature amount is calculated for each person region detected by the person region detection means, and a person region having a similar feature amount of the person region in the plurality of frame images is determined as a person region of the same person, and Human trajectory generation means for generating a human trajectory by connecting the gravity center positions of the human area;
The feature amount is calculated for each person trajectory generated by the person trajectory generation unit, and it is determined whether or not the feature amount of the person trajectory satisfies a preset action condition. When the condition is satisfied, a person action determination unit that determines that the person is performing an action corresponding to the action condition,
It is made to function as a person action judging program characterized by things.

Images