JP6904651B2 - Programs, devices and methods that recognize a person's behavior using multiple recognition engines - Google Patents

Description

The present invention relates to a technique for recognizing a person's behavior from video data.

In recent years, by using deep learning, the recognition accuracy in human behavior recognition has been dramatically improved.
Conventionally, there is a moving object recognition technique for detecting the movement of an object from a moving feature (optical flow) (see, for example, Non-Patent Document 1). For example, according to Two-stream ConvNets, using CNN (Spatial stream ConvNet) in the spatial direction and CNN (Temporal stream ConvNet) in the time-series direction, the appearance features of objects and backgrounds in the image and the horizontal direction of optical flow. Extract the motion features in the sequence of components and vertical components. By integrating both of these features, behavior is recognized with high accuracy.

There is also a technique of recognizing an action by 3D-CNN using a segment of 64 frames as a processing unit (see, for example, Non-Patent Document 2). According to this technique, a 3D convolution including time axis information is applied to the technique of Non-Patent Document 1. A deep deep model is trained with a large amount of teacher data.
Further, there is also a technique of segmenting video data into N (= 3) equal parts and integrating the scores of each segment (see, for example, Non-Patent Document 3). According to this technique, a long time axis information is applied to the technique of Non-Patent Document 1, and a large amount of teacher data is trained in a deep deep model.

On the other hand, there is also a technique for estimating two-dimensional joint data of a person's skeleton from video data taken by a general Web camera. According to this technique, it is not possible to estimate up to 3D joints, but it does not require a depth sensor such as Kinect®.

FIG. 1 is a system configuration diagram having a recognition device.

According to the system of FIG. 1, the recognition device 1 functions as, for example, a server connected to the Internet. The recognition device 1 has a recognition engine in which a learning model is constructed in advance based on teacher data. When the recognition engine recognizes a person's behavior, the teacher data is a pre-association between the video data in which the person's behavior is reflected and the action target (context).

Each terminal 2 is equipped with a camera, and transmits video data of a person's behavior to the recognition device 1. The terminal 2 is a smartphone or mobile terminal owned by each user and connects to an access network such as a mobile phone network or a wireless LAN.
Of course, the terminal 2 is not limited to a smartphone or the like, and may be, for example, a Web camera installed in the house. Further, the video data captured by the Web camera may be recorded on the SD card, and the recorded video data may be input to the recognition device 1.

Specifically, for example, the user is asked to take a picture of his / her behavior with the camera of his / her smartphone. The smartphone transmits the video data to the recognition device 1. The recognition device 1 estimates a person's behavior from the video data, and uses the estimation result in various applications.
It should be noted that each function of the recognition device 1 may be incorporated in the terminal 2.

Karen Simonyan and Andrew Zisserman, “Two-Stream Convolutional Networks for Action Recognition in Videos,” in NIPS 2014, [online], [Search January 24, 2018], Internet <URL: https://arxiv.org/ abs / 1406.2199.pdf ＞ Joao Carreira, Andrew Zisserman. "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset." CVPR2017 (2017), [online], [Search January 24, 2018], Internet <URL: https: // arxiv.org/abs/1705.07750> Wang, Limin, et al. "Temporal segment networks: Towards good practices for deep action recognition." European Conference on Computer Vision. Springer International Publishing, 2016, [online], [Search January 24, 2018], Internet < URL: http://www.eccv2016.org/files/posters/P-3B-42.pdf ＞ Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR2017 (2017), [online], [Search January 24, 2018], Internet <URL: https: // arxiv .org / abs / 1611.08050 ＞ Soo Kim, Tae & Reiter, Austin. “Interpretable 3D Human Action Analysis with Temporal Convolutional Networks.” CVPRW 2017, [online], [Search January 24, 2018], Internet <URL: http: //ieeexplore.ieee .org / document / 8014941 /? Reload = true ＞ Gunnar Farneback, “Two-Frame Motion Optimization Based on Polynomial Expansion, Image Analysis,” Volume 2749 of the series Lecture Notes in Computer Science, pp 363-370, June 2003, [online], [Search January 24, 2018 ], Internet <http://liu.diva-portal.org/smash/record.jsf?pid=diva2%3A269471&dswid=-8845> OpenPose, [online], [Search January 24, 2018], Internet <URL: https://github.com/CMU-Perceptual-Computing-Lab/openpose> "I tried OpenPose, which can detect bones from videos and photos", [online], [Search on January 24, 2018], Internet <URL: http://hackist.jp/?p=8285> "OpenPose has been upgraded so that you can try 3d pose estimation", [online], [Search on January 24, 2018], Internet <URL: http://izm-11.hatenablog.com / entry / 2017/08/01/140945 ＞

The recognition engine that recognizes the behavior of a person learns the video data in which the person is reflected as teacher data, and is adjusted to estimate the behavior with high accuracy from the video data to be estimated.
However, in an actual environment, the recognition accuracy of the context may decrease due to the influence of the video area other than the person.

In addition, the recognition accuracy of the recognition engine is generally joint recognition-> moving object recognition-> object recognition in descending order.
However, the accuracy of joint recognition is not always high due to the influence of shooting angle, illuminance, occlusion, resolution, and the like. That is, even if the optimally learned recognition engine is used, the recognition accuracy may be improved by using a different type of recognition engine depending on the shooting environment.

Further, the recognition engine is a machine learning engine that is based on classification and gives "behavior" (class) to the video data to be estimated.
However, if the score from the recognition engine is high, the recognition accuracy is not necessarily high.
When the movement of a similar non-object is detected, the score may be relatively medium due to the high recognition accuracy, while the score may be extremely high due to the low recognition accuracy.
For example, even if the same behavior is recognized, a moving object recognition engine with relatively high recognition accuracy will give a relatively medium score, but an object recognition engine with relatively low recognition accuracy will give a relatively high score. May be output.
Considering this, the score calculated by each recognition engine cannot be considered as the recognition accuracy of the action as it is.

Therefore, the present invention can improve the recognition accuracy of actions (contexts) from a comprehensive viewpoint based on the scores of a plurality of recognition engines without being affected by the video area other than the person with respect to the video data. It is intended to provide programs, devices and methods.

According to the present invention, it is an action recognition program that causes a computer to function so as to recognize an action from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from video data,
A joint recognition engine that recognizes actions from skeleton information of video data,
Area extraction means for extracting the enclosed area of skeleton information from video data,
A motion recognition engine that recognizes actions from the enclosed area of video data,
A score integration means that outputs an integrated score in which the scores of the joint recognition engine and the motion recognition engine are weighted and integrated according to each recognition engine for each action.
It is a weight calculation means that inputs multiple training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a value that increases as the score statistical value decreases as a weight. It is characterized by operating a computer.
According to other embodiments in the program of the present invention
The weight calculation means uses the inverse value of the score statistical value (the sum of the inverse values of all recognition engines is 1) as the weight.
It is also preferable to make the computer function as such.

According to the present invention, it is an action recognition program that causes a computer to function so as to recognize an action from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from video data,
A joint recognition engine that recognizes actions from skeleton information of video data,
A joint behavior determination means for determining whether or not the behavior score calculated by the joint recognition engine satisfies a predetermined condition,
Area cutting means for extracting the enclosed area of skeleton information from the video data when it is determined to be true by the joint behavior determination means,
A motion recognition engine that recognizes actions from the enclosed area of video data,
It is characterized in that a computer functions as a score integration means for outputting an integrated score that integrates the scores of the joint recognition engine and the motion recognition engine for each action.

According to other embodiments in the behavior recognition program of the present invention.
As a predetermined condition, the joint behavior determining means preferably causes a computer to function so as to determine whether or not the score of the maximum value or the average value in a plurality of actions is equal to or less than a predetermined threshold value.

According to other embodiments in the behavior recognition program of the present invention.
As a predetermined condition, the joint behavior determining means preferably causes a computer to function so as to determine whether or not the behavior having the maximum score is a predetermined target behavior.

According to other embodiments in the behavior recognition program of the present invention.
The joint recognition engine further outputs important joint parts that satisfy predetermined conditions.
It is also preferable that the region cutting means make the computer function so as to extract the surrounding region including the important joint portion output from the joint recognition engine.

According to other embodiments in the behavior recognition program of the present invention.
The score integration means integrates the scores of each recognition engine with a weight corresponding to the recognition engine .
As a weight calculation means that inputs multiple training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a value that increases as the score statistical value decreases as a weight.
It is also preferable to make the computer function further.

According to other embodiments in the behavior recognition program of the present invention.
It is also preferable that the weight calculation means causes the computer to function so that the inverse value of the statistical value of the score (the sum of the inverse values of all recognition engines is 1) is used as the weight.

According to other embodiments in the behavior recognition program of the present invention.
It is also preferable that the motion recognition engine makes the computer function so that it is based on optical flow.

According to other embodiments in the behavior recognition program of the present invention.
The moving object recognition engine consists of an object recognition engine based on RGB images and a moving object recognition engine based on optical flow.
It is also preferable that the score integrating means causes the computer to function so as to integrate the scores of the joint recognition engine, the moving object recognition engine, and the object recognition engine for each action.

According to other embodiments in the behavior recognition program of the present invention.
It also has a skeleton information normalization means that normalizes the skeleton information by shifting and expanding / contracting it with respect to the coordinate system of the time series.
It is also preferable that the region cutting means causes the computer to function so as to extract the minimum region surrounding the normalized skeleton information as the surrounding region.

According to the present invention, it is a device that recognizes an action from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from video data,
A joint recognition engine that recognizes actions from skeleton information of video data,
Area extraction means for extracting the enclosed area of skeleton information from video data,
A motion recognition engine that recognizes actions from the enclosed area of video data,
A score integration means that outputs an integrated score in which the scores of the joint recognition engine and the motion recognition engine are weighted and integrated according to each recognition engine for each action.
A weight calculation means that inputs multiple training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a value that increases as the score statistical value decreases as a weight. It is characterized by having.

According to the present invention, it is a device that recognizes an action from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from video data,
A joint recognition engine that recognizes actions from skeleton information of video data,
A joint behavior determination means for determining whether or not the behavior score calculated by the joint recognition engine satisfies a predetermined condition,
Area cutting means for extracting the enclosed area of skeleton information from the video data when it is determined to be true by the joint behavior determination means,
A motion recognition engine that recognizes actions from the enclosed area of video data,
Each action is characterized by having a score integration means for outputting an integrated score that integrates the scores of the joint recognition engine and the motion recognition engine.

According to the present invention, it is a recognition method of a device that recognizes an action from video data in which a person is reflected.
The device is
The first step of extracting skeleton information based on human joints from video data in chronological order,
The second step of recognizing behavior joints from skeleton information of video data,
The third step of extracting the enclosed area of skeleton information from the video data,
From the enclosed area of the video data, the fourth step of recognizing the action as a moving object,
For each action, the scores of the second step and the fourth step are weighted corresponding to each step, and the fifth step of outputting the integrated score is executed .
A plurality of training data based on different behaviors are input, score statistics are calculated for each of the second step and the fourth step, and a value that increases as the score statistics decrease is given as a weight. It is characterized by that.

According to the present invention, it is a recognition method of a device that recognizes an action from video data in which a person is reflected.
The device is
The first step of extracting skeleton information based on human joints from video data in chronological order,
The second step of recognizing behavior joints from skeleton information of video data,
The third step of determining whether or not the action score calculated by the second step satisfies a predetermined condition, and
The fourth step of extracting the enclosed area of the skeleton information from the video data when it is determined to be true by the third step,
From surrounding areas of the video data, and a fifth step moving body recognizes action,
It is characterized in that each action is executed with a sixth step of outputting an integrated score in which the scores of the second step and the fifth step are integrated.

According to the program, apparatus and method of the present invention, recognition of behavior (context) of video data from a comprehensive viewpoint based on scores of a plurality of recognition engines without being affected by a video area other than a person. The accuracy can be improved.

It is a system block diagram which has a recognition device. It is a 1st functional block diagram of the recognition device in this invention. It is a 1st image diagram which shows the skeleton information and the enclosed area. It is a 2nd image figure which shows the enclosed area. FIG. 5 is a first functional configuration diagram based on FIG. 2 composed of a plurality of motion recognition engines. It is a 2nd functional block diagram of the recognition device in this invention. FIG. 6 is a second functional configuration diagram based on FIG. 6 composed of a plurality of motion recognition engines. It is a 3rd image figure which shows the enclosed area.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 2 is a first functional configuration diagram of the recognition device according to the present invention.

The recognition device 1 uses a plurality of recognition engines to estimate the behavior (context) of a person from the video data in which the person is reflected.
According to FIG. 2, the recognition device 1 includes a skeleton information extraction unit 11, a joint recognition engine 12, a region cutting unit 131, a moving object recognition engine 132, a score integration unit 14, and a weight calculation unit 15. It also optionally has a skeleton information normalization unit 111. These functional components can be realized by executing a program that makes the computer mounted on the device function. In addition, the processing flow of these functional components can be understood as a method of recognizing an action (context) for video data.

[Skeleton information extraction unit 11]
The skeleton information extraction unit 11 extracts "skeleton information" based on human joints from video data in chronological order (see, for example, Non-Patent Document 4). The skeleton information refers to joint data of a two-dimensional skeleton in which a reliability (0 to 1) is given to each joint. Since the joint data is based on two dimensions, it is extracted from the video data taken by a general Web camera.

Specifically, a skeleton model such as OpenPose (registered trademark) is used to extract characteristic points of human joints (see, for example, Non-Patent Documents 7 to 9). OpenPose is software that can detect multiple human body / hand / face key points in real time from images, and is published by GitHub. For example, 15 key points can be detected in the entire human body shown in the captured image.

FIG. 3 is a first image diagram showing the skeleton information and the enclosed area.

According to FIG. 3, one person is reflected in the video data. The reliability (0 to 9) is calculated for each joint (Nose, Neck, RShoulder, RElbow, ...). The skeleton information refers to information in which the two-dimensional coordinate points of each of the 18 joints and their reliability are linked in each frame.

The derived skeleton information is output to the joint recognition engine 12 and the region cutting unit 131.
The skeleton information may optionally be output to the joint recognition engine 12 and the region cutting unit 131 via the skeleton information normalization unit 111.

[Skeleton information normalization unit 111] (optional)
The skeleton information normalization unit 111 normalizes the skeleton information by shifting / expanding / contracting it with respect to the time-series coordinate system.
The normalized skeleton information is output to the joint recognition engine 12 and the region cutting unit 131.

(S1) The origin coordinates and the expansion / contraction scale are calculated based on the reference frame.
The reference frame is the first frame in which the reliability of "Neck", "LHip", and "RHip" simultaneously detected from a person reflected in the video data exceeds a certain level.
_{The origin coordinates refer to the coordinates p i} (Neck) of the Neck joint of the person i in the reference frame.
The expansion / contraction scale scale _i is the average value of the distance between the Neck and the L Hip and the distance between the Neck and the RH ip set to 100, and is calculated as follows.
scale _i = 200 / (|| p _i (Neck) -p _i (LHip) || + || p _i (Neck) -p _i (RHip) ||)
scale _i : Scale of person i
p _i (): using a two-dimensional coordinate (S2) stretching scale scale _i person i, a skeleton for each person detected in each frame is shifted stretching as follows.
^{_{np i t (j) = scale}} i * (p i t (j) -pi (Neck))
t: frame number
j: Joint number
p _i ^t (j): coordinates of the joint j of person i in the t-th frame
np _i ^t (j): Normalization of joint j person i at the t frame coordinate Incidentally, according to the embodiment described above, the extraction of the reference frame, the relative positional relationship is hardly changed joint "Neck" Although "LHip" and "RHip" are used, "Neck", "L Shoulder", and "R Shoulder" may be used instead of them.

[Joint recognition engine 12]
The joint recognition engine 12 builds a learning model for deep learning in advance based on teacher data in which "behavior" is associated with skeleton information (normalized skeleton information) of video data.
Then, the joint recognition engine 12 recognizes "behavior" from the skeleton information (normalized skeleton information) of the video data by using the learning model (see, for example, Non-Patent Document 5). The joint recognition engine 12 is based on classification, and calculates a score for each class (estimable behavior (context)).
The joint recognition engine 12 recognizes a person's actions such as "drinking", "eating", "running", and "folding" from the angle and position of the person's joints.
Then, the score for each action is output to the score integration unit 14.

[Area cutout section 131]
The area cutting unit 131 extracts the "enclosed area" of the skeleton information from the video data. The enclosed area is the minimum area that surrounds the skeleton information (normalized skeleton information). In this way, by inputting only the enclosed area in which the person is reflected into the moving object recognition engine 132 in the subsequent stage, the recognition process can be speeded up and the accuracy of action recognition can be improved.

According to FIG. 3 described above, first, a “joint bounding box” that is the smallest rectangle so as to include the coordinate points (18 pieces) of all the joints is calculated (short dashed line). Then, from the bounding box of the joint, an enlarged box enlarged at a predetermined ratio is derived as an "enclosed area" (long dashed line).

FIG. 4 is a second image diagram showing the enclosed area.

According to FIG. 4, two people are reflected in the video data.
First, a "joint bounding box" that is the smallest rectangle to include the coordinate points of all joints is calculated. Next, the joint data is divided for each person from the connection configuration of the frame connecting the joints.
Then, for each person, an enlarged box enlarged at a predetermined ratio is derived as a "enclosed area" from the bounding box of the joint.
Further, with respect to FIG. 4, one area that includes the surrounding area of two people within the range of the image area may be redistributed as the “enclosed area”.

[Motion recognition engine 132]
The motion recognition engine 132 also constructs a learning model for deep learning in advance based on teacher data in which "behavior" is associated with video data.
Then, the moving body recognition engine 132 recognizes the "behavior" from the enclosed area of the video data by using the learning model. The motion recognition engine 132 is also based on the classification, and calculates the score for each class (estimable behavior (context)).
Then, the score for each action calculated by each motion recognition engine 132 is output to the score integration unit 14.

FIG. 5 is a first functional configuration diagram based on FIG. 2, which is composed of a plurality of motion recognition engines.

The moving object recognition engine 132 may be composed of two recognition engines as follows.
(1) Object recognition engine based on RGB images (2) Motion recognition engine based on optical flow Of course, the recognition engine included in the motion recognition engine 132 is not limited to two, and three or more different types of recognition engines are combined. It may be a thing.
It is preferable that the types of context "actions" are the same in order to be integrated by the score integration unit 14 in the subsequent stage.

(1) An object recognition engine based on RGB recognition specifically estimates an object (object) reflected in a captured image by using a neural network such as CNN (Convolutional Neural Network) (for example, Non-Patent Document 2). reference).

(2) The moving object recognition engine based on the optical flow extracts the part where the same feature point moves between frames and expresses the movement of the object in the video data as a "vector" (for example, Non-Patent Document 6). reference).

The moving object recognition engine based on the optical flow calculates two optical flows having the same resolution when two adjacent frames in the video data (RGB image) are input. The RGB image is converted into a luminance image Y by the following formula.
Y = 0.299 x R + 0.587 x G + 0.114 x B
Y: Pixel brightness value
R, G, B: R, G, B values of pixels According to the technique of Non-Patent Document 6, the Y component of any pixel is quadratic polynomial as shown in the following equation in a small region of the first frame. Can be expressed in basis.
f ₁ (x) = x ^T A ₁ x + b ₁ ^T x + c ₁
f ₁ (x): Y component of the target pixel
x: Position coordinates of the target pixel of the Y component in the first frame
A ₁ , b ₁ , c ₁ : Coefficients calculated in that area Similarly, the corresponding area of the second frame is as follows.
f ₂ (x) = f ₁ (x−d)
= (X-d) ^TA ₁ (x-d) + b ₁ ^T (x-d) + c _{1
^{= X T A 1 x + (}} b 1 -2A 1 d) T x + d T A 1 d-b 1 T d + c 1
= X ^T A ₂ x + b ₂ ^T x + c ₂
d: Optical flow of target pixel x (difference in position coordinates)
A ₂ , b ₂ , c ₂ : Coefficients calculated in that area The optical flow d is calculated by the following formula.
d = -1 / 2A ₁ ^-1 (b ₂ −b ₁ )

[Score integration unit 14]
As shown in FIG. 2, the score integration unit 14 outputs an “integrated score” that integrates the statistical values of the scores of the joint recognition engine 131 and the motion recognition engine 132 for each “action”.
The statistical value of the score may be the maximum value or the average value of the scores in a plurality of actions.
Further, as shown in FIG. 5, when the moving body recognition engine is composed of the object recognition engine 1321 and the moving body recognition engine 1322, the score integration unit 14 performs the joint recognition engine 131, the object recognition engine 1321 and the moving body recognition for each action. The scores of each engine 1322 are integrated.

Here, the score integration unit 14 adds a "weight" corresponding to the recognition engine to each score of the recognition engine 13 to calculate the ^{integrated score SA all (act) for the action act.}
SA ^RGB (act): Object recognition engine (RGB image) score SA ^flow (act): Motion recognition engine (optical flow image) score SA ^skeleton (act): Joint recognition engine (skeleton information) score w ^skeleton : Joint Weight for recognition engine score w ^RGB : Weight for object recognition engine score w ^flow : Weight for motion recognition engine score SA ^all (act) = w ^skeleton SA ^skeleton (act) + w ^RGB SA ^RGB (act) + w ^flow SA ^flow (act)
Then, the behavior of the video data is estimated by the following formula.
best _act = arg max _act (SA ^all (act))

In this way, by assigning different "weights" depending on the recognition engine, it is possible to avoid the unfairness that "the higher the score from the recognition engine, the higher the recognition accuracy is not necessarily high".

[Weight calculation unit 15]
The weight calculation unit 15 inputs a plurality of training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a larger value of "weight" as the score statistical value is lower. Specifically, the inverse value of the statistical value (for example, the average value) of the score (the sum of the inverse values of all recognition engines is 1) may be set as the "weight". The "weight" is calculated as follows, for example.
MSA ^skeleton = max _act (SA ^skeleton (act))
MSA ^RGB = max _act (SA ^RGB (act))
MSA ^flo _w = max _act (SA ^flow (act))
w ^skeleton = e ^-MSAskeleton / (e ^-MSAskeleton + e ^-MSARGB + e ^-MSAflow )
w ^RGB = e- ^MSARGB / (e- ^MSAskeleton + e- ^MSARGB + e- ^MSAflow )
w ^flow = e- ^MSAflow / (e- ^MSAskeleton + e- ^MSARGB + e- ^MSAflow )
max: Maximum value (statistical value)

Specifically, the scores calculated by the joint recognition engine 131 and the motion recognition engine 132 (RGB image recognition and optical flow image recognition) are complementarily and fairly combined. That is, in order to normalize the scale between the scores of each recognition engine, the reciprocal of the score (for example, the average value) of each classifier is set.
As a result, a recognition engine that tends to calculate the overall score high and a recognition engine that tends to output the overall score low can be combined and the scores can be integrated at a ratio according to the tendency. ..

For example, with respect to the input video data, the recognition accuracy of the optical flow recognition engine 1322 is higher than the recognition accuracy of the RGB recognition engine 1321. On the other hand, the score of the behavioral statistics in the optical flow recognition engine 1322 is lower than the score of the behavioral statistics in the RGB recognition engine 1321. In that case, the weight for the score of the optical flow recognition engine 1322 is increased, and the weight for the score of the RGB recognition engine 1321 is decreased. In particular, it is effective to increase the "weight" for a low score non-linearly.

FIG. 6 is a second functional configuration diagram of the recognition device according to the present invention.

According to FIG. 6, as compared with FIG. 2, skeleton information (normalized skeleton information) is input only to the joint recognition engine 12, and the joint action is determined according to the score of each action of the joint recognition engine 12. Part 16 is being executed.

[Joint behavior determination unit 16]
The joint behavior determination unit 16 determines whether or not the behavior score calculated by the joint recognition engine 12 satisfies a predetermined condition.
On the other hand, the area cutting unit 131 extracts the enclosed area of the skeleton information from the video data when it is determined to be true by the joint behavior determination unit 16.

The joint behavior determination unit 16 executes one of the following two embodiments as a predetermined condition.
(Determination 1) The joint behavior determination unit 16 determines, as a predetermined condition, whether or not the score of the maximum value or the average value in a plurality of actions is equal to or less than a predetermined threshold value. This means that the motion recognition engine 132 in the subsequent stage is made to function only when the score of the joint recognition engine 12 is low.

(Determination 2) The joint behavior determination unit 16 determines, as a predetermined condition, whether or not the action having the maximum score is a predetermined purpose action. This means that the motion recognition engine 132 in the subsequent stage functions only when the action of the joint recognition engine 12 is a predetermined purpose action.
The predetermined purpose action is recognizable by the joint recognition engine 12, such as "reading", "writing", and "typing".

When the joint behavior determination unit 16 determines that the result is true by the determination 1 or the determination 2, the joint behavior determination unit 16 outputs the video data to the area cutting unit 131. On the other hand, if it is determined to be false, the score for each action of the joint recognition engine 12 is output to the score integration unit 14.
The area cutting unit 131 extracts the enclosed area of the skeleton information from the video data only when it is determined to be true by the joint behavior determination unit 16.

FIG. 7 is a second functional configuration diagram based on FIG. 6 configured by a plurality of motion recognition engines.

Similar to FIG. 5, the moving object recognition engine 132 is composed of an object recognition engine 1321 based on an RGB image and a moving object recognition engine 1322 based on an optical flow. In this case, the area cutting unit 131 inputs the enclosed area of the video data to both recognition engines 132.

FIG. 8 is a third image diagram showing the enclosed area.

According to FIG. 8, the joint recognition engine 12 outputs an important joint portion satisfying a predetermined condition. By making the joint recognition engine 12 function first, it is possible to roughly recognize what kind of behavior it is. The important joint parts are "hand", "head", "upper body", "lower body", and "whole body". (See FIG. 8 (a)). Here, among the classified joint parts, the larger the displacement amount and / or the number of displacements in a unit time, the more important the joint part is determined.

On the other hand, the region cutting portion 131 extracts the surrounding region including the important joint portion output from the joint recognition engine 12.
In the case of the important joint part "hand", as shown in FIG. 8B, a bounding box (short dashed line) that is the smallest rectangle including the joints "hand" and "elbow" in the skeleton information is extracted. Then, from the bounding box, an enlarged box (long broken line) enlarged at a predetermined ratio is derived as a "enclosed area".
Further, for the important joint part "hand", a circle having a constant radius may be formed with the hand as the center of the circle, and a bounding box based on the edge thereof may be calculated.

As described in detail above, according to the program, apparatus and method of the present invention, the video data is not affected by the area other than the person, and from a comprehensive viewpoint based on a plurality of recognition engines. It is possible to improve the recognition accuracy of actions (contexts).

In particular, according to the present invention, since the action is recognized only in the human area of the video data to be estimated, the recognition accuracy of the context is improved.
Further, by using a plurality of recognition engines of different types, it is possible to improve the recognition accuracy even in various shooting environments without depending on the estimation result of a specific recognition engine.
Further, the difference between the score and the recognition accuracy in different types of recognition engines can be eliminated by giving a "weight" according to the statistical value of the score for each recognition engine.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Recognition device 11 Skeleton information extraction unit 111 Skeleton information normalization unit 12 Joint recognition engine 131 Area cutout unit 132 Motion recognition engine 1321 RGB recognition engine 1322 Optical flow recognition engine 14 Score integration unit 15 Weight calculation unit 16 Joint behavior judgment unit 2 terminals

Claims (15)

It is an action recognition program that makes a computer function so that it recognizes actions from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from the video data,
A joint recognition engine that recognizes actions from the skeleton information of the video data,
An area cutting means for extracting the enclosed area of the skeleton information from the video data, and
A motion recognition engine that recognizes actions from the enclosed area of the video data,
A score integration means that outputs an integrated score in which the scores of the joint recognition engine and the motion recognition engine are weighted and integrated corresponding to each recognition engine for each action.
A weight calculation means that inputs a plurality of training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a value that increases as the score statistical value decreases as a weight. A behavior recognition program characterized by the functioning of a computer. The weight calculation means uses an inverse value of the statistical value of the score (the sum of the inverse values of all recognition engines is 1) as a weight.
The behavior recognition program according to claim 1, wherein the computer functions as described above. It is an action recognition program that makes a computer function so that it recognizes actions from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from the video data,
A joint recognition engine that recognizes actions from the skeleton information of the video data,
A joint behavior determination means for determining whether or not the behavior score calculated by the joint recognition engine satisfies a predetermined condition,
When the joint behavior determination means determines that the data is true, the area cutting means for extracting the enclosed area of the skeleton information from the video data, and the area cutting means.
A motion recognition engine that recognizes actions from the enclosed area of the video data,
A behavior recognition program characterized in that a computer functions as a score integration means for outputting an integrated score that integrates the scores of the joint recognition engine and the motion recognition engine for each action. The third aspect of claim 3, wherein the joint behavior determining means causes a computer to function as the predetermined condition to determine whether or not the score of the maximum value or the average value in a plurality of actions is equal to or less than a predetermined threshold value. Behavior recognition program. The joint action determining means, wherein as the predetermined condition, the score of the maximum value actions, action according to claim 3, characterized in that causes a computer to function so as to determine whether or not the predetermined intended behavior Recognition program. The joint recognition engine further outputs important joint parts that satisfy predetermined conditions.
The action according to any one of claims 3 to 5 , wherein the region cutting means causes a computer to function so as to extract an enclosed region including the important joint portion output from the joint recognition engine. Recognition program. The score integration means integrates the scores of each recognition engine with a weight corresponding to the recognition engine .
A weight calculation means that inputs a plurality of training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a value that increases as the score statistical value decreases as a weight.
The behavior recognition program according to any one of claims 3 to 6 , wherein the computer is further operated. The claim is characterized in that the weight calculation means causes the computer to function so that the inverse value of the statistical value of the score (the sum of the inverse values of all recognition engines is 1) is used as the weight. The behavior recognition program according to item 7. The behavior recognition program according to any one of claims 1 to 8, wherein the motion recognition engine causes a computer to function so as to be based on an optical flow. The moving object recognition engine includes an object recognition engine based on an RGB image and a moving object recognition engine based on an optical flow.
Any one of claims 1 to 8, wherein the score integrating means causes a computer to function to integrate the scores of the joint recognition engine, the moving object recognition engine, and the object recognition engine for each action. The behavior recognition program described in the section. Further having a skeleton information normalization means for normalizing the skeleton information by shifting / expanding / contracting it with respect to a time-series coordinate system.
The behavior recognition according to any one of claims 1 to 10, wherein the area cutting means causes a computer to function so as to extract a minimum area surrounding the normalized skeleton information as the surrounding area. program. It is a device that recognizes actions from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from the video data,
A joint recognition engine that recognizes actions from the skeleton information of the video data,
An area cutting means for extracting the enclosed area of the skeleton information from the video data, and
A motion recognition engine that recognizes actions from the enclosed area of the video data,
A score integration means that outputs an integrated score in which the scores of the joint recognition engine and the motion recognition engine are weighted and integrated corresponding to each recognition engine for each action.
A weight calculation means that inputs a plurality of training data based on different behaviors, calculates a score statistical value for each recognition engine, and assigns a value that increases as the score statistical value decreases as a weight. A device characterized by having. It is a device that recognizes actions from video data in which a person is reflected.
A skeleton information extraction means that extracts skeleton information based on human joints in chronological order from the video data,
A joint recognition engine that recognizes actions from the skeleton information of the video data,
A joint behavior determination means for determining whether or not the behavior score calculated by the joint recognition engine satisfies a predetermined condition,
When the joint behavior determination means determines that the data is true, the area cutting means for extracting the enclosed area of the skeleton information from the video data, and the area cutting means.
A motion recognition engine that recognizes actions from the enclosed area of the video data,
A device characterized by having a score integration means for outputting an integrated score that integrates the scores of the joint recognition engine and the motion recognition engine for each action. It is a recognition method of a device that recognizes actions from video data in which a person is reflected.
The device is
The first step of extracting skeleton information based on human joints from the video data in chronological order,
From the skeleton information of the video data, the second step of recognizing the action joint and
A third step of extracting the enclosed area of the skeleton information from the video data, and
A fourth step of recognizing an action as a moving object from the enclosed area of the video data,
For each action, the scores of the second step and the fourth step are weighted corresponding to each step, and the fifth step of outputting the integrated score is executed .
A plurality of training data based on different behaviors are input, a score statistical value is calculated for each of the second step and the fourth step, and a value that increases as the score statistical value becomes lower is given as a weight. <br / > A device recognition method characterized by this. It is a recognition method of a device that recognizes actions from video data in which a person is reflected.
The device is
The first step of extracting skeleton information based on human joints from the video data in chronological order,
From the skeleton information of the video data, the second step of recognizing the action joint and
The third step of determining whether or not the action score calculated by the second step satisfies a predetermined condition, and
When the result is determined to be true by the third step, the fourth step of extracting the enclosed area of the skeleton information from the video data and
From the surrounding region of the image data, and a fifth step moving body recognizes action,
A method of recognizing a device, which comprises executing a sixth step of outputting an integrated score in which the scores of the second step and the fifth step are integrated for each action.

Images