JP2018181273A - Image processing apparatus, method thereof, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize behavior of a plurality of persons from a video image.SOLUTION: An image processing apparatus includes: acquisition means of acquiring a video image including time-series still images; detection means of detecting one or more objects in each still image from the video image; feature quantity extraction means of extracting feature quantities corresponding to each of the objects from the still images; object integration means of integrating feature quantities corresponding to each of the objects in the still images; time-series integration means of integrating the feature quantities integrated in the still images, for the time-series still images; and identification means of identifying behavior of the object in the video image, on the basis of the feature quantities integrated for the time-series still images.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technology for recognizing the behavior of an object such as a person from a moving image. In particular, the present invention relates to a technology for recognizing an action performed by a plurality of persons.

  Recognition of person's action by image analysis (hereinafter referred to as action recognition) is a useful technique in applications such as monitoring, marketing, and sports analysis. For example, a monitoring application that detects a specific action of a subject from a video based on a recognition result of action recognition, a video search using an action label as a key indicating the type of action, counting of actions of interest in marketing, sports video There are various applications such as Stats analysis.

  In the case of the video taken at that time, it is common that a large number of people appear simultaneously in one frame. As an action recognition method using such a video, in Patent Document 1, even in a crowded situation where many people appear and hiding frequently occurs, long trajectory information is generated using motion prediction of a person, thereby making it possible to display in the video. There is disclosed a technology for stably recognizing individual actions of a plurality of persons. Recognition of human behavior by image analysis (hereinafter referred to as action recognition) is a useful technique in applications such as monitoring, marketing, and sports analysis. For example, a monitoring application that detects a specific action of a subject from a video based on a recognition result of action recognition, a video search using an action label as a key indicating the type of action, counting of actions of interest in marketing, sports video There are various applications such as Stats analysis.

  In the case of the video taken at that time, it is common that a large number of people appear simultaneously in one frame. As an action recognition method using such a video, in Patent Document 1, even in a crowded situation where many people appear and hiding frequently occurs, long trajectory information is generated using motion prediction of a person, thereby making it possible to display in the video. There is disclosed a technology for stably recognizing individual actions of a plurality of persons.

  Furthermore, in the application of surveillance and sports analysis, it is thought that it becomes possible to lead to application with high added value by enabling recognition of a coordinated action involving multiple persons in a video. For example, it is possible to annotate the observer with the situation of a plurality of persons in the video in a more intuitive and easy-to-understand expression (eg, "a matrix is made", "a dispute is occurring", etc.). In addition, it is applied to detection of a crime performed by a plurality of people in cooperation, and in sports, it is an application example such as analysis of team play becomes possible.

  In Non-Patent Document 1, as a technology for recognizing the cooperative action of a plurality of persons, recognition of cooperative actions of a plurality of persons is performed, and recognition of primitive individual actions of whether individual persons are standing or walking is performed. Do. Then, hierarchical graphical recognition of interaction behavior between two parties (face-to-face, in-line, etc.), overall collaborative behavior recognition (discussion, walking-side-by-side, etc.) It is realized using a model.

Patent No. 5285575 gazette

Wongun Choi and Silvio Savarese, “A unified framework for multi-target tracking and collective activity recognition”, ECCV 2012

  As mentioned above, recognition of actions involving multiple people in surveillance and sports is expected to have wide application. Patent Document 1 recognizes individual actions of a plurality of persons, but does not recognize an action performed by a plurality of persons in cooperation. Further, in Non-Patent Document 1, for recognition of individual behavior of a person, a spatiotemporal feature value created from a rectangular parallelepiped in spacetime is extracted, and a discrimination score obtained using a separately learned classifier is used. There is. As such, since the feature amount itself has a temporal width, there is a feature that the identification result becomes rough in time. Also, feature extractors, classifiers and graphical models are independent modules, which could not be consistently optimized throughout.

  According to one aspect of the present invention, an image processing apparatus includes acquisition means for acquiring moving images including time-series still images, detection means for detecting one or more objects for each still image from the moving images, In the still image, a feature amount extraction unit for extracting feature amounts corresponding to each of the objects from the still image, an object integration unit for integrating feature amounts corresponding to each of the objects in the still image, and Time series integration means for integrating the feature quantities of the integrated object for the time series still image, and the action of the object in the moving image is identified based on the feature quantity integrated for the time series still image And identification means.

  According to the present invention, feature quantities representing individual motions of a person appearing in each frame of a moving image are extracted, feature quantities of a plurality of people are integrated, and temporal integration is performed to make sense of individual actions of a plurality of people. Allows accurate identification of the actions to be attached.

It is a figure which shows an example of a camera arrangement | positioning, and an example of the still image image | photographed with two cameras. It is a figure which shows an example of the system configuration | structure at the time of recognition. It is a flow chart which shows an example of processing at the time of recognition. It is a figure which shows an example of a result of person detection, and an example of a result which sorted the detected person area. It is a figure which shows an example of the result of person detection in two flames | frames, and an example of the result of having sorted the detected person area | region. It is a figure which shows an example of the neural network structure used by the process at the time of recognition and learning. It is the figure which expand | deployed the neural network shown in FIG. It is a figure explaining integration of a person series and integration of a time series which are realized by controlling a neural network shown in FIG. Fig. 9 shows a neural network equivalent to the controlled neural network shown in Fig. 8; It is a figure which shows an example of the system configuration | structure at the time of learning. It is a flowchart which shows an example of the process at the time of learning. It is a figure which shows the neural network structure using the person area | region in a flame | frame, and coordinate data of a person. It is a figure which shows an example of the neural network structure used by the process at the time of recognition and learning. FIG. 7 is a diagram showing an example of a person detection result of two frames at the same time taken by two cameras and an example of a result of sorting the detected person area. It is a figure which shows an example of the neural network structure used by the process at the time of recognition and learning. It is a figure which shows an example of the system configuration | structure at the time of recognition (the 2). It is a figure which shows an example of the neural network structure used by the process at the time of recognition and learning. It is a figure which shows the control state of LSTM.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

First Embodiment
In the first embodiment, a method of recognizing an action label that can be added to the action of a plurality of players from a moving image will be described by taking a futsal as an example.

  FIG. 1 is a diagram for explaining a moving image assumed in the present embodiment. Reference numeral 102 in FIG. 1A denotes a futsal coat. Reference numerals 103 to 112 denote cameras. X, Y and Z indicate the X axis, Y axis and Z axis of three-dimensional coordinates (world coordinates) whose origin is defined as 113. As shown in FIG. 1A, it is assumed that a plurality of cameras are arranged around a futsal coat and a moving image is photographed by the cameras. FIG. 1B shows an example of one frame (still image) 201 of futsal moving image acquired by the camera 103. 202 indicates a ball, and 203 to 209 indicate persons AG, respectively. Thus, in the moving image, there are a plurality of persons (including players and referees) and a ball as objects. An example of one frame 301 acquired by the camera 104 is shown in FIG. Reference numeral 302 denotes a ball, and 303 to 311 denote persons AK, respectively. The ball 302 and the persons AE (303 to 307) in FIG. 1 (c) are the same objects or persons as the ball 202 and the persons AE (203 to 207) in FIG. 1 (b).

  Each of the cameras 103 to 112 is calibrated, and internal and external parameters of the cameras are acquired. That is, the stereo method can be applied to the image acquired by each camera, and the corresponding points can be projected on world coordinates set with the reference point of the coat as the origin. In this embodiment, the center mark of the coat is set as the origin 113, and a three-dimensional space having an X axis, a Y axis, and a Z axis is set as world coordinates. The position of the person or the ball on the world coordinates can be acquired in combination with the detection of the person or the ball appearing in the image.

  Also, in this futsal action recognition, it is assumed that action labels of five types of multi-category are recognized: pass, shoot, dribble, keep and clear. The recognition of the action labels (pass, shoot, dribble, keep, clear) dealt with here is not enough to recognize the individual's actions, and the actions that need to be comprehensively identified by the individual actions of multiple persons related to the ball. It is a label.

  FIG. 2A is a diagram showing a functional configuration of the behavior recognition apparatus 1000 which is the image processing apparatus of the present embodiment. The action recognition apparatus 1000 of the present embodiment includes a moving image acquisition unit 1001, a human object detection unit 1002, a human area extraction unit 1003, a human area sort unit 1004, an integrated control signal generation unit 1005, and an image feature quantity extraction unit 1006. Furthermore, it has a person series integration unit 1007, a time series integration unit 1008, and an action label identification unit 1009. The details of each of these functions will be described later with reference to FIG.

  FIG. 3A is a flowchart showing an example of processing at the time of recognition in the present embodiment. The outline of the entire process will be described using this flowchart.

  First, in S1001, the moving image acquisition unit 1001 acquires a frame sequence of a moving image composed of a plurality of still images. In S1002, the human object detection unit 1002 detects the position and size of each of the person and the ball appearing in the frame acquired in S1001. When there are a plurality of persons in the frame, a plurality of positions and sizes corresponding to the number of persons are detected. In S1003, the human area acquisition unit 1003 acquires a human area (hereinafter, simply referred to as a human area) based on the position and size of the human detected in S1002. In S1004, the person area sorting unit 1004 sorts the person area acquired in S1003 based on the position of the ball and the person detected in S1002. Details of the sorting will be described later.

  In S1005, the integrated control signal generation unit 1005 generates a signal for controlling a person series integration unit 1007 and a time series integration unit 1008 described later based on the number of persons present in the frame detected in S1002. In S1006, the image feature quantity extraction unit 1006 extracts an image feature quantity corresponding to the person area acquired in S1003. When there are a plurality of persons in the frame, the image feature amount is extracted for each area corresponding to the plurality of persons. Image feature quantities to be used will be described later. The extracted image feature quantity is referred to as a person feature quantity.

  In S1007, the person series integration unit 1007 performs an object integration process of integrating a plurality of person feature quantities corresponding to the number of people in the frame acquired in S1006. Details of this integration process will be described later. As a result of this processing, a plurality of person feature quantities are acquired. In S1008, the time-series integration unit 1008 performs processing to further temporally integrate the person feature quantities integrated in S1007 respectively corresponding to the plurality of frames acquired in S1001. Details of this integration process will be described later. As a result of this processing, a plurality of frames and a plurality of person feature amounts are acquired. In S1009, the action label identification unit 1009 identifies an action label based on the plurality of person feature quantities of the plurality of frames integrated in S1008. This is done frame by frame. The classifier to be used will be described later.

  Next, more specific contents of each process will be described according to the flowchart shown in FIG.

  In S1001, a moving image captured by a multi-camera arranged as shown in FIG. 1A is acquired. However, in the present embodiment, in the subsequent steps other than human object detection in S1002, a multi-viewpoint moving image is not used, and a plurality of frames captured by any one camera are used. In this embodiment, the camera assumes a resolution of Full HD (1920 × 1080 pixels) and a speed of about 30 frames per second, and obtains continuous 30 frames (for one second). However, even if a plurality of frames are acquired under different conditions, such as acquiring every few frames, acquiring with a slower camera, acquiring for a longer time, etc., if they differ by several times, this embodiment The functions realized by are not significantly compromised. In addition, although a moving image shot by a camera may be obtained directly, it may be stored in an external storage device, and a plurality of predetermined frames may be obtained therefrom.

  Next, in S1002, human detection and ball detection are performed on each frame of the multi-camera moving image acquired in S1001, and using the detection results for the multiple camera frames, the final positions of the human and the ball and the human area are displayed. get. For human detection and ball detection in one frame, a known object detection method such as AdaBoost may be used. In this case, it is also possible to use human detection, in particular a detector trained to detect a human face.

  Subsequently, by searching for the corresponding point from the detection result of one camera frame at a certain moment and the detection result of another camera frame, by applying the stereo method as described above, the world coordinates of the detection result The position above (in the real world) can be obtained. Correspondence point search may be performed using known techniques such as feature amounts of ORB, corner detection such as FAST, metrics such as Hamming distance, and approximate nearest neighbor search methods such as kd-tree. As a result of these, the positions of the ball and the face of each person are obtained as three-dimensional positions (X, Y, Z) on world coordinates. This process is performed for each frame acquired in S1001. As a result, the ball and the face position of each person are acquired for each frame.

  Next, in S1003, from the face position on world coordinates of each person acquired in S1002, an area (person area) on the frame of each person is acquired as a bounding box. The bounding box is a rectangular area specified by four parameters of the position (X, Y) and the width (W, H) on each frame. The bounding box is set so as to cover the person by giving a width of several meters in front and rear left and right with reference to the height of the person.

  FIG. 4A is a drawing in which results of human detection and ball detection are performed on the frame of FIG. 1B. Reference numeral 402 denotes a ball detection position, and reference numerals 403 to 409 denote person areas of the persons A to G, respectively. A bounding box is set for the person detection position, and the position of the ball is further indicated.

  Subsequently, the person area is cut out from the original frame (size: 1920 × 1080 pixels) and resized to a fixed size. Since the human area has various sizes, the human area of the original frame may be enlarged or reduced. Bicubic interpolation is applied for enlargement, and nearest neighbor interpolation is applied for reduction. Further, in the present embodiment, the size of the human area after resizing is 256 × 256 pixels, and the human area after resizing is hereinafter simply referred to as a human area. The above process is performed for each frame, and as a result, the person area for each frame is acquired.

  Next, in S1004, the person area acquired in S1003 is sorted in descending order based on the distance between the ball and the person area. Here, the distance is the Euclidean distance between the person and the ball obtained from the three-dimensional position on the world coordinates of the person and the ball acquired in the person object detection step S1002.

  FIG. 4B shows the results of resizing in S1003 and sorting in this step for one frame. Reference numerals 502 to 508 denote the resized person areas of the persons A to G, respectively. Human regions of various sizes in FIG. 4A are resized to uniform resolution, and sorted in descending order from the region 502 of the person G farthest from the ball to the region 508 of the closest person A. This process is performed for each frame, and as a result, the sorted person area for each frame is acquired. Furthermore, the sorting results for each frame are connected in frame order, and series data of the person area arranged in one dimension is acquired. At this time, even if the number of persons is different for each frame, they are linked in the order of frames. This series data is hereinafter referred to as a person area series.

  FIG. 5 shows an example in which person areas of two frames different in the number of persons are sorted and connected. In FIG. 5A, reference numeral 601 denotes a frame at time I corresponding to the first frame, and reference numerals 602, 603, and 604 denote person areas of persons A, B, and C, respectively. In FIG. 5B, 701 is a frame at time II corresponding to the second frame, and 702 and 703 are person areas of persons A and B, respectively. The person area 602 in FIG. 5 (a) and the person area 702 in FIG. 5 (b) correspond to the same person A, and the same person B as the person area 603 in FIG. 5 (a) and the person area 703 in FIG. It corresponds to In FIG. 5A, there are three persons A, B, and C, and in FIG. 5B, two persons A and B exist. In this case, the person area is sorted based on the distance for each frame, and the two person areas of the second frame are connected after the three person areas of the first frame. As a result, a person area sequence shown in FIG. 5 (c) is obtained.

  In FIG. 5C, reference numerals 802, 803, and 804 denote a person C and a person B and a person area of a person A in the first frame, respectively, and 805 and 806 denote a person area of the person B and a person A in the second frame. The human region series (802 to 806) is thus one-dimensional series data. Also, although the sorting criterion is the Euclidean distance between the person and the ball here, sorting may be performed on another criterion. For example, when a uniform person ID is given to all the frames for the person detection result, the IDs may be sorted in ascending order.

  In the present embodiment, the processes of S1006-S1009 are implemented by a network structure of a neural network combining a convolutional neural network, a recursive neural network, and a softmax discriminator. In the following, a convolutional neural network is also referred to as CNN, and a recurrent neural network is also referred to as RNN. Further, in the present embodiment, a long short-term memory (LSTM) which is a type of RNN is used as the RNN. In S1005, a signal for controlling a recursive neural network is created.

  An outline of a network structure of a neural network that executes the steps of S1006-S1009 is shown in FIG. First, the details of S1006-S1009 will be described using FIG. Next, the control signal generated in S1005 will be described based on a more specific example.

The network structure 901 of FIG. 6 has modules of an input 902, CNN 903, LSTM1 (904), LSTM 2 (905), FC 906, and Softmax 907. The image feature extraction processing performed in S1006 is realized by the CNN 903. The CNN 903 is a neural network composed of multiple layers used for image recognition. The lower layer CNN's middle layer is known to extract primitive geometric features such as lines, points and patterns in low order, and complex features in high order parts and objects that combine parts. . Also, it is disclosed in the following paper by Donahue et al. That high-precision classification can be performed by applying the feature quantities of the CNN middle layer already trained on large-scale data to another classification task.
J Donahue, Y Jia, O Vinyals, J Hoffman, N Zhang, E Tzeng, T Darrell, T Darrell, "DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition", arXiv 2013
In S1006, each person area of the person area series created in S1004 is input to the CNN (903), and an image feature amount is acquired. Here, as the feature amount of CNN, the feature amounts may be acquired from a plurality of intermediate layers, or only the feature amounts of some intermediate layers may be used.

Next, person series integration processing in S1007 is realized by LSTM 1 (904). Long short-term memory (LSTM) is a type of recursive neural network. In a recursive neural network, in general, the current input vector _xt and the hidden state vector _ht-1 one period before are input to the network, and the current hidden state vector _ht is calculated and output. LSTM is a neural network that controls input, forgetting, and output internally. According to the notation disclosed in the paper of Donahue et al. Below, it has (input gate, forget gate, output gate, input modulation gate) and cell unit. Then, by controlling the input x _t , forgetting and output according to the input at a certain time and the hidden state h _t one period before, it is possible to identify short and long complex time series patterns.
Donahue J. et al. , “Long-term recurring convolutional networks for visual recognition and description”, CVPR 2015
Input _{x t,} hidden states _{h t,} cell _{c t} and the input, forgetting, gate output to control the output _{i t,} f t, updates _{o t} are shown in the following formula (1).
i _t = σ (W _xi x _t + W _hi h _t-1 + b _i )
_{f t = σ (W xf x} t + W hf h t-1 + b f)
o _t = σ (W _xo x _t + W _hO h _t-1 + b _o )
_{g t = tanh (W xc x} t + W hc h t-1 + b c)
c _t = f _t · c _{t -1} + i _t · g _t
h _t = o _t · tan h (c _t ) (1)
Here, sigma () is the sigmoid function, tanh () is hyperbolic tangent function, g _t is input to the cell, - represents the product of each element. In addition, W _xi , W _hi , W _xf , W _hf , b _i , b _f , b _o , b _c are weights and biases of the input gate, the forget gate, the output gate, and the input modulation gate.

  The LSTM has a state in which two external controls are possible (hereinafter referred to as a control state). In this embodiment, these two control states are referred to as "update" and "reset".

In the present embodiment, the case of functioning as the equation (1) is referred to as "update" of LSTM. And "reset" is realized by forcing the output of the forge gate to f _t = 0 by an external signal.
i _t = σ (W _xi x _t + W _hi h _t-1 + b _i )
o _t = σ (W _xo x _t + W _hO h _t-1 + b _o )
_{g t = tanh (W xc x} t + W hc h t-1 + b c)
c _t = i _t · g _t
h _t = o _t · tan h (c _t ) (2)
The control state is

  From the person area sequence created in S1004, the feature amount (person feature amount) is extracted via the CNN 903 and input to the LSTM1 (904). A person feature amount is recursively input to the LSTM 1 (904), and the hidden state is updated. Further, the control state is set to "reset" at the beginning of the series for each frame, and switched to "update" otherwise, and the person feature quantities for each frame are integrated. The switching of the control state will be described later based on more specific cases.

Next, the time series integration process in S1008 is realized by LSTM2 (905). In addition to "update" and "reset", LSTM2 (905) has another control state, "hold". To update LSTM of formula (1), by a forget Gate, input The Gate, Forcing the values of _{output gate f t = 1, i} t = 0, o t = 1,
i _t = 0
f _t = 1
o _t = 1
_{g t = tanh (W xc x} t + W hc h t-1 + b c)
c _t = f _t · c _{t -1} + i _t · g _t
h _t = o _t · tan h (c _t ) (3)
In equation (3), regardless of the input to LSTM,
c _t = c _{t -1}
h _t = tan h (c _{t -1} )
It becomes. This is the state hidden state h _t, cell c _t does not change for any input. In this embodiment, this is set as "holding" as the third control state of LSTM.

In this process, while LSTM 1 (904) performs integration of person feature quantities, the state (hidden state h _t , cell c _t ) is not changed by switching LSTM 2 (905) to “hold”. Then, at the end of the integration of the person feature amounts for each frame, by switching to "update", the person feature amount recursively integrated in the frame is received as an input for each frame, and the state is updated. Do integration. The switching of the control state will be described later based on more specific cases.

Finally, the action label identification process in S1009 is realized by FC 906 and Softmax (907). The FC 906 inner multiplies the weight matrix with respect to the hidden state h _t of the LSTM 2 (905), and obtains scores (inner product score) of the number corresponding to the number of labels of the action label. Furthermore, in Softmax (907), the inner product score is converted to a probability (a real number of 0 or more and 1 or less) by the Softmax function. The identification score expressed by the probability corresponding to the action label is obtained by the above processing.

  The details of each process of the image feature extraction process (S1006), the person series integration process (S1007), the time series integration process (S1008), and the action label identification process (S1009) have been described above with reference to FIG. . Next, control signals generated in the integrated control signal generation process (S1005) will be described based on more specific cases. In S1005, in order to realize integration and time-series integration of individual motions of a plurality of persons presented in the present embodiment, LSTM1 (904) responsible for person series integration processing (S1007) and LSTM2 responsible for time series integration processing (S1008) 905) and creating a control signal.

  FIG. 7 shows an example of the case where the person area sequence created in S1004 is input to the neural network of the network structure shown in FIG. Here, 1002 is a person area of person C at time I, 1003 is a person area of person B at time I, 1004 is a person area of person A at time I, and 1005 is a person area of person B at time II , 1006 is the person area of the person A at time II. These person areas are the same as the person areas in FIG. 5 (c). 1007 is CNN, 1008 is a first layer LSTM, 1009 is a second layer LSTM, 1010 is FC, and 1011 is Softmax. The modules 1007 to 1011 are the same as the modules 903 to 907 in FIG.

  FIG. 7 is a diagram in which the recursive neural network is expanded in the time direction, and the vertical line 1012 represents the propagation path of the signal and error between units, and the horizontal line 1013 is the signal and error in the time direction. Represents a propagation path. The person area series (1002 to 1006) created in S1004 are input to the CNN 1007 in order from the left of the series.

  With respect to person area series (1002 to 1006 in FIG. 7) extending over a plurality of frames, integration of person series and integration of time series by two LSTMs (1008, 1009) are realized. Therefore, the control state (“update”, “hold”, “reset”) of the LSTM is switched by the control signal. FIG. 18A shows a control state performed on the person area series 1002 to 1006 in FIG. 7.

  At the beginning of the sequence (n = 1), reset two LSTMs. The LSTM 1 integrates human feature quantities in a frame, and the LSTM 2 integrates human feature quantities integrated in the LSTM 1 for each frame. For this process, n = 2 makes LSTM1 "update", LSTM2 "reset", n = 3 makes LSTM1 "update", and LSTM2 "reset". As a result, integration of the person feature amount of the first frame by LSTM1 and the person feature amount of the first frame integrated by LSTM1 by LSTM2 are received as n = 3 only. Although n = 1 and n = 2 set LSTM2 to "Reset", it is important to integrate the first person with n = 3 and "Reset", n = 1 and n = 2 to control LSTM 2 The state may be anything. Next, at n = 4, "re-set" LSTM1 again and hold LSTM2. By setting both LSTM1 and LSTM2 to “update” at n = 5, the person feature of the second frame is integrated by LSTM 1 and the person feature of the integrated second frame is received as n = 5 by LSTM 2 . As a result, in the case of LSTM2, the input at n = 3 (first frame) and the input at n = 5 (second frame) are integrated (integration in time direction).

  FIG. 8 shows propagation paths (1123) of signals and errors when the control of FIG. 18 (a) is performed on the network of FIG. The person area 1102 to 1106 is the same as 1002 to 1006 in FIG. CNN, LSTM1, LSTM2, FC, Softmax are the same as CNN 1007, LSTM1 (1008), LSTM 2 (1009), FC 1010, Softmax 1011 in FIG. As for LSTM1 and LSTM2, “update” is indicated by a white background rectangle (1116 etc.), “reset” by a hatched pattern rectangle (1109 etc.), and “hold” by a dot pattern rectangle (1114 etc.).

  The reference numeral 1109 denotes "reset" of the LSTM 1 at n = 1 in FIG. 18A, and 1112 denotes the "reset" of the LSTM 2 at n = 1 in FIG. 18A. Reference numeral 1110 denotes “update” of LSTM 1 at n = 2 in FIG. 18A, and reference numeral 1113 denotes “reset” of LSTM 2 at n = 2 in FIG. 18A. 1124 is “update” of LSTM 1 at n = 3 in FIG. 18A, and 1116 is “reset” of LSTM 2 at n = 3 in FIG. 18A. In FIG. 18A, 1111 is a “reset” of the LSTM 1 at n = 4, and 1114 is a “hold” of the LSTM 2 at n = 4 in FIG. 18A. 1125 is "update" of LSTM 1 at n = 5 in FIG. 18A, and 1117 is "update" of LSTM 2 at n = 5 in FIG. 18A.

  In the LSTM 1, the person feature value of the first frame (time I) is integrated by “reset” 1109, “update” 1110, and “update” 1124. Next, the internal state is temporarily reset by "Reset" 1111 and "Update" 1125 is performed again, so that only the person feature value of the second frame (time II) is integrated without propagating the signal and error at 1127. . In LSTM2, “reset” 1112, “reset” 1113, and “reset” 1116 are made to receive signals only from LSTM 1 (1124) in which the person feature value of the first frame is integrated when n = 3. Next, “hold” 1114 is set, the internal state is made unchanged, and “update” 1117 is set again. By doing this, the signal of LSTM2 (1116) that receives the integrated human feature quantity of the first frame and the signal of LSTM1 (1125) where the human feature quantity of the second frame is integrated are integrated in the time direction I do.

  The internal state of LSTM 2 of each frame is propagated to FC and Softmax (1121, 1122), and the identification score is output. Softmax's hatched pattern rectangles (1118, 1119, 1120) indicate "ignore" which does not evaluate the error, but this will be described in detail in the process of learning.

  FIG. 9 shows a diagram of a network having a structure equivalent to the network when the control shown in FIG. 8 is performed. Person areas 1202 to 1206 are the same as 1002 to 1006 in FIG. Further, CNN (1207), LSTM1 (1208, 1209), LSTM2 (1210), FC (1211), and Softmax (1212) are the same as the elements of the same names in FIG.

  Person C area 1202, person B area 1203 and person A area 1204 in the first frame (time I) are integrated by LSTM1 through CNN, and the internal state of LSTM1 obtained by integrating the person feature quantities of the first frame is LSTM2 ( 1210). Subsequently, the internal state of LSTM1 is reset, and person B area 1205 and person A area 1206 of the second frame (time II) are newly integrated by LSTM1, and the internal state of LSTM 1 is obtained by integrating the person feature value of the second frame. It is input to LSTM2 (1213). In LSTM2, the internal state of LSTM2 in the first frame and the input from LSTM1 are received, and the information in the first frame and the information in the second frame are integrated. The internal state of LSTM2 of each frame outputs the identification score of the action label by Softmax via FC. As described above, the network shown in FIG. 9 is executed by the network shown in FIG. 7 and the control shown in FIG. 18 (a).

  The above is the process at the time of recognition, which recognizes the motion of 30 frames obtained in the moving image acquisition step S1001 based on the motion of a plurality of people. After this, the process at the time of recognition may be similarly performed on the next 30 frames, but the process at the time of recognition may be performed such that several frames overlap. That is, after the identification from the first frame to the 30th frame of the futsal moving image is performed by a certain recognition process, the 15th to 45th frames may be processed next. In that case, multiple results of a certain frame are averaged to obtain the final result. By redundantly recognizing in this way, a frame is recognized multiple times by different sequences, and the result is more robust.

  Next, a learning method of the action label identification unit used in the person series integration unit used in the person series integration process, the time series integration unit used in the time series integration process, and the action label identification process will be described.

  FIG. 10 is a diagram showing a functional configuration of the learning device 5000 in the present embodiment. The learning device 5000 includes a person area extraction unit 5001, a person area sorting unit 5002, an integrated control signal learning label generation unit 5003, and a parameter parameter optimization unit 5004. The learning device 5000 further includes a learning data holding unit 5005 and a network parameter holding unit 5006 as storage units.

  FIG. 11 is a flowchart showing an example of processing related to learning in the present embodiment. Here, the outline of each process and the function of each unit shown in FIG. 10 will be described.

  In S5001, the person area extraction unit 5001 extracts the area of the person present in the frame making up the moving image from the moving image and the person detection result stored in the learning data holding unit 5005. This process is the same person area extraction process as S1003 described in the recognition process of this embodiment. The details of the data stored in the learning data storage unit 5005 will be described later.

  In S5002, the person area set in S5001 is uniformly resized, and sorting is performed according to a certain standard. Since this process is the same as the person area sorting process of S1004 described in the process at the time of recognition of the present embodiment, the detailed description will be omitted.

  In S5003, the integrated control signal learning label generation unit 5003 generates a control signal and a learning label based on the number of persons present in the frame detected in S5001 and the action label attached to the frame. These are used to learn the parameters of the neural network used in the image feature quantity extraction unit 1006, the person series integration unit 1007, the time series integration unit 1008, and the action label identification unit 1009 used in the process of recognition.

  In S5004, optimization of parameters of the neural network is executed with the person series created in S5002 as an input and the learning label created in the integrated control signal learning label creating step S5003 as a target value.

  The above S5001 to S5004 are repeated by the number N of iterations set in advance. The final parameters and the parameters in the middle of the iteration are stored in the network parameter storage unit 5006.

  Next, more specific contents of integrated control signal learning label creation (S5003) and parameter update (S5004) which are different from the processing at the time of recognition in the flowchart shown in FIG. 10 will be described. Also, data stored in the learning data storage unit 5005 will be described.

  The learning data holding unit 5005 stores a moving image and a correct answer label (action label) corresponding to the action label of the futsal recognized in the present embodiment, a person detection result of each frame in the moving image, and a ball detection result. The action labels are "pass", "shoot", "dribble", "keep" and "clear". It is assumed that the moving image is composed of arbitrary plural frames, and the correct answer label is attached to each frame.

  In S5001, 30 consecutive frames are selected at random from a moving image composed of an arbitrary number of frames to which a certain action label is attached, and a person area is extracted using a person detection result of each frame. Extraction of a person area is processing similar to S1003 in processing at the time of recognition.

  In S5003, in addition to creation of the control signal of LSTM performed in S1005 in the processing at the time of recognition of this embodiment, creation of a learning label for learning CNN, LSTM, and Softmax discriminator is performed. The generation of the control signal of LSTM is the same processing as the processing in S1005, and thus the detailed description is omitted here. As a result of this process, an integrated control signal similar to that shown in FIG. 18A is created.

  The Softmax discriminator is provided with a learning label attached to the moving image at the time of generation of a signal for “updating” LSTM 2 among the generated control signals of LSTM. Otherwise, set the "ignore" label on the learning label. The "ignore" label is a special label that prevents Softmax's loss function from being evaluated if it is set.

  As shown in the upper part of FIG. 8, when the softmax discriminator inputs the person area (1104) of person A at time I and the person area (1106) of person A at time II to execute time series integration, learning labels (1121, 1122). Otherwise, the "ignore" label is given (1117).

  In step S5004, the parameter optimization unit 5004 performs parameter optimization of the CNN, LSTM, and Softmax discriminators corresponding to the image feature quantity extraction unit 1006, person series integration unit 1007, time series integration unit 1008, and action label identification unit 1009. .

Here, according to the integrated control signal generated in S5003, the LSTM is appropriately controlled to any of the control state of reset, hold, or normal state, and the learning label generated simultaneously is given to the Softmax discriminator. Parameter optimization is performed by applying the Back Propagation Throut Time (BPTT) method described in Graves et al.
A. Graves and J. Schmidhuber. “Framewise Phoneme Classification with Bidirectional LSTM Networks”. In Proc. International Joint Conference on Neural Networks IJCNN'05
The Softmax discriminator uses the cross entropy error as a loss function, evaluates the loss function when a label other than the neglected label is given, and calculates the error. In the LSTM, three control states of reset, hold, and normal state are switched by a control signal, and parameters are updated in cases other than hold. In CNN, fine tuning is performed in this process using large-scale data and learned parameters as initial values. However, even if the CNN fine tuning is not performed, the functions realized by the present embodiment are not significantly impaired. Therefore, CNN fine tuning may be omitted. When omitted, the parameters of CNN are fixed to the learned parameters for large scale data.

  In the present embodiment, a method of identifying an action label to be added to by individual actions of a plurality of persons assuming recognition of play in soccer, futsal, rugby in sports has been described.

  In these sports, for example, identification of movement such as pass turning and tackle is not sufficient for individual player's movement alone, but it is not necessary to always handle the movement of all players, and individual movement of several players related to the ball It can be said that it can be recognized by the framework that deals with In the case where such action recognition that is meaningfully performed by a plurality of cooperative actions is performed by video analysis, it is necessary to have a framework capable of coping with the change in the number of people for each frame of the moving image, as in Non-Patent Document 1. In addition, assuming applications in soccer, futsal and rugby, it is necessary in principle to be able to recognize with frame-based time resolution in principle, and to have a framework in which individual motions of multiple people can be integrated to recognize overall behavior. Furthermore, by optimizing the whole of each part that realizes feature value extraction, integration of individual actions, integration of time series, and identification of action labels, it can be expected to improve accuracy beyond optimizing each means individually. .

  As described above, according to the present embodiment, the behavior recognition apparatus 1000 integrates feature amounts representing individual motions of a plurality of persons, and further integrates them temporally to identify a behavior label. In this way, in response to the change in the number of people in each frame of the moving image, recognition is performed in frame units, and furthermore, individual actions of a plurality of people are integrated to enable identification of the entire action label. Furthermore, by optimizing the entire neural network that realizes feature value extraction, integration of individual actions, integration of time series, and identification of action labels, it is possible to identify action labels with high accuracy.

(Form 1 of derivation of Embodiment 1)
In the first embodiment, the method of acquiring the image feature amount from the person area of the still image constituting the moving image and identifying the action label has been described. However, information available in the problem setting in the first embodiment exists in addition to the image feature of the person area. For example, in the first embodiment, detection of the ball position is performed in human object detection in S1002. Therefore, an area of an arbitrary size centered on the ball (hereinafter referred to as a ball area) may be added to the person area and used.

  Further, in the human object detection in S1002, the three-dimensional positions of the human and the ball are acquired. The coordinate values of the person and the ball may be used in connection with the image feature amount of the person area or the ball area. When using the ball area in addition to the person area, the ball area may be added to the end of the person area series of the person detection result of the frame taken by a certain camera at a certain time. For example, a series is created in which the ball area is added to the end of the person area sorting result (501) shown in FIG. 4 (b). Similarly, a ball area may be added to the end of the person area series for frames at different times to create an image series of partial areas to be input to the network.

In the first embodiment, as shown by 1002 to 1006 in FIG.
{Time I person C area, time I person B area, time I person A area, time II person B area, time II person A}
It is. Here, a sequence “{x1, x2, x3,..., Xn}” surrounded by braces indicates sequence data to be input to the network. When using the ball area in addition to the person area, the input to the network is
{Time I person C area, time I person B area, time I person A area, time I ball area, time II person B area, time II person A, time II ball area}
It becomes.

  If the ball area is a frame not detected, the ball area may be ignored to create a sequence of image areas. Alternatively, interpolation processing such as linear interpolation may be performed from frames before and after successful ball detection to estimate the ball position, or a moving image including a frame in which no ball is detected may be excluded from the recognition target.

  In various sports, such as futsal, soccer, rugby, balls generally move faster than people. Therefore, in the case of using a general frame rate moving image for ball detection, detection failures often increase. And in simple interpolations, such as linear interpolation, it is possible that an error becomes large. Therefore, in order to reduce the influence of an error caused by interpolation or the like, the ball area to be extracted may be a wider area than the human area. Specifically, in the first embodiment, the person area is set to an area of about 2 to 4 m, while the ball area is acquired from a wide range of about 5 to 10 m. In this way, even if there is an error in ball detection, the probability that the ball is included in the ball area increases.

(Modified form 2 of Embodiment 1)
In the first embodiment, the method of acquiring the image feature amount from the person area of the still image constituting the moving image and identifying the action label has been described. Furthermore, in the first modification of the first embodiment, the method of connecting the ball area to the person area and identifying the action label from the ball and the area of the person has been described. In this embodiment, a method of using metadata associated with an image in addition to the image will be described.

  As already described in the first embodiment, the ball and the person obtain coordinate values in three dimensions by object detection and stereo method. Therefore, the three-dimensional coordinate values of the person and the ball can be used together with the image feature amount of the person area or the ball area. In such a case, the image feature quantities and their coordinate values may be connected at the subsequent stage of the image feature extraction unit 1006 implemented by CNN in the first embodiment, and may be input to the person series integration unit 1007.

  FIG. 12 shows an example of a network structure in the case of using an image area of a person and three-dimensional coordinate values. 1702, 1704 and 1706 are the same as 1002, 1003 and 1004 in FIG. Reference numerals 1703, 1705, and 1707 denote coordinate data of the person A, the person B, and the person C, respectively. CNN 1708, LSTM 1 (1710), LSTM 2 (171 1), FC 1712 and Softmax 1713 are the same as CNN 90), LSTM 1 (904), LSTM 2 (905), FC 906 and Softmax 907 in FIG. Concat 1709 is a connection module.

  Here, the coordinate data (1703, 1705, 1707) may be calculated using the distance from the ball or the distance from the camera in addition to the position (X, Y, Z) on the three-dimensional coordinate of the person. Furthermore, other metadata such as the speed and acceleration calculated using data of previous time and team ID may be used. The connection module 1709 is a module that connects the image feature quantity extracted by the CNN 1708 and the coordinate data. This concatenated data is input to LSTM 1 (1710). The connection module 1709 may simply connect two feature quantities in this way, but may inner-multiply a weight matrix to reduce the dimensions to lower dimensions. In this case, FC is added between Concat 1709 and LSTM 1 (1710), and learning is performed in the same manner as the procedure described in the learning process of the first embodiment.

  In the first embodiment, the coordinate data is acquired by applying the stereo method to the object detection result for the multi-viewpoint image of the multi-camera on which the camera calibration has been performed. The coordinate data may be acquired by attaching a GPS (Global Positioning System) device to the player.

(Form 3 of derivation of Embodiment 1)
In the first embodiment, the method of integrating the person series and integrating the time series to identify the action label has been described. At that time, two series of integration were performed in the order of first integrating the person series and then integrating the time series, but the order is not limited to this. That is, time series integration may be performed first, and then person series integration may be performed.

In this case, the time when the person appears for each person is sorted to create a person area series. As shown in FIGS. 5A and 5B, consider again the case where there are three persons A, B and C at time I and two persons A and B at time II. At this time, person C is present only at time I, person B is present at both times I and II, and person A is also present at both times I and II. become.
{Time I person C area, time I person B area, time II person B area, time I person A area, time II person A}
Here, the order of persons was sorted in descending order of the distance from the ball (persons C, B, A), and the times were sorted in ascending order (time I, II). This is input to the input (902) of the network already shown in FIG. 6, and LSTMs 11 and 2 (904 and 905) are controlled to be integrated in order of time series integration and person series integration. The control states of LSTM1 and LSTM2 at this time are as shown in FIG.

  Here, with n = 1, in LSTM1, “time reset” is performed in order to integrate the time I person C area, and “reset” is also performed in LSTM2. Since the person C does not exist at time II, in LSTM 2, the first person is integrated at n = 1. The next person B is integrated with n = 2 and n = 3. Therefore, LSTM1 is "reset" at n = 2 and LSTM2 is "held", whereby the person B area at time I is integrated in LSTM1. By “updating” both LSTM1 and LSTM2 with n = 3, the person B at time II is integrated in LSTM1, and the result of integrating the person B area in time I and II is received in LSTM2 and the internal state is updated . Next, similarly, “n” resets LSTM1 with n = 4, and “holds” LSTM2, thereby integrating the person A area at time I with LSTM1. By “updating” both LSTM1 and LSTM2 at n = 5, the person A at time II is integrated in LSTM1, and the result of integrating the person A area in time I, II is received in LSTM 2 and the internal state is updated. .

  FIG. 13 shows propagation paths (1411) of signals and errors when the person area sequence is input to the network of FIG. 6 and the control of FIG. 18 (b) is performed. Reference numeral 1402 is the same as the time I person A area shown at 802 in FIG. 5C. Reference numeral 1403 is the same as the time I person B area shown at 803 in FIG. 5C. 1404 is the same as the time II person B area shown in 805 of FIG. 5C. 1405 is identical to the time I person A area shown at 804 in FIG. 5C. 1406 is the same as the time II person A area shown at 806 in FIG. 5C. CNN, LSTM1, LSTM2, FC, Softmax are the same as CNN (903), LSTM1 (904), LSTM2 (905), FC (906), Softmax (907) in FIG.

  As for LSTM1 and LSTM2, “update” is indicated by a white background rectangle (1409), “reset” is indicated by a hatched pattern rectangle (1407), and “hold” is indicated by a dot pattern rectangle (1408). Also, the shaded pattern rectangle (1410) indicates Softmax's "ignore", and the white background rectangle (1412) indicates Softmax's "normal operation" (an operation that is not "ignore"). By the control of FIG. 18B, the signal and the error are propagated as 1411, and time series integration and person series integration are realized in this order.

  By carrying out as described above, integration of person series and time series can be performed in any order.

Second Embodiment
In the present embodiment, a method of performing action recognition in the case where the same person is photographed by a plurality of cameras will be described with respect to futsal moving images photographed by a plurality of cameras. At that time, integration of three types of information: integration of the same person area of a plurality of cameras (viewpoint integration), integration of person feature quantities representing individual motions of a plurality of persons (object integration), and integration of time series I do.

  Similar to the first embodiment, the action labels to be identified in this embodiment are five types of action labels: “pass”, “shoot”, “dribble”, “keep”, and “clear”.

  Further, in the present embodiment, as in the first embodiment, futsal moving images photographed by a plurality of cameras arranged around the futsal court are used. FIG. 1 is a diagram already described, but is a diagram for explaining an example of this camera arrangement and an example of one frame photographed by two cameras.

  FIG. 2B is a diagram showing a functional configuration of the behavior recognition apparatus 2000 described in the present embodiment. The action recognition apparatus 2000 according to this embodiment includes a multi-camera moving image acquisition unit 2001, a human object detection unit 2002, a human area extraction unit 2003, a human area sort unit 2004, an integrated control signal generation unit 2005, and an image feature quantity extraction unit 2006. . Further, it has a camera series integration unit 2007, a person series integration unit 2008, a time series integration unit 2009, and an action label identification unit 2010. Details of each of these functions will be described below with reference to FIG.

  FIG. 3B is a flowchart showing an example of processing at the time of recognition in the present embodiment.

  In S2001, the multi-camera moving image acquisition unit 2001 acquires a frame sequence of moving images composed of a plurality of still images captured by a plurality of cameras. The human object detection in S2002 and the human area extraction in S2003 are the same processes as S1002 and S1003 at the time of recognition processing in the first embodiment, and therefore the description thereof is omitted.

  The person area sorting process (S2004), the integrated control signal creation process (S2005), and the image feature quantity extraction process (S2006) are the same processes as S1004-1006 at the recognition process in the first embodiment. However, since there is a difference, the difference will be described together with other processing.

  In step S2007, the camera series integration unit 2007 integrates the person areas of the same person captured by a plurality of cameras.

  The person series integration process (S2008), the time series integration process (S2009), and the action label identification process (S2010) are the same processes as S1007-1009 at the recognition process in the first embodiment. However, since there is a difference, the difference will be described together with other processing.

  Next, more specific contents of each process will be described according to the flowchart shown in FIG.

  A multi-camera moving image acquisition step S2001 acquires a multi-view moving image using a multi-camera arranged as shown in FIG. 1 (a). It is assumed that the videos of each camera are synchronized. 1 (b) and 1 (c) show the camera 103 (FIG. 1 (a); hereinafter referred to as camera 1) and the camera 104 (FIG. 1 (a); hereinafter referred to as camera 2) capturing the same moment as described above. It is assumed that such synchronized frames are acquired from each camera.

In the person area sorting step S2004, the person area in each frame of the multi-view moving image acquired in the multi-camera moving image acquisition step S2001 is sorted.
In this process, when the same person is photographed by a plurality of frames and cameras, for example, when person A is photographed by camera 1, camera 2, and camera 4 of frame I, the person area of person A is Sequences arranged in numerical order,
{Frame I Person A Camera 1, Frame I Person A Camera 2, Frame I Person A Camera 4}
As a camera series.
Similarly, the person B is photographed with the camera 1 and the camera 2,
{Frame I Person B Camera 1, Frame I Person B Camera 2}
If a camera series is obtained, the person series is a series of nested camera series,
{{Frame I Person A Camera 1, Frame I Person A Camera 2, Frame I Person A Camera 4}, {Frame I Person B Camera 1, Frame I Person B Camera 2}}
It becomes.
Furthermore, in the frame II, when the person A is photographed by the cameras 1 and 2 and the person B is photographed by the cameras 2 and 3, the time series is a series of nested camera series and person series,
{{{Frame I Person A Camera 1, Frame I Person A Camera 2, Frame I Person A Camera 4}, {Frame I Person B Camera 1, Frame I Person B Camera 2}}, {{Frame II Person A Camera 1 , Frame II Person A Camera 2}, {Frame II Person B Camera 2, Frame II Person B Camera 3}}}
It becomes.
Sorting of the person area is performed by arranging the nested time series created in this way in one dimension.
{Frame I Person A Camera 1, Frame I Person A Camera 2, Frame I Person A Camera 4, Frame I Person B Camera 1, Frame I Person B Camera 2, Frame II Person A Camera 1, Frame II Person A Camera 2, Frame II Person B Camera 2, Frame II Person B Camera 3}
FIG. 14 shows results of human detection (FIGS. 14A and 14B) in two frames at the same time taken by the cameras 1 and 2 and results of sorting of human regions (FIG. 14C). Here, reference numeral 1302 denotes a ball detection result on the frame of the camera 1, and 1303-1309 respectively denote human regions of the persons A to G on the frame photographed by the camera 1. Reference numeral 1402 denotes a ball detection result on the frame of the camera 2, and reference numerals 1403 to 1413 respectively denote person regions of the persons A to K on the frame photographed by the camera 1. The ball 1302 photographed with the camera 1 and the persons AE (1303-1307) and the ball 1402 photographed with the camera 2 and the persons AE (1403-1407) are the same object and person.

At this time, the nested person series is
{{Person G camera 1}, {person K camera 2}, {person J camera 2}, {person F camera 1}, {person J camera 1}, {person H camera 2}, {person E camera 1, person E camera 2}, {person D camera 1, person D camera 2}, {person C camera 1, person C camera 2}, {person B camera 1, person B camera 2}, {person A camera 1, person A camera 2}}
It becomes. The sorting result of arranging the nested person series in one dimension is 1504 to 1519 in FIG. That is,
{Person G camera 1, person K camera 2, person J camera 2, person F camera 1, person J camera 1, person H camera 2, person E camera 1, person E camera 2, person D camera 1, person D camera 2 , Person C camera 1, person C camera 2, person B camera 1, person B camera 2, person A camera 1, person A camera 2}
It becomes.

  Here, as in the first embodiment, each person is sorted based on the distance from the ball, but as described above, when a uniform person ID is obtained in the previous frame, it is sorted according to the order of the person ID It is also good.

  Next, integrated control signal creation step S2005 generates control signals used in the steps of camera series integration step S2007, person series integration step S2008, and time series integration step S2009. The camera series integration unit 2007, the person series integration unit 2008, and the time series integration unit 2009 are controlled by the control signal.

  In the first embodiment, the person series integration unit 1007 and the time series integration unit 1008 are realized by two layers of LSTM. In this embodiment, a camera series integration unit 2007, a person series integration unit 2008, and a time series integration unit 2009 are realized by three layers of LSTM (LSTM1, LSTM2, and LSTM3) each having one layer.

In each layer of LSTM, nested three-hierarchical series will be integrated in each hierarchy. That is, the nested sequence of the person area detected in two frames of one camera and two cameras shown in FIG.
{{{Person G camera 1}, {person K camera 2}, {person J camera 2}, {person F camera 1}, {person J camera 1}, {person H camera 2}, {person E camera 1, 1 Person E camera 2}, {person D camera 1, person D camera 2}, {person C camera 1, person C camera 2}, {person B camera 1, person B camera 2}, {person A camera 1, person A Camera 2}}}
It becomes. In this case, the control state of LST M1 that integrates the camera series is controlled to “reset” at the beginning of the camera series and “update” at the time of integration,
{Reset, Reset, Reset, Reset, Reset, Reset, Reset, Update, Reset, Update, Reset, Update, Reset, Reset, Update, Reset, Reset, Update}
The control signal is generated to be in the state series.
In addition, in LSTM2 in which person series are integrated, reset at the beginning of the person series, update at integration, and "hold" if the camera series one layer lower is other than the last element. That is,
{Reset, Update, Update, Update, Update, Update, Update, Update, Update, Hold, Update, Hold, Update, Hold, Update, Hold, Update, Hold, Update}
It becomes.

Similarly, in LSTM3 that integrates in time series, the last element is reset in order to "reset" at the beginning of time series. Anything other than that may be used, but for convenience, "reset". That is,
{Reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset, reset}
It becomes.

  A network in which the control state is actually switched with respect to the sequence of the person area in FIG. 14C is illustrated in FIG. The rectangles described as CNN, LSTM1, LSTM2, LSTM3, FC, and Softmax in the figure are CNN, and three-layer LSTM and Softmax classifiers, respectively. A hatched rectangle (1618) represents LSTM "reset", a dotted rectangle (1619) represents "hold", and a white background rectangle (1620) represents "update". A shaded pattern rectangle (1621) indicates the Softmax discriminator "ignore", and a white background rectangle (1622) indicates the Softmax discriminator normal operation (non-ignore state).

  The subsequent steps (image feature quantity extraction step S2006, camera series integration step S2007, person series integration step S2008, time series integration step S2009, action label identification step S2010) are realized in the same manner as in the first embodiment. . That is, it is realized by a neural network combining CNN, LSTM, and Softmax classifiers. The network structure is as shown in FIG. That is, the camera series, the person series, and the time series are integrated if the same recognition processing as in the first embodiment is executed with the structure and the control state shown in FIG. The identification result of the action label is obtained.

By performing as described above, it is possible to recognize an action label in which multimodal sequence information (in this embodiment, a camera sequence, a person sequence, and a time sequence) are integrated.
In this embodiment, the camera series, the person series, and the time series are integrated in this order, but it is also possible to integrate the time series, the person series, and the camera series in this order.

  Furthermore, the procedure described in the present embodiment can also be applied to integration of more general types of sequence information. For example, consider a case where a player wears a heart rate sensor, an acceleration sensor, and a GPS sensor and can acquire heart rate data, velocity / acceleration data, and position data from those sensors. In this case, it is possible to perform integration of three types of integration of a plurality of sensor data of each person, integration of sensor data of a plurality of persons, and integration of time series by the same procedure as this embodiment. When each player wears a different type of sensor, or when the number of worn sensors is different for each player, the integration of multiple sensor data can be expected to have the effect of absorbing those differences. .

(Embodiment 3)
In the first and second embodiments, person detection is performed on a frame acquired by a camera, and action label identification is performed using a local person area based on the result.
In these modes, if person detection does not operate correctly, a non-person area may be erroneously input, which may lead to misidentification.

  In this embodiment, in addition to the human region which is a local part of the image, the image feature from the entire image is extracted and used for identification of the action label to explain a method for reducing misidentification of human detection. Do.

  Similar to the first and second embodiments, the action labels to be identified in this embodiment are five types of action labels: “pass”, “shoot”, “dribble”, “keep”, and “clear”.

  Further, in the present embodiment, as in the first and second embodiments, futsal moving images photographed by a plurality of cameras arranged around the futsal court are used. FIG. 1 is a diagram already described, but is a diagram for explaining an example of this camera arrangement and an example of one frame photographed by two cameras.

  FIG. 16A shows a functional configuration of the behavior recognition apparatus 3000 described in the present embodiment.

  The action recognition apparatus 3000 of this embodiment has a global feature quantity extraction unit 3010 in addition to the functional configuration of the action recognition apparatus 1000 of the first embodiment. Details of each of these functions will be described below with reference to FIG.

  FIG. 3C is a flowchart showing an example of processing at the time of recognition in the present embodiment.

  Here, since S3001 to S3005 are the same processes as S1001 to 1005 at the time of the recognition process in the first embodiment, the description will be omitted.

  The image feature quantity extraction process (S3006) is the same process as the image feature quantity extraction step S1006 at the recognition process in the first embodiment, but is partially different, so the difference will be described together with other processes. .

  In S3007, the global feature extraction unit 3010 extracts global image features from the entire frame acquired in the moving image acquisition step S3001.

  The person series integration process (S2008), the time series integration process (S2009), and the action label identification process (S2010) are the same processes as S1007-1009 at the recognition process in the first embodiment. However, since there is a difference, the difference will be described together with other processing.

  Next, more specific contents will be described according to the flowchart shown in FIG. The present embodiment is different from the first embodiment in the processing after the image feature quantity extraction processing (S3006), and the other processing is the same as that of the first embodiment. Therefore, each different processing will be described.

  In the first embodiment, the respective units functioning in S1006-S1009, the image feature extraction unit 1006, the person series integration unit 1007, the time series integration unit 1008, and the action label identification unit 1009 combine CNN, LSTM and Softmax discriminators Was realized. Also in the present embodiment, the image feature quantity extraction unit 3006, the person series integration unit 3007, the time series integration unit 3008, and the action label identification unit 3009 are realized by CNN, LSTM, and Softmax discriminators. In addition, the global feature extraction unit 3010 used in the global image feature extraction step S3007 is also realized by CNN.

  The structure of a network configured of these is as shown in FIG. 17 when it is illustrated in the same manner as the diagram (FIG. 9) showing the network structure in the first embodiment. A connection control unit (1315) is inserted between LSTM1 and LSTM2 of the network structure (FIG. 9) in the first embodiment. In the combined operation unit (1315), the entire image (1302, 1308) at each time is input to the CNN (1313), and the global image feature value extracted there and the integrated result of the person series by LSTM1 (1314) are connected. Ru. The feature values after connection are input to LSTM 2 (1316) that performs time series integration.

  Here, the entire image (1302, 1308) at each time input to the CNN (1313) has a size adjusted to the CNN without cropping the image of Full HD (1920 x 1080 pixels) acquired by the moving image acquisition means 3001. It is a resized image. The resized image is, for example, 227 × 227 pixels.

  Also, the CNN (1312) that extracts the image feature amount from the human region and the CNN (1313) that extracts the global image feature amount from the entire image may have the same structure and share the same weight parameters. Alternatively, another weight parameter may be set with the same structure, or another structure or another weight parameter may be set.

  As FIG. 17 shows, the image feature-value of the person area | region of the time I is integrated recursively by LSTM1, and it is connected with the global image feature-value extracted from the whole image by the connection operation module. The concatenated feature quantities are input to LsTM 2 that performs time series integration. Similarly, for the time II, the person area is recursively integrated by LSTM 1 and linked with the global image feature, and thereafter, time series integration is recursively performed by LSTM 2. The score of the action label is calculated and output by the Inner-product unit and the Softmax unit for each time-series integration by LSTM2.

  By performing as described above, the feature quantity extracted from the entire image is linked with the feature quantity of the local human region including the detection error obtained by the human detection, and time-series integration is performed. Thereby, identification of the action label which reduced the error which person detection contains can be performed.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

1000 action recognition apparatus 1001 motion image acquisition unit 1002 person object detection unit 1003 person region extraction unit 1004 person region sort unit 1005 integrated control signal creation unit 1006 image feature amount extraction unit 1007 person sequence integration unit 1008 time series integration unit 1009 activity label identification unit

Claims (14)

Acquiring means for acquiring a moving image including a time-series still image;
Detection means for detecting one or more objects for each still image from the moving image;
Feature quantity extraction means for extracting the feature quantity corresponding to each of the objects from the still image;
An object integrating unit that integrates feature amounts corresponding to the respective objects in the still image;
A time-series integration unit that integrates feature quantities of an object integrated in the still image with respect to the time-series still image;
An image processing apparatus, comprising: identification means for identifying the action of the object in the moving image based on the integrated feature amount of the time-series still image. Area extraction means for extracting an area corresponding to each of the objects from the still image;
A creation unit configured to create an area sequence in which the areas extracted from each of the time-series still images are arranged;
The image processing apparatus according to claim 1, wherein the feature amount extraction unit extracts feature amounts from regions corresponding to the respective objects in the region series.   The image processing apparatus according to claim 2, wherein the creation unit sorts the regions of the plurality of objects based on the positions of the objects.   3. The image processing apparatus according to claim 2, further comprising control means for controlling the control states of the object integration means and the time series integration means based on the detection result by the detection means. The object integration means has at least two control states of reset and update;
The control means resets the control state of the object integration means at the beginning of the series of feature quantities of the object in the still image, and updates the control state of the object integration means at other times. 5. The image processing apparatus according to claim 4, wherein the object integration means integrates a series of objects in the still image. The time series integration means has at least three control states of reset, hold and update;
The control means resets the control state of the time series integration means at the initial stage of the series, and brings the control state of the time series integration means at the end of the series of feature quantities of the objects integrated for each still image, By holding the control state of the time-series integration means at other times, the time-series integration means integrates a series of feature quantities of the object integrated for each still image. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the detection unit detects a real-world position of the object.   The image processing apparatus according to claim 1, wherein the detection unit detects a person and a predetermined object different from the person as the object.   The image processing apparatus according to claim 1, wherein the acquisition unit acquires a moving image obtained by photographing an object at a plurality of viewpoints simultaneously.   The detection means detects, for each still image constituting a moving image taken from the plurality of viewpoints, the same object in the still images of the plurality of viewpoints, and detects the position of the object. The image processing apparatus according to claim 9, characterized in that: The image processing apparatus further comprises viewpoint integration means for integrating a series of feature quantities of the objects of the plurality of viewpoints,
11. The image processing apparatus according to claim 10, wherein the object integration unit integrates feature amounts of objects of a plurality of viewpoints integrated for each of the objects. The image processing apparatus further comprises global feature quantity extraction means for extracting a global image feature quantity from the whole still image constituting the moving image,
The image processing apparatus according to claim 1, wherein the time-series integration unit integrates the feature amount of the object integrated for each still image and the global image feature amount for each still image. An acquisition step of acquiring a moving image including a time-series still image;
Detecting one or more objects for each still image from the moving image;
A feature amount extraction step of extracting feature amounts corresponding to each of the objects from the still image;
An object integration step of integrating feature amounts corresponding to each of the objects in the still image;
A time-series integration step of integrating feature quantities of an object integrated in the still image with respect to the time-series still image;
An identification step of identifying the action of the object in the moving image based on the integrated feature amount of the time-series still image.   A program causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images