JP2021128691A - Multi-modal action recognition method, apparatus, and program - Google Patents

Abstract

To provide a multi-modal action recognition method, an apparatus, and a program that recognize a relation between a posture of a person and a behavior of an object in association with a modal of a contact action of the person to the object.SOLUTION: An posture estimation unit 104 estimates a posture of a person based on joint points extracted from a camera image. A sensor signal acquisition unit 105 acquires a sensor signal changing according to movement of an object 2. A behavior estimation unit 106 estimates a behavior of the object based on the sensor signal. A contact modal estimation unit 107 estimates a contact action modal such as a timing when the person contacts the object and a position on the object where the person contacts. An action recognition unit 108 recognizes a causal relation between the posture of the person and a behavior of the object based on a relation between the posture of the person and the behavior of the object synchronized with time.SELECTED DRAWING: Figure 3

Description

The present invention relates to a method, a device and a program for recognizing a user's behavior based on multimodal information, and in particular, recognizes the behavior of a person estimated from a camera image and the behavior of an object handled by the person as multimodal information. Multimodal behavior recognition methods, devices and programs to be performed.

Patent Document 1 describes information obtained based on information input by a user as a method for estimating the posture of a player existing in a sports image by a processor of a posture estimation device, and is a sports image of a game to be estimated. A technique for estimating the posture of an estimated target player who is a player other than the specific player existing in the sports image to be estimated by receiving the reference posture information that specifies the joint position of the specific player existing in the sport image. Is disclosed.

In Patent Document 2, in order to calculate the influence of the golfer's joint during the golf swing on the head speed, the position coordinate data of a predetermined part of the golfer's body during the swing of the club by the golfer and the position coordinate of the predetermined part of the club. The data is acquired, the ground reaction force data of the golfer during the swing is acquired, the position coordinate data of the predetermined part of the golfer's body, the position coordinate data of the predetermined part of the club, the acquired ground reaction force data, and the rigid body link. Based on the model, the joint torque for each golfer's joint is calculated, and the contribution of the calculated joint torque for each joint of the golfer and the force generated by the movement of the golfer during the swing to the head speed of the club is calculated for each golfer's joint. The technology to calculate is disclosed.

Japanese Patent No. 6589144 JP-A-2019-110990

Patent Document 2 analyzes the behavior of a golf club (club head speed) in relation to the posture (joint torque) of the golfer, but the golfer keeps the golf club from the start to the end of the swing. Therefore, the influence of the contact behavior such as the golfer's posture and contact position at the timing when the golfer touches the golf club (holds the grip) on the subsequent swing can be ignored.

On the other hand, in ball games such as soccer and baseball, the behavior of the ball changes according to the posture when the player makes contact with the ball and the position of contact with the ball, which greatly affects the development of the game. Therefore, it is important to recognize the relationship between the posture of a person and the behavior of an object in relation to the mode of contact behavior.

An object of the present invention is to solve the above technical problems and provide a multimodal behavior recognition method, device and program for recognizing the relationship between a person's posture and the behavior of an object in relation to the mode of contact behavior of the person with respect to the object. There is.

In order to achieve the above object, the present invention has the following configuration in a multimodal behavior recognition device that recognizes behavior by using a camera image of a person and a sensor signal for detecting the behavior of an object as multimodal information. There is a feature.

(1) Time synchronization between means for estimating the posture of a person based on a camera image, means for acquiring a sensor signal that changes according to the movement of an object, and means for estimating the behavior of an object based on the sensor signal. Based on the relationship between the posture of the person and the behavior of the object, the means for recognizing the causal relationship between the posture of the person who makes contact with the object and the behavior of the object is provided.

(2) The means for estimating the timing when the person touches the object is further provided, and the causal relationship between the posture of the person at the timing when the person touches the object and the behavior of the object after the timing is recognized.

(3) The means for estimating the position on the object in contact with the person is further provided, and the causal relationship between the posture of the person, the behavior of the object, and the position on the object in contact with the person is recognized.

(4) Further provided with a means for estimating the timing when the person touches the object and a means for estimating the position on the object where the person touches, the posture of the person at the timing when the person touches the object, and after that timing. The behavior of the object and the causal relationship of the position on the object that the person touches are recognized.

According to the present invention, the following effects are achieved.

(1) Based on the relationship between the time-synchronized attitude of the person and the behavior of the object, the causal relationship between the attitude of the person who makes contact with the object and the behavior of the object is recognized, so that it is similar to soccer or baseball. In addition, in behavior recognition in a field where the behavior of the ball changes according to the posture when the player touches the ball and the position of contact with the ball, which has a great influence on the development of the game, the posture of the person and the object It becomes possible to recognize the relationship with the behavior in relation to the mode of the contact behavior.

(2) Since the timing when the person touches the object is estimated and the causal relationship between the posture of the person at the timing when the person touches the object and the behavior of the object after the timing is recognized, the posture of the person and the object are recognized. When the relationship with the behavior of the person depends on the timing T when the person touches the object, the causal relationship between the posture of the person at the timing T when the person touches the object and the behavior of the object after the timing can be recognized. Become.

(3) Since the position on the object that the person touched was estimated and the causal relationship between the posture of the person, the behavior of the object, and the position on the object that the person touched was recognized, the posture of the person and the behavior of the object When the relationship of is dependent on the position P on the object in contact with the person, the posture of the person, the behavior of the object, and the causal relationship of the position P on the object in contact with the person can be recognized.

(4) Estimate the timing when the person touches the object and the position on the object where the person touches, the posture of the person at the timing when the person touches the object, the behavior of the object after that timing, and the object where the person touches. Since the causal relationship of the position of is recognized, the relationship between the posture of the person and the behavior of the object depends on both the timing T when the person touches the object and the position P on the object where the person touches. In addition, it becomes possible to recognize the posture of the person at the timing T when the person touches the object, the behavior of the object after the timing, and the causal relationship of the position P on the object where the person touches.

It is a figure which showed the structure of the behavior recognition system to which this invention is applied. It is a functional block diagram of the sensor system built in the soccer ball. It is a functional block diagram of a multimodal behavior recognition system. It is a figure which showed the result of the posture estimation at the timing of an inside kick. It is a flowchart of the multimodal behavior recognition procedure to which this invention is applied.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a configuration of a behavior recognition system to which the present invention is applied. The posture of a person estimated based on a camera image and the behavior of a detected object are processed as multimodal information and applied to the object. Recognize the causal relationship between the posture of a person who makes contact and the behavior of the object.

In the present embodiment, a case where the posture of the person and the behavior of the soccer ball are estimated and the causal relationship between the posture when the person makes contact with the soccer ball and the subsequent behavior of the soccer ball is recognized as an example. do.

The multimodal behavior recognition system includes a local terminal 1 having a camera and a communication function, an object 2 having various motion sensors and a communication function built-in to transmit a sensor signal to the local terminal 1 and the like, and the local terminal 1 and, for example, Wi. -The main configuration is a behavior recognition server 3 that communicates via Fi, a radio base station BS, and a network NW, and executes multimodal behavior recognition based on information acquired from a local terminal 1. If the processing capacity of the local terminal 1 is sufficiently high and the multimodal action recognition can be executed only by the local terminal 1, the action recognition server 3 may be omitted.

FIG. 2 is a functional block diagram showing the configuration of the sensor system 20 built in the soccer ball 2, and displays the 3-axis acceleration sensor 201, the 3-axis gyro sensor 202, the 3-axis magnetic sensor 203, and the output signals of each sensor. The main structure is a control unit 204 for processing, a battery 205 for supplying power to each sensor and the control unit 204, and a wireless communication interface 206 such as Bluetooth (registered trademark).

The acceleration in the three-axis direction and the angular velocity in the three-axis direction and the geomagnetism in the three-axis direction measured by the sensors 201, 202, and 203 in a predetermined sampling cycle are collectively stored in real time or temporarily together with the measurement time information. Then, it is transmitted from the wireless communication interface 206 to the local terminal 1 and the like.

FIG. 3 is a functional block diagram showing the configuration of the main part of the multimodal action recognition system 100 realized by the local terminal 1 and the action recognition server 3 in cooperation with each other, and is a camera 101, a person area extraction unit 102, and a joint. The main components are the point extraction unit 103, the posture estimation unit 104, the sensor signal acquisition unit 105, the behavior estimation unit 106, the contact mode estimation unit 107, and the behavior recognition unit 108. Each configuration may be distributedly implemented in the local terminal 1 and the action recognition server 3, or may be implemented only in the local terminal 1.

Such a local terminal 1 and an action recognition server 3 implement an application (program) that realizes each function described later on a general-purpose computer or mobile terminal equipped with a CPU, a memory, an interface, a bus connecting them, and the like. It can be configured by. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or programmed. In the present embodiment, a case where the local terminal 1 is substituted by a smartphone or a tablet terminal will be described as an example.

The camera 101 captures a moving image of a person in contact with an object (soccer ball in the present embodiment) 2 and outputs the camera image in frame units. The person area extraction unit 102 extracts a person area from each frame image of the camera image. For example, SSD (Single Shot Multibox Detector) can be used to extract the person area.

The joint point extraction unit 103 extracts the joint points registered in advance as extraction targets from the person area of the frame image. The existing articulated point extraction technology (Cascaded Pyramid Network) can be used to extract the articulated points. The posture estimation unit 104 estimates the posture of the person based on the extracted joint points.

The sensor signal acquisition unit 105 acquires a sensor signal that changes according to the movement of the sensor system 20 mounted on the object 2 handled by the person. In the present embodiment, the acceleration sensor signal, the motion sensor signal, and the geomagnetic sensor signal are acquired from the object 2.

The behavior estimation unit 106 estimates the behavior of the object based on the sensor signal acquired by the sensor signal acquisition unit 105. The rotation direction detection unit 106a detects the rotation direction of the object. The rotation speed detection unit 106b detects the rotation speed of the object. The movement direction detection unit 106c detects the movement direction of the object. The moving speed detection unit 106d detects the moving speed of the object. The acceleration detection unit 106e detects the acceleration of the object. The altitude detection unit 106f detects the altitude of the object.

The contact mode estimation unit 107 includes a contact timing estimation unit 107a and a contact position estimation unit 107b. The contact timing estimation unit 107a estimates the timing T when a person contacts the object 2 based on the movement of the object 2. In the present embodiment, the timing at which a person kicks the soccer ball 2 is estimated. The contact position estimation unit 107b estimates the position P on the object in contact with the person based on the behavior of the object 2. In the present embodiment, the position on the soccer ball kicked by the person is estimated. The contact position P may include the position of the part of the person when the person comes into contact with the object.

The action recognition unit 108 includes a time synchronization unit 108a, a posture estimation unit 108b at the time of contact, and a posture / contact relationship recognition unit 108c, and includes the contact position P, the behavior of the soccer ball after the contact timing T, and the person at the contact timing T. The causal relationship between the posture of the person concerned and the behavior of the soccer ball 2 is recognized based on the posture of the soccer ball 2.

The time synchronization unit 108a synchronizes the posture estimation time of the person and the behavior estimation time of the object. The contact posture estimation unit 108b estimates the posture of the person at the contact timing T when the person contacts the object. In the present embodiment, the posture when a person kicks the soccer ball 2 is estimated. The posture / contact relationship recognition unit 108c recognizes the posture of the person at the timing T when the person contacts the soccer ball 2, the behavior of the object after the timing T, and the causal relationship of the position P on the object with which the person contacts.

Figure 4 is a graph showing the results of a player pose estimation in contact timing inside Kick (Kick timing), (can be represented by skeleton Fs connecting the finger and heel) attitude of the right foot F _R of a person on the ground On the other hand, it is almost parallel and is in contact with the ball at the center position of the _{skeleton F S.}

At this time, if the contact position (kick position) is _{estimated to be slightly above the side of the ball and the center position of the skeleton F S} , and if the forward rotation without slippage is estimated as the behavior of the ball after the contact timing, the right foot F _R is It can be recognized that it is a preferable inside kick method to kick the side surface and the upper part of the soccer ball 2 in a posture substantially parallel to the ground.

On the other hand, the contact position is presumed to be slightly above the side surface of the soccer ball 2 and _{the center position of the skeleton F S} , but the player's right foot F _R is not substantially parallel to the ground, and the behavior of the ball after the contact timing. When it is estimated that include slip and reverse rotation, the inside kick in a posture in which the right foot F _R is not substantially parallel to the ground can even recognize a kick position is not a desired inside kick be appropriate ..

FIG. 5 is a flowchart showing a procedure of multimodal behavior recognition to which the present invention is applied. In step S1, the posture of a person is estimated based on a joint point for each frame image. In step S2, a sensor signal that changes according to the movement of the soccer ball 2 is acquired from the soccer ball 2. In step S3, the contact timing T and the contact position P of the person with respect to the ball 2 are estimated based on the sensor signal acquired from the soccer ball 2.

In step S4, the behavior of the ball after the contact timing T is estimated based on the time series of the sensor signals acquired from the soccer ball 2. In the present embodiment, the behavior of the soccer ball 2 is estimated based on the rotation direction, rotation speed, movement direction, movement speed, acceleration, and altitude of the soccer ball 2.

In step S5, a causal relationship between the kick posture of the person and the subsequent behavior of the ball is recognized based on the posture (kick posture) of the person at the contact timing T, the kick position P, and the behavior of the ball after the kick.

According to the present embodiment, as in soccer and baseball, the behavior of the ball changes according to the posture when the player makes contact with the ball, the position of contact with the ball, and the like, which has a great influence on the development of the game. In behavior recognition in the field, it becomes possible to recognize the relationship between the posture of a person and the behavior of an object in relation to the mode of contact behavior.

In the above embodiment, the causal relationship between the posture of the person and the behavior of the object in the inside kick has been described as being recognized in relation to the posture of the kicking foot, but the present invention is not limited to this. Instead, it may be associated with the posture of another skeleton.

Further, in the above embodiment, the causal relationship between the posture of the person making the pass and the behavior of the ball has been described, but the causal relationship between the posture of the person receiving the pass and the behavior of the ball is recognized. Is also good.

Further, in the above embodiment, the case where the person comes into direct contact with the ball has been described as an example, but there is also a case where a tool (bat, stick, golf club, etc.) substantially integrated with the person comes into contact with the ball. It can be applied in the same way.

On the other hand, in the above embodiment, the timing T when the person touches the object and the position P on the object where the person touches are estimated, the posture of the person at the timing T when the person touches the object, and the behavior of the object after the timing. And, although it has been described as recognizing the causal relationship of the position P on the object in contact with the person, the present invention is not limited to this.

For example, if the relationship between the posture of a person and the behavior of an object does not depend on the position on the object that the person has contacted, but only the timing T at which the person has contacted the object, the contact position estimation unit 107b may be used. Omitted, the action recognition unit 108 may recognize the causal relationship between the posture of the person at the timing T when the person comes into contact with the object and the behavior of the object after the timing.

Further, if the relationship between the posture of the person and the behavior of the object does not depend on the timing T when the person touches the object, but only the position P on the object in which the person touches, the contact timing estimation unit 107a May be omitted, and the action recognition unit 108 may recognize the posture of the person, the behavior of the object, and the causal relationship of the position P on the object in contact with the person.

Further, if the relationship between the posture of the person and the behavior of the object does not depend on either the timing T when the person contacts the object or the position P on the object where the person contacts, the contact timing estimation unit 107a and the contact. The position estimation unit 107b may be omitted, and the behavior recognition unit 108 may recognize only the posture of the person, the behavior of the object, and the causal relationship.

1 ... local terminal, 2 ... soccer ball, 3 ... action recognition server, 101 ... camera, 102 ... person area extraction unit, 103 ... joint point extraction unit, 104 ... attitude estimation unit, 105 ... sensor signal acquisition unit, 106 ... behavior Estimating unit, 107 ... Contact mode estimation unit, 108 ... Behavior recognition unit, 201 ... 3-axis acceleration sensor, 202 ... 3-axis gyro sensor, 203 ... 3-axis geomagnetic sensor, 204 ... Control unit, 205 ... Battery, 206 ... Wireless communication interface

Claims (14)

In a multimodal behavior recognition device that recognizes behavior by using a camera image of a person and a sensor signal that detects the behavior of an object as multimodal information.
A means of estimating the posture of a person based on camera images,
A means of acquiring a sensor signal that changes according to the movement of an object,
A means for estimating the behavior of an object based on the sensor signal, and
Based on the relationship between the time-synchronized posture of the person and the behavior of the object, it is characterized by providing a means for recognizing the causal relationship between the posture of the person who makes contact with the object and the behavior of the object. Multimodal behavior recognition device. Further equipped with a means for estimating the timing when a person comes into contact with an object,
The multimodal behavior recognition according to claim 1, wherein the means for recognizing the causal relationship recognizes a causal relationship between the posture of the person at the timing when the person comes into contact with the object and the behavior of the object after the timing. Device. Further equipped with a means for estimating the position on the object in contact with the person,
The multimodal behavior recognition device according to claim 1, wherein the means for recognizing the causal relationship recognizes the causal relationship between the posture of the person, the behavior of the object, and the position on the object in contact with the person. A means of estimating when a person touches an object,
Further provided with a means for estimating the position on the object in contact with the person,
The means for recognizing the causal relationship is characterized by recognizing the causal relationship of the posture of the person at the timing when the person comes into contact with the object, the behavior of the object after the timing, and the position on the object with which the person comes into contact. The multimodal behavior recognition device according to 1. The means for estimating the behavior of the object according to any one of claims 1 to 4, wherein at least one of the rotation direction, the rotation speed, the movement direction, the movement speed, the acceleration and the altitude of the object is estimated. Multimodal behavior recognition device. The multimodal behavior recognition device according to claim 5, wherein the sensor signal includes at least one of a three-axis acceleration sensor signal, a three-axis gyro sensor signal, and a three-axis geomagnetic sensor signal. In a multimodal behavior recognition method in which a computer recognizes a person's camera image and a sensor signal for detecting the behavior of an object as multimodal information.
Estimate the posture of the person based on the camera image,
Acquires a sensor signal that changes according to the movement of an object,
The behavior of the object is estimated based on the sensor signal,
A multimodal behavior recognition method characterized by recognizing a causal relationship between the posture of a person who makes contact with an object and the behavior of the object based on the relationship between the posture of the person and the behavior of the object synchronized in time. .. Further estimate the timing when a person touches an object,
The multimodal behavior recognition method according to claim 7, wherein the causal relationship between the posture of the person at the timing when the person comes into contact with the object and the behavior of the object after the timing is recognized. Further estimate the position on the object that the person touches,
The multimodal behavior recognition method according to claim 7, wherein the causal relationship between the posture of the person, the behavior of the object, and the position on the object in contact with the person is recognized. Further estimating the timing when the person touches the object and the position on the object where the person touches,
The multimodal behavior recognition method according to claim 7, wherein the posture of the person at the timing when the person touches the object, the behavior of the object after the timing, and the causal relationship of the position on the object with which the person touches are recognized. .. In a multimodal behavior recognition program that recognizes behavior by using a camera image of a person and a sensor signal that detects the behavior of an object as multimodal information.
The procedure for estimating the posture of a person based on the camera image,
The procedure for acquiring sensor signals that change according to the movement of an object,
The procedure for estimating the behavior of an object based on the sensor signal, and
Based on the relationship between the time-synchronized posture of the person and the behavior of the object, the procedure for recognizing the causal relationship between the posture of the person who makes contact with the object and the behavior of the object, and
A multimodal behavior recognition program that lets a computer execute. It also includes a procedure for estimating when a person touches an object.
The multimodal behavior recognition according to claim 11, wherein in the procedure for recognizing a causal relationship, the causal relationship between the posture of the person at the timing when the person comes into contact with the object and the behavior of the object after the timing is recognized. program. It also includes a procedure for estimating the position on an object that a person has come into contact with.
The multimodal behavior recognition program according to claim 11, wherein the procedure for recognizing a causal relationship recognizes a causal relationship between the posture of a person, the behavior of an object, and the position of a person in contact with the object. The procedure for estimating the timing when a person touches an object,
It also includes a procedure for estimating the position on an object that a person has come into contact with.
The procedure for recognizing a causal relationship is characterized in recognizing a causal relationship between the posture of the person at the timing when the person comes into contact with the object, the behavior of the object after the timing, and the position on the object with which the person comes into contact. The multimodal behavior recognition program according to 11.

Images