JP2023175557A - Information processing program and information processing method - Google Patents

Abstract

To provide a novel technique for estimating changes in a person's facial expression during exercise by AU and the exercise intensity of the person based on the changes in facial expression, wherein this technique, unlike traditional methods of estimating exercise intensity, is applicable even to the types of exercise involving large body movements.SOLUTION: An information processing program causes a computer to perform the processes of acquiring a video image in which exercise including the movement of a person's body is captured, estimating the state of movement of the exercise based on skeletal information of the person contained in the video image, estimating the orientation of the person's face based on positional information of the person's facial organs in the video image, estimating the expression of the face from the video image, and assigning weights to a first feature quantity for the orientation of the face and a second feature quantity for the estimated expression based on the relationship between the estimated state of motion and the orientation of the face, and estimating the intensity of exercise based on the first feature quantity to which the weight is assigned and the second feature quantity to which the weight is assigned.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing program and an information processing method.

Facial expressions play an important role in nonverbal communication. Facial expression estimation is an essential technology for developing computers that understand and support people. In order to estimate facial expressions, we must first define a method for describing facial expressions. AU (Action Unit) is known as a method for describing facial expressions. AU indicates facial movements involved in facial expressions, which are defined based on anatomical knowledge of facial muscles. Techniques for estimating AU have also been proposed.

A typical form of an AU estimation engine that estimates AUs is based on machine learning based on a large amount of training data. (presence/absence) and intensity (intensity of occurrence) are used.

Furthermore, for example, there is a technique of estimating a change in the facial expression of a person during exercise using AU, and estimating the exercise intensity of the person based on the change in facial expression. If such exercise intensity can be estimated, it will be possible to determine whether a person is exercising with an appropriate load, for example, from a video of a person exercising during rehabilitation exercise or exercising at a gym. .

Japanese Patent Application Publication No. 2020-188860 Japanese Patent Application Publication No. 2013-111449 US Patent Application Publication No. 2020/0074155

However, with conventional exercise intensity estimation methods, it is possible to estimate the intensity for exercises that do not involve large body movements, such as riding a stationary bike (registered trademark), but it is possible to estimate the intensity for exercises that involve large body movements. may not be possible.

For example, in the case of a squat exercise, which is an example of an exercise that involves large body movements, the timing at which the exercise intensity appears is when the body lowers and then rises. Although the timing at which this exercise intensity appears is the timing at which it is desired to measure the exercise intensity, the exercise intensity cannot be estimated at this timing, and the estimation accuracy may deteriorate.

One aspect is to further improve the accuracy of estimating exercise intensity.

In one aspect, the information processing program acquires a video of a person's exercise including body movements, estimates the state of the person's movement based on skeletal information of the person included in the video, and estimates the state of the person's movement in the video. The direction of the face of the person is estimated based on the position information of the facial organs of the person, the facial expression is estimated from the video, and the direction of the face is estimated based on the relationship between the estimated movement state and the direction of the face. 1 and a second feature for the estimated facial expression, and based on the weighted first feature and the weighted second feature. Then, the computer executes the process of estimating the intensity of the exercise.

In one aspect, the accuracy of estimating exercise intensity can be further improved.

FIG. 1 is a diagram illustrating an example of the relationship between face orientation and AU value. FIG. 2 is a diagram illustrating an example of a difference in face orientation depending on the timing of movement. FIG. 3 is a diagram showing a configuration example of an information processing system according to this embodiment. FIG. 4 is a block diagram showing a configuration example of the information processing device 10 according to the present embodiment. FIG. 5 is a diagram showing an example of exercise intensity estimation according to this embodiment. FIG. 6 is a flowchart showing an example of the flow of exercise intensity estimation processing according to the present embodiment. FIG. 7 is a diagram showing an example of the hardware configuration of the information processing device 10 according to the present embodiment.

Examples of the information processing program and information processing method according to the present embodiment will be described in detail below based on the drawings. Note that this embodiment is not limited to this example. Moreover, each embodiment can be combined as appropriate within a consistent range. The embodiment will be described using AU as an example, but is not limited thereto.

FIG. 1 is a diagram illustrating an example of the relationship between face orientation and AU value. FIG. 1 shows an example of nine AU values estimated using existing technology from images of the same person captured with an expressionless face while changing the direction of the face. The middle image is an example when the face is facing forward, and the surrounding images are examples when the face is facing in the direction of the arrow. Further, the numerical value below each image is the AU value estimated from each image. Note that the AU value takes a value from 0 to 5, and each value in FIG. 1 is the average of all 17 types of AU.

Referring to Figure 1, even for the same expressionless expression, the AU value changes depending on the direction of the face, and if the AU value when the face is facing forward is assumed to be positive, the estimation accuracy of the AU value when the face is facing other than frontally deteriorates. I know what you're doing. Therefore, when estimating the exercise intensity using a change in the facial expression of a person during exercise, that is, the AU value, the estimation accuracy of the exercise intensity can be improved by performing the estimation in consideration of the direction of the face.

FIG. 2 is a diagram illustrating an example of a difference in face orientation depending on the timing of movement. The example in FIG. 2 shows the difference in face orientation depending on the timing of the squat exercise. As shown on the left side of Figure 2, when a person is squatting in a squat exercise, the person's face is directed downward. On the other hand, as shown on the right side of FIG. 2, when a person stands up during a squat exercise, the person's face faces upward. In this way, even with the same exercise, the direction of the face differs depending on the timing. Therefore, in this embodiment, the timing at which the exercise intensity appears, for example, in the case of a squat exercise, determines which direction the person's face is facing when the body lowers and then rises, and the exercise intensity takes into account the direction of the face. Estimate.

Next, the configuration of an information processing system for estimating exercise intensity in consideration of the direction of the face will be described. FIG. 3 is a diagram showing a configuration example of an information processing system according to this embodiment. As shown in FIG. 3, the information processing system 1 is a system in which an information processing device 10 and a camera device 100 are connected to each other via a network 50 so as to be able to communicate with each other.

For the network 50, various communication networks, such as an intranet used within a factory, can be employed, regardless of whether it is wired or wireless. Further, the network 50 is not a single network, and may be configured, for example, by connecting an intranet and the Internet via a network device such as a gateway or other devices (not shown). Note that if the information processing device 10 and the camera device 100 are directly connected, or if the information processing device 10 has a built-in camera function equivalent to that of the camera device 100, the network 50 does not need to be included in the information processing system 1. .

The information processing device 10 may be, for example, a desktop PC (Personal Computer) or a notebook PC, or may be a mobile terminal such as a smartphone or a tablet PC.

For example, the information processing device 10 acquires a video of a person's movement using the camera device 100, and estimates the state of the movement based on the skeletal information of the person included in the video. Further, the information processing device 10 estimates the orientation of the person's face, for example, based on position information of the person's facial organs in the video. Further, the information processing device 10 estimates a facial expression by estimating a person's AU from the video, for example. Then, the information processing device 10 weights the respective feature amounts for the estimated facial orientation and facial expression based on, for example, the relationship between the estimated motor movement state and the facial orientation of the person. The intensity of exercise is estimated based on the calculated features.

Note that although the information processing device 10 is shown as one computer in FIG. 3, it may be a distributed computing system configured of a plurality of computers, for example. Further, the information processing device 10 may be a cloud computer device managed by a service provider that provides cloud computing services.

The camera device 100 is, for example, a camera for photographing a person. Images captured by the camera device 100 are transmitted to the information processing device 10 at any time or at a predetermined timing. Note that, as described above, the camera device 100 may have a camera function built into the information processing device 10.

(Functional configuration of information processing device 10)
Next, the functional configuration of the information processing device 10, which is the main body of execution of this embodiment, will be explained. FIG. 4 is a block diagram showing a configuration example of the information processing device 10 according to the present embodiment. As shown in FIG. 4, the information processing device 10 includes a communication section 20, a storage section 30, and a control section 40.

The communication unit 20 is a processing unit that controls communication with other devices such as the camera device 100, and is, for example, a communication interface such as a network interface card, or a USB interface.

The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and is realized by, for example, a storage device such as a memory or a hard disk. The storage unit 13 stores, for example, video information 31, a motion analysis model 32, and a facial muscle estimation model 33.

The video information 31 stores, for example, a video shot by the camera device 100, that is, a plurality of captured images that are a series of frames of a moving image.

The motion analysis model 32 includes, for example, information regarding a machine learning model for estimating the type and timing of a motion performed by a person from skeletal information of the person in the video, and a model for constructing the machine learning model. Remember parameters. The machine learning model is generated by machine learning, for example, using a person's skeletal information as a feature quantity and using the type and timing of the exercise performed by the person as a correct label. Note that the machine learning model may be generated by the information processing device 10, or may be trained and generated by another information processing device.

The facial muscle estimation model 33 stores, for example, information regarding a machine learning model for estimating the intensity of AU occurrence from a video of a person, and model parameters for constructing the machine learning model. The machine learning model is generated by machine learning, for example, using a video captured by the camera device 100, that is, a captured image, as a feature quantity, and using the AU occurrence intensity as a correct label. Note that the machine learning model may be generated by the information processing device 10, or may be trained and generated by another information processing device.

Note that the above information stored in the storage unit 30 is just an example, and the storage unit 30 can store various information in addition to the above information.

The control unit 40 is a processing unit that controls the entire information processing device 10, and is, for example, a processor. The control unit 40 includes a motion analysis unit 41, a face orientation estimation unit 42, a facial muscle estimation unit 43, a facial energy calculation unit 44, and the like. Note that each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

The motion analysis unit 41 estimates the state of motion including the body movement of the person based on the skeletal information of the person included in the video captured by the camera device 100 using, for example, existing technology.

The face orientation estimating unit 42 estimates the orientation of the person's face based on the positional relationship of each part of the person's facial organs in the video captured by the camera device 100, for example, using existing technology.

For example, the facial muscle estimating unit 43 estimates the facial expression of a person from an image captured by the camera device 100. For example, the facial muscle estimating unit 43 uses existing technology to estimate AUs that indicate facial movements related to expression expression, which are defined based on anatomical knowledge of facial muscles. The facial expression is estimated based on the AU.

For example, the facial energy calculation unit 44 calculates a first feature amount for the facial orientation and a second feature amount for the estimated facial expression based on the relationship between the estimated movement state and the facial orientation. give weight to. Then, the facial energy calculation unit 44 estimates the intensity of the movement based on the weighted first feature amount and the weighted second feature amount.

Next, the exercise intensity estimation executed by the information processing device 10 will be described in more detail. FIG. 5 is a diagram showing an example of exercise intensity estimation according to this embodiment.

First, as shown in FIG. 5, for example, a video imaged by the camera device 100 is input to the information processing device 10 (step S1). The input video may be stored in the video information 31 by the information processing device 10.

Next, the information processing device 10 detects a person from the input video using, for example, an existing object detection algorithm or an existing skeletal estimation algorithm, generates skeletal information of the person, and performs actions performed by the person. Analyze the motion that is occurring (step S2). Note that these human detection, skeletal information generation, and motion analysis may be performed for each frame of the input video.

Here, existing object detection algorithms include object detection algorithms such as YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), and Faster R-CNN (Convolutional Neural Network) using deep learning. It's good to be there. Further, the existing skeleton estimation algorithm may be, for example, an existing skeleton estimation algorithm such as CPN (Cascaded Pyramid Network).

Furthermore, the analysis of the movement performed by a person involves estimating the type and timing of the movement performed by the person, based on the generated skeletal information, for example. These estimations may be performed, for example, using a machine learning model generated by machine learning, using skeletal information as a feature amount and using the type and timing of movement as a correct label. Note that if the type of exercise is determined in advance, only the timing of the exercise may be estimated. The estimation result of the movement timing may be, for example, a normalized value such as 0 to 1 that increases as the movement progresses towards the latter half. This is because the intensity of the exercise is more likely to be expressed in facial expressions as the latter half of the exercise progresses. More specifically, the estimation result of exercise timing (hereinafter referred to as "exercise timing value") is, for example, in the case of a squat exercise, 0.1 for the first squat and 0 for the second squat. .2, . . . the 10th time may have a value of 1.

Next, the information processing device 10 estimates the direction of the person's face from the input video using, for example, an existing face direction estimation algorithm (step S3). The face orientation estimation result (hereinafter referred to as "face orientation value") is a normal value such as 0 to 1, where the value for the front is set to 0 and the value increases as the face orientation is upward or downward. It may be a converted value. Alternatively, the value of the face direction may be the angle of the face direction when the front value is 0°. This is because people tend to look up or down when exercise becomes difficult.

Next, the information processing device 10 estimates the facial muscles of the person from, for example, the input video (step S4). The estimation may be performed, for example, using a machine learning model generated by machine learning, using a video captured by the camera device 100, that is, a captured image, as a feature quantity, and using the AU occurrence intensity as a correct label. As shown in FIG. 5, there are a total of 17 types of AU generation intensity, that is, AU values, and the values range from 0 to 5, starting from the lowest generation intensity. Then, the facial expression of the person is estimated based on the value of each AU. Note that the individual values of AUs do not necessarily correspond to facial expressions, and facial expressions can be estimated by combinations of AUs and AU values within the combinations. Furthermore, the processing order of the motion analysis, facial orientation estimation, and facial muscle estimation in steps S2 to S4 does not have to be in the order shown in FIG. may be processed.

Next, the information processing device 10 estimates the intensity of the exercise performed by the person, based on the results of the motion analysis, facial orientation estimation, and facial muscle estimation in steps S2 to S4, for example (step S5). More specifically, the information processing device 10 calculates the intensity of the exercise performed by the person using, for example, the following equation (1).

The exercise intensity calculated by equation (1) is the sum of the exercise intensities calculated for each frame of the video. Further, in Equation (1), α may be, for example, the timing value of the movement estimated in step S2. Further, β may be, for example, a weighting coefficient for the value of AU estimated in step S4. Further, γ may be a weighting coefficient for the face orientation value calculated in step S3. Note that the AU value used in equation (1) may be, for example, the average of all 17 types of AU values. Further, regarding the weighting coefficient β, for example, when the face is facing forward, β may be set to 1 so as to use the value of AU, and a weight may be given. Further, regarding the weighting coefficient γ, for example, when the direction of the face is the same as the direction of movement, γ may be set to 1 so as to use the value of the direction of the face, and a weight may be assigned.

(Processing flow)
Next, the flow of exercise intensity estimation processing executed by the information processing device 10 will be described using FIG. 6. FIG. 6 is a flowchart showing an example of the flow of exercise intensity estimation processing according to the present embodiment. The estimation process shown in FIG. 6 may be executed, for example, at regular intervals or every time a video is received from the camera device 100.

First, as shown in FIG. 6, the information processing device 10 acquires, from the video information 31, a video of a person photographed by the camera device 100 and input to the information processing device 10, for example (step S101). Note that in the estimation process shown in FIG. 6, the video captured by the camera device 100 is processed almost in real time, so the video is transmitted from the camera device 100 at any time and stored in the video information 31.

Next, the information processing device 10 analyzes, for example, the exercise performed by the person in the video acquired in step S101 (step S102). More specifically, as explained in FIG. 5, for example, a person is detected from an input video using an existing object detection algorithm or an existing skeleton estimation algorithm, and skeletal information of the person is generated. Then, the movements the person is performing are analyzed. Note that the processes after step S102 may be executed for each frame of the input video.

If the estimated exercise timing is not a timing with a high exercise load according to the exercise analysis in step S102 (step S103: No), the information processing device 10 sets the timing value α to 0, for example (step S104). In this way, for example, by setting the timing value α to 0, in a frame where the exercise load is not high, α in equation (1) becomes 0, and the exercise intensity for the frame becomes 0.

After execution of step S104, if there is a next frame (step S112: Yes), the process is repeated from step S102 for the next frame.

On the other hand, if the estimated exercise timing is a timing with a high exercise load (step S103: Yes), the information processing device 10 sets the timing value α to a value greater than 0 and less than or equal to 1 (step S105). . More specifically, in the case of a timing with a higher exercise load, for example, in the case of 10 squat exercises, the information processing device 10 may set the timing value α so that it approaches 1 as the second half progresses. . As a result, when calculating the exercise intensity using equation (1), the information processing device 10 calculates the timing with a high exercise load with more weight than the timing with a low exercise load. It can be controlled as follows.

Next, the information processing device 10 estimates, for example, the orientation of the face of the person in the video acquired in step S101, based on the positional relationship of each part of the facial organs of the person (step S106). Then, the information processing device 10 determines, for example, whether the estimated face direction in step S106 is the front direction. Note that the estimation result in step S106 may be, for example, a value that approaches 1 as the angle of the face direction increases, with the front value being 0. In this case, it may be determined that the object is facing forward.

If it is determined that the face direction estimated in step S106 is frontal (step S107: Yes), the information processing device 10 sets, for example, a value of 1 to the weighting coefficient β and a value of 0 to γ (step S108). ). Also, at this time, the information processing device 10 calculates, for example, the value of AU. Thereby, when the face direction is not the front direction, the information processing device 10 can be controlled so as to use the value of AU and not use the value of the face direction when calculating the exercise intensity using equation (1). Note that the AU value may be, for example, the average of all 17 types of AU values. After executing step S108, the process advances to step S112.

On the other hand, if it is determined in step S106 that the estimated face orientation is not frontal (step S107: No), the information processing device 10 determines, for example, whether the face orientation is the same as the movement direction. (Step S109). Note that the determination of whether or not the direction of the face is the same as the direction of movement is not limited to determining whether or not the values indicating each direction match, but rather, for example, determining whether the values indicating each direction match or not. This may be a determination as to whether or not it is within the range.

If it is determined that the direction of the face is not the same as the direction of movement (step S109: No), the information processing device 10 sets, for example, a value of 0 to the weighting coefficient β and a value of 0 to γ (step S110). Since it is not appropriate to use the frame in this case for calculating the exercise intensity, the information processing device 10 can control the frame so that neither the AU value nor the face direction value are used. After executing step S110, the process advances to step S112.

On the other hand, if it is determined that the direction of the face is the same as the direction of movement (step S109: Yes), the information processing device 10 sets, for example, a value of 0 to the weighting coefficient β and a value of 1 to the weighting coefficient γ (step S111). . As a result, when the direction of the face is not the same as the direction of movement, the information processing device 10 uses the value of the face direction and controls so that the value of AU is not used when calculating the movement intensity using equation (1). can. After executing step S111, the process advances to step S112.

If there is a next frame (step S112: Yes), the process is repeated from step S102 for the next frame. On the other hand, if there is no next frame (step S112: No), the information processing device 10 calculates the exercise intensity using, for example, equation (1) and the timing value α, weighting coefficients β and γ for each frame. (Step S113). After executing step S113, the exercise intensity estimation process shown in FIG. 6 ends.

(effect)
As described above, the information processing device 10 acquires a video of a person's exercise including body movements, and estimates the state of the movement based on the skeletal information of the person included in the video. The direction of the face of the person is estimated based on the position information of the facial organs of the person in the video, the expression of the face is estimated from the video, and the state of the estimated movement and the orientation of the face are estimated. A weight is assigned to each of a first feature amount for the facial orientation and a second feature amount for the estimated facial expression based on the relationship, and the first feature to which the weight is assigned is The intensity of the movement is estimated based on the amount and the second feature amount to which the weight is given.

As a result, the information processing device 10 estimates the state of movement, the direction of the face, and the AU of the person in the video, and weights the direction of the face and the AU based on the relationship between the state of movement and the direction of the face. Estimate exercise intensity. Thereby, the information processing device 10 can further improve the accuracy of estimating exercise intensity.

Further, the information processing device 10 causes the computer to execute processing for determining whether the estimated orientation of the face is frontal, and the processing for assigning the weight is performed when the estimated orientation of the face is frontal. If it is determined that this is the case, the process includes a process of setting a weighting coefficient to 0 to give the weight to the first feature amount.

Thereby, the information processing device 10 can further improve the accuracy of estimating exercise intensity.

Further, the information processing device 10 causes the computer to execute a process of determining whether or not the estimated orientation of the face is frontal, and if it is determined that the estimated orientation of the face is not frontal, the information processing device 10 The process of determining whether or not the orientation of the face and the direction of the movement are the same and assigning the weight is, when it is determined that the orientation of the face and the direction of the movement are the same, Adding the weight to the feature includes a process of setting a weighting coefficient to 0.

Thereby, the information processing device 10 can further improve the accuracy of estimating exercise intensity.

Further, the information processing device 10 causes the computer to execute a process of determining the timing of the exercise from the video, and the process of estimating the intensity includes a process of estimating the intensity further based on the timing of the exercise. .

Thereby, the information processing device 10 can further improve the accuracy of estimating exercise intensity.

(system)
Information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be arbitrarily changed unless otherwise specified. Furthermore, the specific examples, distributions, numerical values, etc. described in the examples are merely examples, and may be changed arbitrarily.

Furthermore, the specific form of distribution and integration of the components of each device is not limited to what is illustrated. For example, the motion analysis unit 41 of the information processing device 10 in FIG. 4 may be distributed to multiple processing units, or the motion analysis unit 41 and the face orientation estimation unit 42 of the information processing device 10 may be integrated into one processing unit. You can. In other words, all or part of the components may be functionally or physically distributed and integrated in arbitrary units depending on various loads and usage conditions. Furthermore, all or any part of each processing function of each device can be realized by a CPU and a program that is analyzed and executed by the CPU, or can be realized as hardware using wired logic.

FIG. 7 is a diagram showing an example of the hardware configuration of the information processing device 10 according to the present embodiment. As shown in FIG. 7, the information processing device 10 includes a communication interface 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Further, each part shown in FIG. 7 is interconnected by a bus or the like.

The communication interface 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores programs and a DB (Data Base) for operating each processing unit and each function of the information processing device 10 shown in FIG. 4 and the like.

The processor 10d is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or the like. Further, the processor 10d may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor 10d reads a program that executes the same processing as each processing unit shown in FIG. 4 etc. from the HDD 10b and deploys it in the memory 10c, thereby executing a process to realize each function described in FIG. 4 etc. It is a hardware circuit.

Further, the information processing device 10 can also realize the same functions as in the embodiments described above by reading the above program from a recording medium using a medium reading device and executing the read program. Note that the programs in other embodiments are not limited to being executed by the information processing device 10. For example, the above embodiments may be similarly applied to cases where another computer or server executes a program, or where these computers or servers cooperate to execute a program.

This program may be distributed via a network such as the Internet. In addition, this program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and is read from the recording medium by the computer. It may also be executed by being read.

Regarding the embodiments including the above examples, the following additional notes are further disclosed.

(Additional note 1) Obtain a video of a person's exercise including body movements,
estimating the state of the movement of the movement based on skeletal information of the person included in the video;
Estimating the orientation of the person's face based on position information of facial organs of the person in the video,
estimating the facial expression from the video;
Based on the relationship between the estimated movement state and the face orientation, a weight is given to each of a first feature amount for the facial orientation and a second feature amount for the estimated facial expression. ,
Information characterized by causing a computer to perform a process of estimating the intensity of the exercise based on the first feature amount to which the weight is assigned and the second feature amount to which the weight is assigned. Processing program.

(Additional note 2) causing the computer to execute a process of determining whether the estimated orientation of the face is frontal;
The process of assigning the weight is as follows:
As described in supplementary note 1, the method includes a process of setting a weighting coefficient to 0 as the weighting to the first feature quantity when it is determined that the estimated face orientation is a frontal direction. information processing program.

(Additional note 3) causing the computer to execute a process of determining whether the estimated orientation of the face is frontal;
If it is determined that the estimated orientation of the face is not frontal, determining whether or not the orientation of the face and the direction of the movement are the same;
The process of assigning the weight is as follows:
Supplementary note characterized in that, when it is determined that the direction of the face and the direction of the movement are the same, the weighting coefficient is set to 0 for the second feature amount as the weighting. 1. The information processing program described in 1.

(Additional note 4) causing the computer to execute a process of determining the timing of the movement from the video;
The process of estimating the strength is as follows:
The information processing program according to supplementary note 1, further comprising: estimating the intensity based on the timing of the exercise.

(Additional note 5) Obtain a video of a person's exercise including body movements,
estimating the state of the movement of the movement based on skeletal information of the person included in the video;
Estimating the orientation of the person's face based on position information of facial organs of the person in the video,
estimating the facial expression from the video;
Based on the relationship between the estimated movement state and the face orientation, a weight is given to each of a first feature amount for the facial orientation and a second feature amount for the estimated facial expression. ,
Information characterized in that a computer executes a process of estimating the intensity of the exercise based on the first feature amount to which the weight is assigned and the second feature amount to which the weight is assigned. Processing method.

(Additional Note 6) The computer executes a process of determining whether the estimated direction of the face is frontal,
The process of assigning the weight is as follows:
According to appendix 5, the method includes a process of setting a weighting coefficient to 0 as the weighting to the first feature amount when it is determined that the estimated face direction is a frontal direction. information processing methods.

(Additional Note 7) The computer executes a process of determining whether the estimated direction of the face is frontal,
If it is determined that the estimated orientation of the face is not frontal, determining whether the orientation of the face and the direction of the movement are the same;
The process of assigning the weight is as follows:
A supplementary note characterized in that, when it is determined that the direction of the face and the direction of the movement are the same, a process of setting a weighting coefficient to 0 as the weighting to the second feature amount is included. 5. The information processing method described in 5.

(Additional Note 8) The computer executes a process of determining the timing of the exercise from the video,
The process of estimating the strength is as follows:
The information processing method according to appendix 5, further comprising: estimating the intensity based on the timing of the exercise.

(Additional note 9) A processor,
a memory operably connected to the processor;
An information processing device comprising:
Obtain footage of a person's physical movements, including movement,
estimating the state of the movement of the movement based on skeletal information of the person included in the video;
Estimating the orientation of the person's face based on position information of facial organs of the person in the video,
estimating the facial expression from the video;
Based on the relationship between the estimated movement state and the face orientation, a weight is given to each of a first feature amount for the facial orientation and a second feature amount for the estimated facial expression. ,
An information processing device configured to perform a process of estimating the intensity of the exercise based on the first feature amount given the weight and the second feature amount given the weight. .

1 Information processing system 10 Information processing device 10a Communication interface 10b HDD
10c memory 10d processor 20 communication unit 30 storage unit 31 video information 32 motion analysis model 33 facial muscle estimation model 40 control unit 41 motion analysis unit 42 facial orientation estimation unit 43 facial muscle estimation unit 44 facial energy calculation unit 50 network 100 camera device

Claims (5)

Obtain footage of a person's physical movements, including movement,
estimating the state of the movement of the movement based on skeletal information of the person included in the video;
Estimating the orientation of the person's face based on position information of facial organs of the person in the video,
estimating the facial expression from the video;
Based on the relationship between the estimated movement state and the face orientation, a weight is given to each of a first feature amount for the facial orientation and a second feature amount for the estimated facial expression. ,
Information characterized by causing a computer to perform a process of estimating the intensity of the exercise based on the first feature amount to which the weight is assigned and the second feature amount to which the weight is assigned. Processing program. causing the computer to execute a process of determining whether the estimated direction of the face is frontal;
The process of assigning the weight is as follows:
2. The method according to claim 1, further comprising a process of setting a weighting coefficient to 0 to give the weight to the first feature quantity when it is determined that the estimated direction of the face is a frontal direction. The information processing program described. causing the computer to execute a process of determining whether the estimated direction of the face is frontal;
If it is determined that the estimated orientation of the face is not frontal, determining whether or not the orientation of the face and the direction of the movement are the same;
The process of assigning the weight is as follows:
A claim characterized in that, when it is determined that the direction of the face and the direction of the movement are the same, the weighting coefficient is set to 0 for the second feature amount as the weighting. The information processing program according to item 1. causing the computer to execute a process of determining the timing of the movement from the video;
The process of estimating the strength is as follows:
The information processing program according to claim 1, further comprising: estimating the intensity based on the timing of the exercise. Obtain footage of a person's physical movements, including movement,
estimating the state of the movement of the movement based on skeletal information of the person included in the video;
Estimating the orientation of the person's face based on position information of facial organs of the person in the video,
estimating the facial expression from the video;
Based on the relationship between the estimated movement state and the face orientation, a weight is given to each of a first feature amount for the facial orientation and a second feature amount for the estimated facial expression. ,
Information characterized in that a computer executes a process of estimating the intensity of the exercise based on the first feature amount to which the weight is assigned and the second feature amount to which the weight is assigned. Processing method.

Images