JP7459658B2 - Physical ability presentation method and physical ability presentation device - Google Patents

Description

This disclosure relates to a physical ability presentation method and a physical ability presentation device.

In recent years, there has been progress in the development of technology to estimate human posture. Non-Patent Document 1 discloses that the angles of the joints of a person doing radio calisthenics are calculated using a motion sensor device, and the difference between the calculated angles and the ideal angles is compared before and after training in a rehabilitation system.

Shohei Ohno and 4 others, “Proposal of analysis method of physical training effect using Kinect”, Volume 3, pp. 367-368 [online], 2018, FIT2018 (17th Information Science and Technology Forum) , [Retrieved March 26, 2020], Internet 30 <URL: https://www.ieice.org/publications/conference-FIT-DVDs/FIT2018/data/pdf/K-027.pdf>

According to the technique described in Non-Patent Document 1, it is possible to evaluate the physical ability of a subject after training. However, for example, it is not good for the subject's body to push himself too hard in order to get a good evaluation. Furthermore, new goals that are set based on evaluations that the target person has acquired through effort will become goals that are more difficult for the target person than necessary. For this reason, an evaluation obtained by forcing the subject to obtain it cannot be said to be an appropriate evaluation of the subject's physical ability. On the other hand, an evaluation obtained without the subject exerting his/her ability to the best of his/her ability cannot be said to be an appropriate evaluation of the subject's physical ability.

The purpose of this disclosure is to present information that allows for an appropriate assessment of a subject's physical ability while taking into account the burden placed on the subject.

A physical ability presentation method according to an aspect of the present disclosure includes the steps of generating posture information of a subject from an image including the subject, acquiring biological information of the subject, and determining the posture of the subject based on the posture information. a step of identifying the target person's load on the target person's posture based on the biological information; a step of identifying the target person's load on the target person's posture based on the posture information; and a step of presenting specific information regarding the physical ability of the subject by combining the subject's load on the posture.

A physical ability presentation device according to another aspect of the present disclosure includes a generation unit that generates posture information of a subject from an image including the subject, an acquisition unit that acquires biometric information of the subject, a posture identification unit that identifies the posture of the subject based on the posture information, a load identification unit that identifies the load on the subject relative to the posture of the subject based on the biometric information, and a presentation unit that combines the posture identified by the posture identification unit and the load identified by the load identification unit to present specific information regarding the physical ability of the subject.

According to the present disclosure, it is possible to present information that allows for an appropriate assessment of the subject's physical ability while taking into account the burden on the subject.

1 is a diagram showing the overall configuration of a system according to an embodiment of the present invention; FIG. 13 is a diagram showing the relationship between the positions of feature points of a subject and the joint angle of the right shoulder. This is a histogram showing the degree of shoulder movement of the subject. 1 is a graph showing changes in the subject's shoulder joint angle. It is a figure showing the stage of strength and the stage of evaluation. 1 is a table showing the relationship between the subject's shoulder joint angle and strength and the target angle and target strength. 1 is a graph showing the relationship between a subject's shoulder joint angle and strength and a target angle and target strength. It is a table showing past measurement results and current measurement results of the subject. It is a graph showing the relationship between the joint angle and strength of the subject's shoulder, and the target angle and target strength, based on the subject's past measurement results. It is a flowchart which shows the content of control for evaluating a subject's physical ability. FIG. 3 is a diagram showing an example of a matching degree evaluation screen. FIG. 13 is a diagram showing an example of a symmetry evaluation screen. This is a first evaluation sheet showing the physical ability of the subject. This is a second evaluation sheet showing the physical ability of the subject. It is a flowchart which shows the content of control for evaluating a subject's physical ability. It is a flowchart which shows the content of control for evaluating a subject's physical ability.

Embodiments of the present disclosure will be described in detail below with reference to the drawings. In addition, the same reference numerals are attached to the same or corresponding parts in the drawings, and the description thereof will not be repeated.
[Embodiment 1]
<System configuration>
FIG. 1 is a diagram showing the configuration of the entire system according to the first embodiment. The control device 100 collects information regarding the physical ability of a person 20 whose physical ability is to be evaluated (hereinafter referred to as a target person). The control device 100 evaluates the physical ability of the subject 20 based on the information collected from the subject 20.

The control device 100 is connected to a communication device 12 installed in the house 10 of the subject 20 via a communication network such as the Internet 30. The communication device 12 is connected to a camera 13 for photographing the subject 20. A plurality of facial muscle sensors 14 are attached to the subject 20 for detecting the facial expressions of the subject 20. The facial muscle sensor 14 detects potential signals emitted from the facial muscles of the subject 20. The facial muscle sensor 14 is wirelessly connected to the communication device 12. The facial expression specified by the detection values of the plurality of facial muscle sensors 14 is an example of the subject's 20 biological information.

The camera 13 is installed in an optimal location for photographing the subject 20. For example, if the movement of the shoulder joint of the subject 20 is to be evaluated, it is desirable to install the camera 13 in a location where it is easy to photograph the shoulder movement of the subject 20. For example, if the location is a kitchen, the camera 13 is installed in the kitchen as shown in FIG. 1. The shooting direction of the camera 13 is near the shoulder of the subject 20. The camera 13 may be provided with a function that automatically switches the shooting direction to near the shoulder of the subject 20 when the subject 20 moves around the kitchen.

In the following, in the first embodiment, we will explain the evaluation of the shoulder movement of subject 20 as an example of evaluating the physical ability of subject 20.

The camera 13 transmits the captured video to the communication device 12. The facial muscle sensor 14 transmits the detection values of the facial muscles of the subject 20 to the communication device 12. The communication device 12 transmits the video captured by the camera 13 and the detection values of the facial muscle sensor 14 to the control device 100 via a communication network such as the Internet 30.

The control device 100 has a processor 101, a memory 102, and a communication unit 103. These units are connected to each other so that they can communicate with each other.

The processor 101 is typically an arithmetic processing unit such as a CPU (Central Processing Unit) or an MPU (Multi Processing Unit). The processor 101 realizes the processing of the control device 100 by reading and executing a program stored in the memory 102. Note that while the example in FIG. 1 illustrates a configuration with a single processor, the control device 100 may also be configured with multiple processors.

The memory 102 is realized by nonvolatile memory such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The memory 102 stores programs executed by the processor 101, data used by the processor 101, and the like.

Note that the memory 102 may be a CD-ROM (Compact Disc - Read Only Memory), a DVD-ROM (Digital Versatile Disk - Read Only Memory), a USB (Universal Serial Bus) memory, a memory card, a FD (Flexible Disk), a hard disk, an SSD (Solid State Drive), a magnetic tape, a cassette tape, an MO (Magnetic Optical Disc), an MD (Mini Disc), an IC (Integrated Circuit) card (excluding memory cards), an optical card, a mask ROM, or an EPROM, as long as it can non-temporarily record a program in a format that can be read by the control device 100, which is a type of computer.

The communication unit 103 is connected to the Internet 30 and to the display unit 200. The communication unit 103 is an interface for communicating with the communication device 12 and the display unit 200 installed in the home 10 of the subject 20.

The display unit 200 is composed of a liquid crystal display panel or the like. The display unit 200 displays, for example, the results of calculations executed by the processor 101. Typically, the display unit 200 displays an evaluation regarding the physical ability of the subject 20.

The control device 100 receives a video of the subject 20 and the detection values of the facial muscle sensor 14 from the communication device 12. The video of the subject 20 is composed of a number of frame images. The control device 100 associates the frame images with the detection values of the facial muscle sensor 14 received at the time the frame images were received, and stores them in the memory 102.

The control device 100 estimates the posture of the subject 20 based on the frame images stored in the memory 102. The control device 100 estimates the posture of the subject 20 using a trained model for position estimation. The trained model for position estimation is a neural network model that inputs features contained in the frame images and outputs the positions of the joints of the human body as an estimation result. As the trained model for position estimation, for example, OpenPose developed by CMU (Carnegie Mellon University), Human-pose-estimation developed by Microsoft, or PoseNet developed by Google can be used.

In addition to posture estimation, the control device 100 reads out the detected values of the facial muscle sensor 14 from the memory 102. The detected value to be read is the detected value stored in the memory 102 in association with the frame image used for posture estimation. The control device 100 estimates the facial expression of the subject 20 based on the read detection value. The facial expression of the subject 20 can be estimated using various AI (Artificial Intelligence) techniques for facial expression estimation. Various AIs for facial expression estimation take as input values detected by the facial muscle sensor 14, the subject's face photographed by a camera, or feature points of the face such as the position and shape of the eyes and mouth, and output the output. It is a type of facial expression. The types of facial expressions are classified into a finite number of categories, such as smiling and suffering, for example. AI for facial expression estimation can be trained using a dataset of input and output pairs. By inputting the detected values of the facial muscle sensor 14 or the subject's face photographed by a camera to the trained AI, it is possible to obtain the classification of the specified facial expression as an output as an inference result.

The control device 100 calculates the angles of the joints of the subject 20 from the posture estimation results. For example, if the location to be evaluated is the movement of the shoulder of the subject 20, the control device 100 calculates the joint angle of the shoulder of the subject 20. In addition, the control device 100 compares the results of facial expression estimation with a plurality of step-by-step reference values, or defines the degree of load for each facial expression classification, such as if the face has a painful look, the load is high. By doing so, the degree of load (severity) that the subject 20 feels when moving the shoulder is specified. Hereinafter, the "degree of load" will be referred to as "intensity".

The control device 100 evaluates the physical ability of the subject 20 based on the angle and strength of the joint. Furthermore, the control device 100 displays the results of those evaluations on the display section 200 and transmits them from the communication section 103 via the Internet 30 to the communication device 12 installed in the house 10 of the subject 20. The communication device 12 is connected to the personal computer 16 of the subject 20. The subject 20 can check his or her own evaluation using the personal computer 16.

The control device 100 stores different evaluation criteria for each subject 20. For example, if the subject 20 is undergoing shoulder rehabilitation, it is conceivable to set in the control device 100 a reference value corresponding to the goal value of the rehabilitation.
<Calculating the shoulder joint angle>
FIG. 2 is a diagram showing the relationship between the positions of feature points of the subject 20 and the shoulder joint angle. A video of the subject 20 is composed of a plurality of frame images. FIG. 2 shows one of the plurality of frame images. Reference numerals 20a to 20k shown in FIG. 2 indicate feature points estimated from the image of the subject 20. The control device 100 estimates these feature points 20a to 20k from the frame images by using a pose estimation technique such as OpenPose. Note that when the part to be observed is the finger of the subject 20, the feature points of the finger joints may be estimated.

As shown in FIG. 2, the right shoulder joint angle of subject 20 is found from the coordinates of feature point 20b corresponding to the right elbow joint, feature point 20c corresponding to the right shoulder joint, and feature point 20d corresponding to the right hip joint, among feature points 20a to 20k. Specifically, when the shoulder-elbow vector is a and the shoulder-hip joint vector is b, the right shoulder joint angle can be calculated using the following formula (1).

A video of the subject 20 includes various movements of the subject 20. For example, when focusing on the joint angle of the right shoulder of the subject 20, the following method can be used to determine the shoulder joint angle of the subject 20. First, from among a plurality of frame images in which feature points have been extracted by the posture estimation technology, frame images including feature points 20b, 20c, and 20d as shown in FIG. 2 are selected. Next, the shoulder joint angle of the subject 20 is calculated based on each of the selected frame images. The maximum joint angle of the calculated joint angles is determined as the shoulder joint angle of the subject 20. Note that this method is merely one example, and other methods may be adopted to determine the shoulder joint angle of the subject 20.

If it is desired to identify the range of shoulder movement of the subject 20 in daily life, it is advisable to create a histogram using the shoulder joint angle calculated based on each frame image constituting the video. Alternatively, if it is desired to know the change in the shoulder joint angle of the subject 20 throughout the day, it is advisable to create a graph in which the shoulder joint angle calculated based on the frame images corresponds to the time when the frame images were captured. These histograms and graphs are created by the control device 100 and displayed on the display unit 200 as necessary. Each will be described below with reference to Figs. 3 and 4.
<Differences in shoulder joint angle depending on the subject>
FIG. 3 is a histogram showing the degree of shoulder joint angle of a certain subject A and a certain subject B. In FIG. 3, the vertical axis shows the class of the shoulder joint angle, and the horizontal axis shows the degree for the class. LA and LB show curves obtained by linearizing the relationship between the class and the degree. LA shows the curve corresponding to the subject A. LB shows the curve corresponding to the subject 20B. As shown in FIG. 3, the degree of shoulder movement differs depending on the subject 20. The subject A moves his shoulder joint in a wide range from an angle of less than 20 degrees to an angle of more than 160 degrees. On the other hand, the subject 20B has a narrower range of shoulder movement than the subject A, and rarely moves his shoulder joint beyond 120 degrees. In this way, the range of shoulder movement differs depending on the person. In addition, the range of shoulder movement generally tends to become narrower as the person gets older.
<Changes in shoulder joint angle over time>
Fig. 4 is a graph showing changes in the shoulder joint angle of a certain subject 20. The vertical axis shows the shoulder joint angle, and the horizontal axis shows time. As shown in Fig. 4, the shoulder joint angle changes significantly depending on the time. The manner in which this change occurs differs significantly depending on the type of work the subject 20 was doing at the time of observation.

The joint velocity information can be calculated from the time series data of the joint positions shown in FIG. 4. The slope of the graph and the joint velocity are correlated. Instead of the joint angle obtained from the video, the joint velocity information obtained from the video may be used as the subject of evaluation. The shoulder joint velocity information is an index for determining whether the subject can move his/her shoulder smoothly. Also, instead of the shoulder joint velocity information, the arm joint velocity information may be used as the subject of evaluation. By using the arm joint velocity information, it can be determined whether the subject 20 is raising his/her arm slowly. Also, by using the leg joint velocity information, the walking speed of the subject 20 can be estimated. The reference speed for the leg joint velocity information may be determined based on the walking speed of an elderly person who requires care.

By referring to FIG. 3 or 4, the degree of movement of the shoulder joint of the subject 20 can be grasped. However, with only the information shown in FIGS. 3 and 4, it is not possible to understand whether the subject 20 moved the shoulder joint with sufficient margin or whether he moved the shoulder joint by force. Therefore, in the present embodiment, the degree (intensity) of the load that the movement imparts to the subject 20 is calculated by focusing on the facial expression of the subject 20 in addition to the movement of the shoulder joint of the subject 20.

FIG. 5 is a diagram showing the intensity levels and the evaluation levels. In this embodiment, the degree of the load given to the subject 20 is evaluated in 11 levels from -5 to +5 in increments of 1. The median value 0 corresponds to a normal feeling, neither painful nor comfortable. The higher the value from 0, the higher the intensity, i.e., the degree of discomfort felt by the subject 20. On the other hand, the lower the value from 0, the higher the degree of comfort felt by the subject 20. As shown in FIG. 5, seven levels from A to G are set as the evaluation levels. A indicates the highest evaluation, and G indicates the lowest evaluation. Note that the illustrated intensity levels and evaluation levels are merely examples, and the number of levels may be further increased or, conversely, reduced.
<Evaluation of the subject's physical ability>
Fig. 6 is a table showing the relationship between the shoulder joint angle and strength of a certain subject A and the target angle α and target strength β, and an evaluation based thereon. Fig. 7 is a graph created based on the table of Fig. 6. These tables and graphs are created by the control device 100. The created tables and graphs are stored in the memory 102. Furthermore, the control device 100 reads out the tables and graphs stored in the memory 102 and displays them on the display unit 200, and distributes them via the Internet 30 to the personal computer 16 owned by the subject A, etc.

Referring to FIG. 6, the table shows the date and time when the shoulder joint angle of subject A was measured, the target angle α, subject A's angle αa, target strength β, subject A's strength βa, and the evaluation E given to subject A. "Target angle α" indicates the shoulder joint angle that was set in advance as a target in accordance with subject A's physical condition. "Target strength β" indicates the target value of strength for the target angle α.

"Angle of subject A" indicates the calculated shoulder joint angle of subject A. “Intensity of subject A” indicates the intensity of subject A calculated corresponding to “angle of subject A”. The "evaluation" is determined from among the seven levels shown in FIG. 5 based on the calculated joint angle of the shoulder of the subject A and the calculated strength of the subject A. The table shown in FIG. 6 shows an evaluation E determined based on the shoulder joint angle αa of the subject A and the strength βa of the subject A. The control device 100 adds data in the table shown in FIG. 6 every time the shoulder joint angle of the subject A is calculated. Hereinafter, the relationship between "target angle α" and "target intensity β" will be expressed as (α, β), and the relationship between "target person A's angle αa" and "target person A's intensity βa" will be expressed as (αa, βa ). Further, (αa, βa) is referred to as the measurement result of subject A.

The control device 100 creates the graph shown in FIG. 7 based on the data shown in FIG. The horizontal axis of the graph shows the angle, and the vertical axis shows the intensity. For the intensity on the vertical axis, the value at the point where the horizontal and vertical axes intersect is set to 0, and the upper value represents a positive value and the lower value represents a negative value. The intensity means that the larger the value, the more difficult (load) it is for the subject 20. Intensity = 0 indicates a normal intensity that the subject 20 does not find difficult or easy. Therefore, the strength of the negative value indicates the degree to which the subject 20 feels comfortable.

The graph in FIG. 7 shows a reference line Ls. The reference line Ls is a line used to evaluate the measurement results (αa, βa) of subject A. The reference line Ls is a line determined, for example, based on the standard physical ability of humans. Note that the reference line Ls may be set for each subject, taking into account the physical ability of each subject, and may also be determined taking into account gender and age. Data for multiple types of reference lines Ls is stored in the memory 102 of the control device 100.

The graph of FIG. 7 shows (α, β) corresponding to the target angle and target intensity of the subject A. (α, β) is on the reference line Ls, and the target intensity β is set to zero. Furthermore, (αa, βa) corresponding to the measurement result of subject A is shown in the graph.

The graph in Figure 7 shows that subject A's shoulder joint angle αa is short of the target angle α by an angle dlx. Furthermore, the reference line Ls on the graph shows that the standard strength for shoulder joint angle αa is βs. The standard strength βs is smaller than subject A's strength βa. The difference between strength βs and strength βa is dly. This means that subject A, who moved his shoulder at joint angle αa, felt a load on his shoulder that was dly heavier than the standard.

The control device 100 evaluates the physical ability of the subject A based on the reference line Ls passing through (α, β) and the data of the subject A (αa, βa). For example, if subject A's data (αa, βa) is located around (α, β), subject A is given a rating of D, which is located in the middle of the seven-level evaluation.

Figure 7 shows other data examples for subject A, including (αa1, βa1), (αa2, βa2), (αa3, βa3), and (αa4, βa4). (αa1, βa1) is an example where the joint angle is close to the target value α, but the strength exceeds the target value β. Anyone looking at this data example can infer that subject A is physically straining himself when moving his shoulder. When subject A's measurement result is (αa1, βa1), the control device 100 gives the subject a lower evaluation than when subject A's measurement result is (α, β).

(αa2, βa2) is an example where the joint angle is close to the target value α, but the intensity is below the target value β. Someone looking at this data example can infer that subject A achieved the target joint angle with an intensity lower than the target level of intensity. When subject A's measurement result is (αa2, βa2), the control device 100 gives the subject a higher evaluation than when subject A's measurement result is (α, β).

(αa3, βa3) is an example in which the joint angle significantly exceeds the target value α and the strength significantly exceeds the reference value βss. Further, (αa4, βa4) is an example in which the joint angle significantly exceeds the target value α, but the strength is significantly lower than the reference value βss. A person viewing these data can evaluate the physical ability of the subject A by considering the relationship between the reference line Ls, the joint angle, and the strength. Furthermore, the control device 100 evaluates the physical ability of the subject A based on the distance between each measurement result of the subject A and the reference line Ls.

Note that the graph shown in FIG. 7 may display the rank of evaluation by the control device 100. For example, it is conceivable to display "evaluation rank=E" next to (αa1, βa1) in the graph shown in FIG. However, even if such evaluation ranks are not displayed, those who view the graph can evaluate the physical ability of subject A from two aspects: joint angle and strength.
<Evaluation based on the subject's past measurement results>
FIG. 8 is a table showing past measurement results and current measurement results of subject A. FIG. 9 is a graph showing the relationship between the shoulder joint angle and strength of the subject A and the target angle and target strength, based on the past measurement results of the subject A. A method for evaluating the current physical ability of a subject A based on past measurement results of the subject A will be described with reference to FIGS. 8 and 9.

Figure 8 shows table T1 showing past data of subject A, and table T2 showing current data of subject A. Table T1 in Figure 8 shows some of the many measurement results of subject A in 2015. Table T2 in Figure 8 shows the measurement results of subject A in 2020. The measurement results of subject A shown in table T2 (αa1, βa1) are the same as the measurement results of subject A shown in Figure 6. However, the target angle, target intensity, and evaluation are different between table T2 in Figure 8 and Figure 6. This is because the evaluation criteria are different between the two.

Pa in FIG. 9 indicates a set of data of subject A recorded in 2015 (Δ in FIG. 9). By using the set Pa, the average shoulder joint angle α' and strength β' of the subject A in 2015 can be calculated. Here, the calculated α' is taken as the target angle, and the calculated β' is taken as the target intensity. (α', β') shown in FIG. 9 indicates the coordinate position.

Furthermore, based on set Pa, it is possible to calculate a reference line Ls' based on the measurement results of subject A in 2015. Figure 9 shows a reference line Ls' obtained by sliding the reference line Ls shown in Figure 8 to the right based on set Pa.

The target angle and target strength determined based on the past measurement results of subject A are (α', β'). From the graph shown in FIG. 9, it can be seen that subject A's shoulder joint angle αa is insufficient by an angle dlx' with respect to the target angle α'. Furthermore, from the reference line Ls', it can be seen that the standard strength for shoulder joint angle αa is βs'. The difference between strength βs' and strength βa is dly'. Comparing the graph shown in FIG. 7 with the graph shown in FIG. 9, "dlx<dlx'" and "dly<dly'". Reference line Ls' has a higher standard than reference line Ls. For this reason, the evaluation in table T2 is F, which is worse than the evaluation shown in FIG. 6.

In this way, the physical ability of subject A may be evaluated (1) based on the standard physical ability of a human, or (2) based on the past physical abilities of subject A. The control device 100 may store either evaluation criterion (1) or (2), or may store only one of them.
<Contents of control for evaluating physical ability>
10 is a flowchart showing the contents of control for evaluating the physical ability of the subject 20. A program corresponding to this flowchart is stored in the memory 102 of the control device 100. The processor 101 of the control device 100 executes the processing shown in the following flowchart based on the program stored in the memory 102.

First, the control device 100 reads an image (frame image) of the subject from a moving image of the subject stored in the memory 102 (step S1). Next, the control device 100 generates posture information of the subject 20 from the image read in step S1 using the trained model for position estimation (step S2). Thereby, for example, as shown in FIG. 2, the joint positions of the subject 20 are specified. Next, the control device 100 acquires biometric information corresponding to the image of the subject read in step S1 from the data stored in the memory 102 (step S3). The biological information is, for example, a detection value of the facial muscle sensor 14.

Next, the control device 100 analyzes the posture from the posture information generated in step S2 (step S4). For example, the control device 100 calculates the shoulder joint angle of the subject 20 in step S4. Next, the control device 100 analyzes the facial expression of the subject 20 using various AI techniques for facial expression estimation from the detected values of the facial muscle sensor 14 read in step S3 (step S5). For example, the control device 100 calculates the intensity (degree of harshness) from the facial expression of the subject 20 in step S4.

Next, the control device 100 compares the analyzed facial expression and the analyzed posture with a reference line (step S6). For example, the distance between the reference line Ls shown in FIG. 7 and the measurement results (αa, βa) of the subject 20 is calculated. Next, the control device 100 calculates an evaluation based on the result of step S6 (step S7). Next, the control device 100 stores the calculated evaluation (step S8). Next, the control device 100 creates an evaluation sheet for the subject 20 (step S9). The evaluation table includes, for example, the table shown in FIG. 6 or 8 and the graph shown in FIG. 7 or 9. Next, the control device 100 distributes the created evaluation sheet to the subject 20 (step S10). For example, the control device 100 transmits the evaluation sheet to the communication device 12 in the home of the subject 20 via the Internet 30.

In the first embodiment described above, an example has been described in which the physical ability of the subject is evaluated based on the joint angle of the subject's shoulder. However, the joint angle of the shoulder is only one example of what is to be evaluated. For example, the evaluation target may be the joint angle of the subject's knee.

In addition, instead of the joint angle, the joint velocity calculated based on the graph in FIG. 4 may be used as the subject of evaluation. In this case, in step S4 in FIG. 10, the control device 100 calculates the joint velocity of the subject's shoulder or the joint velocity of another part. Furthermore, in step S6 in FIG. 10, the control device 100 compares the joint velocity of the subject and the subject's strength with a reference line set for the joint velocity, to evaluate the physical ability of the subject.

According to the first embodiment, daily movements such as the degree of shoulder lift of a subject can be checked by estimating posture and facial expression, and deterioration or abnormality of the subject's body can be detected early. When the joint angle becomes smaller or the joint speed decreases, it is considered that the subject is experiencing abnormalities such as deterioration of the body. In addition, when the subject has a painful facial expression during a specific movement, there is a possibility that the subject is experiencing pain or abnormality in the joint. When such abnormalities are detected, the subject, his/her family, a doctor, etc. can be notified of the abnormality, enabling early detection of abnormalities and early intervention such as rehabilitation.
[Embodiment 2]
<Calculating Health Score>
Next, a second embodiment will be described. The first embodiment is a form in which the physical ability of a subject is evaluated by focusing on static or partial movements such as the joint angles or joint velocities of the subject. The second embodiment is a form in which the physical ability of a subject is evaluated based on continuous movements such as radio calisthenics.

First, the overall system configuration of the second embodiment will be described in comparison with the overall system configuration of the first embodiment shown in FIG. 1. Instead of the facial muscle sensor 14, a heart rate monitor 15 shown in FIG. 11 is attached to the subject 20. The heart rate measured by the heart rate monitor 15 is an example of biometric information. The heart rate measured by the heart rate monitor 15 is transmitted to the communication device 12 shown in FIG. 1. The communication device 12 transmits the heart rate to the control device 100 via a communication network such as the Internet 30.

For example, the subject 20 performs radio calisthenics at home 10 while following a video of the exercises. The movements of the subject 20 are captured by the camera 13. The camera 13 transmits the captured video to the communication device 12. The communication device 12 transmits the video received from the camera 13 to the control device 100 via a communication network such as the Internet 30. The control device 100 stores the heart rate and video of the subject 20 received from the communication device 12 in the memory 102. The memory 102 of the control device 100 stores data of the video of the radio calisthenics in advance. The video of the radio calisthenics includes the movements of a model exercise instructor performing the radio calisthenics.

The control device 100 evaluates the physical ability of the subject 20 based on the video of the subject 20, the heart rate of the subject 20, and the video of radio gymnastics, and transmits the evaluation result to the communication device 12. The results of the evaluation are displayed on the screen of the personal computer 16. The subject 20 uses the personal computer 16 to confirm the evaluation results.
<Evaluation of degree of match>
FIG. 11 is a diagram showing an example of a matching degree evaluation screen. The control device 100 evaluates the degree of agreement between the motions of the target person 20 and the motions of the gymnastics instructor serving as a model based on the video of the target person 20 and the radio gymnastics video, and transmits the evaluation results to the home of the target person 20. 10 communication devices 12. For example, the screen shown in FIG. 11 is displayed on the personal computer 16 of the subject 20. "Video of subject 20" is displayed on the left side of the screen, and "target video" is displayed on the right side of the screen. “Video of subject 20” is a video of one scene of a video of subject 20 doing radio gymnastics. The "target video" is a video corresponding to the "video of target person 20" among the videos of radio exercise. The ``target video'' shows the movements of a gymnastics instructor who serves as a role model. Between the "video of target person 20" and the "target video", an image in which the target person 20 included in the "video of target person 20" and the gymnastics instructor included in the "target video" are superimposed is displayed. has been done. By superimposing the image of the subject 20 and the image of the gymnastics instructor, it becomes easier to understand to what extent the movements of the subject 20 match those of the gymnastics instructor.

It is difficult to imagine that the physiques of Subject 20 and the model gymnastics instructor are completely identical. The size of the subject 20 in the image also changes depending on the distance and angle at which the subject 20 is photographed. For this reason, simply by superimposing the image of the subject 20 and the image of the gymnastics instructor, it is not possible to grasp the degree of coincidence between the movements of the two. Therefore, before superimposing the image of the subject 20 and the image of the gymnastics instructor, the control device 100 corrects the vertical and horizontal sizes of the image of the gymnastics instructor and the vertical and horizontal sizes of the image of the subject 20. For example, the control device 100 corrects the image of the subject 20 to match the length and width of the gymnastics instructor's body. As a result, when the image of the subject 20 and the image of the gymnastics instructor are superimposed, it becomes easier to understand to what extent the movements of the subject 20 match those of the gymnastics instructor.

As shown in FIG. 11, a matching score indicating the degree of matching is displayed below the image in which the subject 20 and the gymnastics instructor are superimposed. Furthermore, comments are displayed below the matching score. The subject 20 can refer to the comments and make an effort to improve the degree of matching.

Below the "image of subject 20", a display area for "heart rate" is provided. This display area displays the heart rate of subject 20 when performing the action shown in the "image of subject 20". Below the "target image", a display area for "target heart rate" is provided. This display area displays the target heart rate of subject 20. The target heart rate can be determined based on various criteria. For example, the target heart rate may be determined based on the standard physical ability of humans, or may be determined for each subject 20, taking into account the age and gender of subject 20. It is desirable to vary the target heart rate depending on the type of exercise from the start to the end of radio calisthenics. The memory 102 of the control device 100 stores the target heart rate for each of multiple timings from the start to the end of radio calisthenics.

By viewing the evaluation screen shown in FIG. 11, the subject 20 can grasp not only the degree of matching but also his or her own heart rate in comparison with the target heart rate. Therefore, the subject 20 can evaluate whether or not he/she is able to perform radio calisthenics well without straining himself/herself. Alternatively, if there is a trainer who instructs the subject 20, by viewing the same screen, the trainer can instruct the subject 20 in a preferred posture for radio calisthenics without straining the subject 20.
<Evaluation of symmetry>
FIG. 12 is a diagram showing an example of a symmetry evaluation screen. The control device 100 evaluates the left-right symmetry of the movement of the subject 20 based on the moving image of the subject 20. Radio calisthenics involves symmetrical movements. In response to bilaterally symmetrical gymnastics, it is desirable that the movements on the left and right sides of the subject 20 match. For example, in the case of a movement in which the body is bent laterally as shown in FIG. 12, it is desirable that the angle at which the body bends is the same on the right and left sides. However, the movements on the left side and the movements on the right side may not match due to body distortion or other factors.

Therefore, the control device 100 provides the subject 20 with a symmetry evaluation screen shown in FIG. The symmetry evaluation screen displays an image of the subject 20 bending his body to the left and an image of the subject 20 bending his body to the right. Furthermore, the symmetry score is displayed on the symmetry evaluation screen. The symmetry score is an evaluation score that indicates how symmetrical the left and right movements are. Further, the symmetry evaluation screen is provided with a display area for displaying the target heart rate and a display area for displaying the heart rate of the subject 20.

By looking at the evaluation screen shown in FIG. 12, the subject 20 can grasp not only the left-right symmetry of the radio calisthenics, but also his own heart rate in comparison with the target heart rate. Therefore, the subject 20 can evaluate whether or not he/she is able to perform radio calisthenics well without straining himself/herself. Alternatively, if there is a trainer who instructs the subject 20, by viewing the same screen, the trainer can instruct the subject 20 in a preferred posture for radio calisthenics without straining the subject 20.

In the evaluation screen shown in FIG. 12, an image of the subject 20 bending his/her body to the right may be superimposed on an image of the subject 20 bending his/her body to the left, and an image of the subject 20 bending his/her body to the left may be superimposed on an image of the subject 20 bending his/her body to the right. Also, one of the two images of the subject 20 bending his/her body to the left and right may be superimposed after being flipped left to right. This makes it easier for the person viewing the screen to visually understand the difference in the degree of bending between the left and right.

The method for calculating the symmetry score will be explained. The control device 100 extracts a radio gymnastics scene that requires bilateral symmetry from the video of the subject 20. The extracted radio gymnastics scene includes two paired images. The two images that form a pair are, for example, as shown in FIG. 12, an image in which the subject 20 is bent to the left and an image in which the subject 20 is bent to the right. The control device 100 estimates feature points (joints) of the subject 20 based on the first image of the two paired images. Further, the control device 100 generates an image obtained by horizontally inverting the second image of the two paired images, and estimates feature points (joints) of the subject 20 based on the generated inverted image. The control device 100 calculates the degree of coincidence between the estimation result based on the first image and the estimation result based on the inverted image by comparing them. Control device 100 determines a symmetry score based on the calculated degree of matching.
<Time series changes>
FIG. 13 is a first evaluation sheet showing the physical ability of the subject 20. The first evaluation sheet shows the relationship between the movement of the subject 20, the movement of the target image, and the heart rate of the subject 20. The first evaluation sheet is one of the data provided to the subject 20 by the control device 100. The horizontal axis of the first evaluation sheet shows the progress of radio gymnastics. The solid line waveform indicates the magnitude of the movement of the subject 20. The broken line waveform indicates the magnitude of the movement of the gymnastics instructor in the target video. The bar line indicates the heart rate of the subject 20. Note that the movement of the subject 20 may be measured at any part of the body. For example, it is possible to measure the movement of the subject 20 based on the movement of the shoulder joint.

The first evaluation sheet shown in FIG. 13 is provided as a video of the radio exercises from start to finish. FIG. 13 shows the state of the video at the time when the radio exercises end. The first evaluation sheet displays the heart rate of the subject 20 and the average heart rate of the subject 20. The heart rate value displayed on the first evaluation sheet changes as the radio exercises progress.

By looking at the first evaluation sheet, the subject 20 or the trainer instructing the subject 20 can understand to what extent the subject 20 is following the movements of the exercise instructor in relation to the progress of the radio exercises. In addition, the first evaluation sheet displays the heart rate and average heart rate, so it is possible to evaluate whether the subject 20 is performing the radio exercises without overexerting himself/herself. The first evaluation sheet may also display the target heart rate set for the subject 20.

Figure 14 is a second evaluation sheet showing the physical ability of the subject 20. The second evaluation sheet is one of the data provided to the subject 20 by the control device 100. The solid line is a line showing the reference of the radio-compatible s movement. The dashed line shows the magnitude of deviation of the subject's movement from the reference line. The movement targeted by the second evaluation sheet is the same as the movement targeted by the first evaluation sheet. However, since the second evaluation sheet shows the deviation of the subject's movement in relation to the reference line, it is easier to grasp the magnitude of deviation between the subject's movement and the movement of the target image than the first evaluation sheet. The second evaluation sheet is provided as a video, like the first evaluation sheet. Furthermore, the second evaluation sheet displays the heart rate, average heart rate, consistency score, symmetry score, exercise intensity score, and health score.

The exercise intensity score is calculated from the heart rate of the subject 20. For example, it is possible to calculate the exercise intensity from the heart rate of the subject 20 measured during radio exercise, the normal heart rate of the subject 20, and the age of the subject 20. The health score is a score comprehensively calculated from the concordance score, symmetry score, and exercise intensity score. Therefore, the higher the health score of subject 20, the more he performed radio calisthenics with movements similar to those of a gymnastics instructor, while maintaining left-right symmetry and without feeling any stress.

By looking at the second evaluation sheet shown in FIG. 14, the subject 20 or the trainer who instructs the subject 20 can more clearly grasp the discrepancy between the movements of the subject 20 and the exercise instructor. In addition, the second evaluation sheet displays a health score, which is a comprehensive score calculated from the agreement score, symmetry score, and exercise intensity score, so that the physical ability of the subject 20 can be comprehensively evaluated.

<Control flow>
15 and 16 are flowcharts showing the details of control for evaluating the physical ability of the subject 20. A program corresponding to this flowchart is stored in the memory 102 of the control device 100. The processor 101 of the control device 100 executes the processing shown in the flowchart below based on the program stored in the memory 102.

First, the control device 100 reads an image (frame image) of the subject 20 from a video of the subject 20 (step S101). Next, the control device 100 generates posture information of the subject 20 from the image read in step S101 using a trained model for position estimation (step S102). This allows the joint positions of the subject 20 to be identified, for example, as shown in FIG. 2.

Next, the control device 100 acquires biometric information corresponding to the image of the subject 20 read in step S101 (step S103). The biological information is, for example, heart rate. The heart rate is specified based on the measured value of the heart rate meter 15, for example. Alternatively, heart rate may be determined from the image using tactile vital sensing technology.

In this case, the heart rate of the subject 20 is determined using the frame image read in step S101 and multiple frame images that precede and follow the frame image in time. The pulse wave is related to the heart rate. The pulse wave changes in conjunction with the contraction of blood vessels, and the contraction of blood vessels is related to blood flow. Blood has the property of absorbing light. Therefore, the heart rate of the subject 20 can be indirectly determined by detecting changes in the light reflectance of the subject's 20 skin based on the image of the subject 20.

Next, the control device 100 analyzes the posture from the posture information generated in step S102 (step S104). For example, the control device 100 analyzes the exercise posture of the subject 20 in step S104. Next, the control device 100 analyzes the target image (step S105), and then corrects the posture of the subject 20 (step S106). Steps S105 and S106 are processes for matching the size of the gymnastics instructor in the target video with the size of the subject 20.

Next, the control device 100 sets the evaluation conditions (step S107). The evaluation conditions are, for example, conditions that determine the standards for calculating the matching score, the symmetry score, and the exercise intensity score. Depending on the setting of the evaluation conditions, the standards for each score can be made higher or lower.

For example, it is conceivable to change the criteria for calculating the matching score, symmetry score, and exercise intensity score depending on the age of the subject. As a result, when the subject is an elderly person, the criteria for each score can be set more leniently than when the subject is middle-aged or elderly. Further, criteria for newly calculating each score may be determined by taking into account the matching score, symmetry score, and exercise intensity score that the subject has acquired in the past.

Next, the control device 100 calculates the degree of matching based on the image of the subject 20 and the image of the model gymnastics instructor (step S108). Next, the control device 100 calculates the exercise intensity based on the heart rate read in step S102 (step S109), and stores the calculation result in the memory 102 (step S110). Next, the control device 100 displays the calculated exercise intensity on the display unit 200 (step S111).

Next, the control device 100 determines whether the radio calisthenics video has ended (step S112). If the radio calisthenics video has not ended, the control device 100 returns the process to step S101. If the radio calisthenics video has ended, the control device 100 calculates a similarity score (step S113), and then calculates an exercise intensity score (step S114).

By evaluating the similarity between the corrected posture information of the subject 20 and the correct posture information of the gymnastics instructor, the degree of match between the movements of the subject 20 and the movements of the gymnastics instructor can be calculated. The match score can be calculated using the following formula (2), where D is the sum of the difference values of each part of the two postures and D0 is the threshold value.

Note that it is also possible to use the confidence calculated together with posture information in OpenPose and add processing to information with low confidence that does not affect the matching score.

Next, the control device 100 extracts two paired images in order to calculate the symmetry score (step S115). Next, the control device 100 calculates a symmetry score based on the two extracted images (step S116). Next, the control device 100 calculates a health score from the matching score, symmetry score, and exercise intensity score (step S117). Next, the control device 100 creates an evaluation sheet (step S118), transmits the created evaluation sheet to the communication device 12 of the subject 20 (step S119), and ends the process.

The health score can be calculated by various calculation methods using the coincidence score, the symmetry score, and the exercise intensity score. For example, the health score may be calculated by adding up the coincidence score, the symmetry score, and the exercise intensity score. Alternatively, a predetermined weighting may be applied to each of the scores before adding them up.
[Variations]
In the first embodiment, the facial expression of the subject is analyzed based on the detection value of the facial muscle sensor 14. However, the facial expression of the subject may be analyzed from an image including the subject. In this case, it is possible to analyze the facial expression of the subject in step S4 using the image read in step S1 of the flowchart in FIG. 10. When analyzing the facial expression from an image, various AI technologies that estimate the facial expression based on the image may be adopted. If the facial expression can be analyzed based on the image, it is not necessary to obtain the detection value of the facial muscle sensor 14 in step S2. Note that the facial expression may be analyzed using the image and the detection value of the facial muscle sensor 14.

In the second embodiment, the exercise intensity was calculated based on the heart rate of the subject 20. However, in the second embodiment, the exercise intensity may be calculated by combining the heart rate and facial expression of the subject 20. The facial expression of the subject 20 may be estimated based on the detected value of the facial muscle sensor 14, or the facial expression of the subject 20 may be estimated from the frame image. Alternatively, the exercise intensity may be calculated based on the facial expression. For example, it is possible to calculate the exercise intensity based on the number of times the subject 20 shows a painful expression. The exercise intensity may be calculated based on the number of times the subject 20 shows a painful expression and the heart rate.

As described above, according to the physical ability presentation method and the physical ability presentation device according to each embodiment and modification, it is possible to present information that can appropriately evaluate the physical ability of the target person while taking into account the load on the target person. .

[Aspects]
It will be understood by those skilled in the art that the above-described embodiment and its modifications are specific examples of the following aspects.

(Section 1) A physical ability presentation method according to one aspect is a physical ability presentation method for presenting information regarding the physical ability of a target person, and includes a step (step) of generating posture information of the target person from an image including the target person. S2), a step of acquiring biological information of the subject (step S3), a step of identifying the posture of the subject based on the posture information (step S4), and a step of identifying the subject's posture based on the biometric information. The step of specifying the load of the subject (step S5), the posture of the subject specified based on the posture information, and the load of the subject corresponding to the posture of the subject specified based on the biological information are combined. and a step of presenting specific information regarding the physical ability of the subject (step S10).

According to the physical ability presentation method described in item 1, the physical ability of the subject can be appropriately evaluated while taking into account the load on the subject.

(2) In the physical ability presentation method described in 1, the specific information includes reference information (reference line Ls) for evaluating the subject's physical ability.

The physical ability presentation method described in paragraph 2 makes it possible to evaluate the subject's physical ability based on the reference information.

(Section 3) In the physical ability presentation method described in Section 2, the specific information includes an evaluation score (evaluation, matching score, symmetry score, exercise intensity score, health score) of the subject.

According to the physical ability presentation method described in Section 3, the physical ability of the subject can be evaluated based on the evaluation score.

(Paragraph 4) In the physical ability presentation method according to any one of Paragraphs 1 to 3, the step of acquiring posture information includes the posture based on the subject's movements (joint angles, joint speeds, radio gymnastics). including the step of obtaining information.

According to the physical ability presentation method described in item 4, it is possible to evaluate the physical ability of the target person regarding the target person's movements.

(5) In the physical ability presentation method described in any one of paragraphs 1 to 4, the biometric information is information about the subject's facial expression (stress).

According to the physical ability presentation method described in Section 5, it is possible to identify the load of the subject based on the subject's facial expression.

(Section 6) In the physical ability presentation method described in Section 5, the step of acquiring biometric information includes a step of acquiring information about the subject's facial expression based on the detection value of a facial muscle sensor (facial muscle sensor 14) attached to the subject's face or based on an image (Step S5).

The physical ability presentation method described in paragraph 6 makes it possible to appropriately obtain information regarding the subject's facial expressions.

(Section 7) In the physical ability presentation method according to any one of Items 1 to 4, the biological information is information regarding the subject's heartbeat (heart rate).

According to the physical ability presenting method described in item 7, the load on the subject can be identified from the subject's heartbeat.

(Section 8) In the physical ability presentation method described in Section 7, the step of acquiring biological information is based on a measured value of a heart rate monitor (heart rate monitor 15) attached to the subject or based on an image. , including the step of acquiring information regarding the subject's heartbeat (step S103).

According to the physical ability presentation method described in Section 8, the subject's heartbeat can be appropriately acquired.

(Clause 9) In the physical ability presentation method described in any one of clauses 1 to 8, the specific information includes multiple past posture information and biometric information of the subject (Figure 8).

The physical ability presentation method described in paragraph 9 makes it possible to identify multiple past posture information and biometric information of the subject.

(10) In the physical ability presentation method described in any one of paragraphs 1 to 6, the posture information includes information on the angles of the subject's joints (Figures 2 and 3).

According to the physical ability presenting method described in item 10, it is possible to evaluate the physical ability of the subject regarding the angle of the joints.

(Section 11) In the physical ability presentation method according to any one of Items 1 to 6, the posture information includes information (FIG. 4) regarding the velocity of the subject's joints.

According to the physical ability presentation method described in Section 11, it is possible to evaluate the physical ability of a subject regarding joint speed.

(Paragraph 12) In the physical ability presentation method according to any one of Paragraphs 1 to 3, the posture information is obtained from the subject when the subject performs a bilaterally symmetrical movement. It includes posture information (Fig. 12: symmetry).

The physical ability presentation method described in paragraph 12 makes it possible to evaluate the subject's physical ability in terms of bilaterally symmetrical movements.

(Section 13) A physical ability presentation device according to another aspect is a physical ability presentation device (control device 100) that presents information regarding the physical ability of a subject, and the posture information of the subject is obtained from an image including the subject. a generation unit that generates (step S2), an acquisition unit that acquires biological information of the subject (step S3), a posture identification unit that specifies the posture of the subject based on posture information (step S4), A load identification unit (step S5) that identifies the load on the subject with respect to the subject's posture based on and a presentation unit (step S10) that presents specific information regarding the physical ability of the user.

According to the physical ability presenting device described in item 13, the physical ability of the subject can be appropriately evaluated while taking into account the load on the subject.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the description of the embodiments described above, and it is intended that all changes within the meaning and range equivalent to the claims are included.

10 House, 12 Communication device, 13 Camera, 14 Facial muscle sensor, 15 Heart rate monitor, 16 Personal computer, 20 Subject, 30 Internet, 100 Control device, 101 Processor, 102 Memory, 103 Communication unit, 200 Display unit.

Claims (10)

A physical ability presentation method for presenting information regarding a target person's physical ability, the method comprising:
generating posture information of the subject from an image including the subject;
acquiring biometric information of the subject;
identifying the posture of the subject based on the posture information;
identifying a load on the subject relative to the subject's posture based on the biological information ;
A combination of the posture of the subject specified based on the posture information and the load of the subject with respect to the posture of the subject specified based on the biological information is used to identify the physical ability of the subject. presenting the information ;
The biological information is acquired by a facial muscle sensor attached to the subject's face or a measured value of a heart rate monitor attached to the subject . The physical ability presentation method according to claim 1, wherein the specific information includes reference information for evaluating the subject's physical ability. The physical ability presentation method according to claim 2, wherein the specific information includes an evaluation score of the subject. The physical ability presentation method according to any one of claims 1 to 3, wherein the step of acquiring the posture information includes the step of acquiring the posture information from the movement of the subject. The physical ability presentation method according to claim 1 , wherein the specific information includes a plurality of pieces of posture information and a plurality of pieces of biometric information of the subject in the past. The physical ability presenting method according to any one of claims 1 to 5, wherein the posture information includes information regarding angles of joints of the subject. The physical ability presentation method according to claim 1 , wherein the posture information includes information regarding velocities of the subject's joints. The physical ability presentation method according to any one of claims 1 to 3, wherein the posture information includes posture information obtained from the subject when the subject is made to perform symmetrical movements. A physical ability presentation device that presents information regarding a subject's physical ability,
a generation unit that generates posture information of the target person from an image including the target person;
an acquisition unit that acquires biological information of the subject;
a posture identification unit that identifies the posture of the subject based on the posture information;
a load identifying unit that identifies a load on the subject relative to the subject's posture based on the biological information ;
a presenting unit that presents specific information regarding the physical ability of the subject by combining the posture specified by the posture specifying unit and the load specified by the load specifying unit ;
The biological information is acquired by a facial muscle sensor attached to the subject's face or a measured value of a heart rate monitor attached to the subject . A physical ability presentation method for presenting information regarding a target person's physical ability, the method comprising:
generating posture information of the subject from an image including the subject;
acquiring biometric information of the subject;
identifying the posture of the subject based on the posture information;
identifying a load on the subject relative to the subject's posture based on the biological information;
A combination of the posture of the subject specified based on the posture information and the load of the subject with respect to the posture of the subject specified based on the biological information is used to identify the physical ability of the subject. presenting the information;
The specific information includes reference information for evaluating the physical ability of the subject,
The reference information is information indicating a relationship between a posture of the subject and a load of the subject with respect to the posture of the subject.

Images