JP2023040882A - Terminal device, information processing method, information processing system, and program - Google Patents

Abstract

To efficiently evaluate a motion of a subject's body while notifying a user photographing a subject whether a proper moving image has been photographed or not.SOLUTION: A terminal device (1) comprises: a display unit (19) which superimposes the skeleton of a subject extracted by using a first algorithm from a moving image including the subject on the subject and displays a result of this; and a derivation unit (17) which refers to the skeleton of the subject extracted by using a second algorithm and derives parameters related to the motion of the subject's body.SELECTED DRAWING: Figure 1

Description

The present invention relates to a terminal device, an information processing method, an information processing system, and a program.

Among techniques for analyzing a subject's gait from a moving image including a walking subject, there is known a technique for increasing the accuracy of analysis. Patent Literature 1 discloses a terminal that acquires situation information including information indicating the shooting situation, which is the internal situation of the shooting device at the time of shooting, and outputs a composite image that includes the situation image representing the situation information in the shot image. ing.

Japanese Patent No. 6895669

By the way, there is also known a technique for evaluating the walking of a subject by analyzing the gait. In this technology, it is known to superimpose and display the extracted skeleton of the target person on the target person in order to notify the user whether or not the captured moving image is suitable for gait analysis. Here, if skeleton extraction for gait analysis is performed during photographing, the speed of skeleton extraction processing is required, and the accuracy of skeleton extraction becomes low, making it impossible to appropriately evaluate the gait of the subject. On the other hand, when skeleton extraction is performed after photographing, there is a problem that it is not possible to notify the user photographing the subject whether or not a suitable moving image for gait analysis has been photographed.

An object of one aspect of the present invention is to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

In order to solve the above problems, a terminal device according to an aspect of the present invention extracts a skeleton of a target person using a first algorithm from a moving image including the target person as a subject. With reference to the skeleton of the subject extracted from the moving image using a display unit that is superimposed on the subject and one or more second algorithms different from the first algorithm, the subject's skeleton and a derivation unit for deriving parameters related to body motion.

According to one aspect of the present invention, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

It is a block diagram which shows the structural example of the terminal device which concerns on Embodiment 1 of this invention. 4 is a sequence diagram showing the flow of processing by the terminal device according to Embodiment 1 of the present invention; FIG. FIG. 4 is a diagram showing an example of a moving image displayed on the display panel by the display unit according to Embodiment 1 of the present invention; FIG. 5 is a diagram showing another example of a moving image displayed on the display panel by the display unit according to Embodiment 1 of the present invention; FIG. 10 is a diagram showing still another example of a moving image displayed on the display panel by the display unit according to Embodiment 1 of the present invention; FIG. 10 is a diagram showing still another example of a moving image displayed on the display panel by the display unit according to Embodiment 1 of the present invention; FIG. 10 is a diagram showing still another example of a moving image displayed on the display panel by the display unit according to Embodiment 1 of the present invention; It is a schematic diagram which shows the outline|summary of the information processing system which concerns on Embodiment 2 of this invention. 8 is a block diagram showing a configuration example of a terminal device and an information processing device included in the information processing system according to Embodiment 2 of the present invention; FIG. FIG. 9 is a sequence diagram showing the flow of processing in the information processing system according to Embodiment 2 of the present invention; 1 is a block diagram illustrating a physical configuration of a computer used as a terminal device and an information processing device; FIG. 4 is a graph showing the relationship between estimated values and measured values in Example 1. FIG. 9 is a graph showing the relationship between estimated values and measured values in Example 2. FIG.

[Embodiment 1]
An embodiment of the present invention will be described below.

(Overview of terminal device 1)
A configuration example of a terminal device 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration example of a terminal device 1 according to this embodiment.

The terminal device 1 according to the present embodiment shoots a moving image including a target person as a subject, refers to the moving image, and performs the following processing.
・Extract the target person's skeleton and display the extracted target person's skeleton superimposed on the target person in the moving image ・Deriving parameters related to the target person's movement ・Deriving accuracy information indicating the accuracy of the derived parameters Here, although a cared person is mentioned as an example of the subject, this does not limit the present embodiment. Further, the user of the terminal device 1 may be a caregiver who takes care of the care recipient, but this does not limit the present embodiment.

(Configuration of terminal device 1)
The terminal device 1 includes a control unit 10, a display panel 12, an imaging unit 13, and a storage unit 14, as shown in FIG.

The display panel 12 is a display that displays data supplied from the control unit 10, which will be described later. As an example, the display panel 12 displays a moving image in which the target person's skeleton is superimposed on the target person.

The imaging unit 13 is a camera capable of capturing moving images. As an example, the imaging unit 13 is directed by the user so that the angle of view of the imaging unit 13 includes the subject. Then, the imaging unit 13 captures an image of the subject performing a predetermined action (walking, arm-raising, etc.), and generates a moving image including the subject as a subject. The imaging unit 13 supplies moving images obtained by imaging to the control unit 10, which will be described later.

The storage unit 14 stores various data referred to by the control unit 10, which will be described later. As an example, the storage unit 14 stores a moving image captured by the imaging unit 13, bone structure information indicating a target person's bone structure, which will be described later, and the like.

(control unit 10)
The control unit 10 controls each component of the terminal device 1 . As an example, the control unit 10 acquires moving images captured by the imaging unit 13 and supplies moving image data to be displayed on the display panel 12 .

The control unit 10 also functions as an acquisition unit 15, a skeleton extraction unit 16, a parameter derivation unit 17 (derivation unit), a probability derivation unit 18, and a display unit 19, as shown in FIG.

Acquisition unit 15 acquires the moving image captured by imaging unit 13 . Acquisition unit 15 stores the acquired moving image in storage unit 14 .

The skeleton extraction unit 16 uses a first algorithm and one or a plurality of second algorithms to execute skeleton extraction processing for extracting a skeleton of a subject from a moving image including the subject as a subject.

Here, the first algorithm and the second algorithm are algorithms for extracting the subject's skeleton. The skeleton of the subject extracted by the first algorithm is referred to in real time by the display unit 19, which will be described later, as first skeleton information indicating the skeleton. Therefore, it is preferable that the first algorithm be an algorithm that performs skeleton extraction processing faster than the second algorithm, which will be described later. More specifically, the first algorithm is preferably faster than the video refresh rate (30 fps or 60 fps). Also, the first algorithm generates first accuracy information indicating the accuracy of the extracted skeleton. The first accuracy information is referred to by the display section 19, which will be described later.

The result of the target person's skeleton extraction process extracted by the second algorithm is referred to by the parameter derivation unit 17, which will be described later, as second skeleton information indicating the skeleton. Although the details will be described later, it is preferable that the skeleton extraction processing by one or more of the second algorithms has higher accuracy than the skeleton extraction processing by the first algorithm.

Further, the skeleton extraction unit 16 is configured to select one or more second algorithms from a plurality of algorithm candidates and extract the skeleton of the subject using the selected one or more second algorithms. good too. For example, the skeleton extraction unit 16 may refer to the skeleton of the subject extracted using the first algorithm and select one or more second algorithms. A specific example of the processing in which the skeleton extraction unit 16 selects one or more second algorithms will be described later.

Further, the skeleton extracting unit 16, when extracting the skeleton using at least one of the first algorithm and the second algorithm, stores error information indicating a frame for which skeleton extraction has failed among the frames included in the moving image. Generate.

Frames for which skeleton extraction failed are frames for which skeletons were not extracted, or frames whose skeletons were not properly extracted during skeleton extraction using the first algorithm or the second algorithm. . As a concrete example,
- A frame for which the return value of the skeleton extraction process could not be acquired because the skeleton extraction process was not completed - The skeleton extraction process was completed, but the return value of the skeleton extraction process was a value outside the specified range (outlier) Examples include, but are not limited to, frames that have been used.

As an example of error information,
・Information indicating the number of frames for which skeleton extraction has failed ・Indicates the temporal position of the frame for which skeleton extraction has failed (the date and time when the frame for which skeleton extraction failed was captured, or the elapsed time from the start of moving image shooting, etc.) Information/In a frame for which skeleton extraction has failed, the information may include, but is not limited to, information indicating a region for which skeleton extraction could not be performed.

The parameter derivation unit 17 refers to the results of the skeleton extraction processing executed by the skeleton extraction unit 16 to derive parameters related to the motion of the subject's body. More specifically, the parameter derivation unit 17 refers to the second skeleton information indicating the skeleton of the subject extracted from the moving image by the skeleton extraction unit 16 using the second algorithm, and calculates the body motion of the subject. Derive the parameters for A parameter related to the motion of the subject's body is a parameter that indicates to what extent the motion was performed by the subject when the motion was performed. Examples of parameters related to the movement of the subject's body include, when the subject performs a movement such as walking, raising an arm, or standing up; Examples include, but are not limited to, left-right difference in motion. A specific example of the process of deriving parameters by the parameter derivation unit 17 will be described later.

The accuracy derivation unit 18 derives second accuracy information indicating the accuracy of the parameter derived by the parameter derivation unit 17, according to the frames for which the skeleton extraction by the skeleton extraction process has failed among the plurality of frames included in the moving image. . More specifically, the accuracy derivation unit 18 derives second accuracy information indicating the accuracy of the parameters derived by the parameter derivation unit 17 by referring to the error information generated by the skeleton extraction unit 16 . The accuracy of a parameter is the degree of certainty of the parameter, and the higher the accuracy of the parameter, the higher the reliability of the parameter. A specific example of the process of deriving the second accuracy information by the accuracy deriving unit 18 will be described later.

The display unit 19 generates moving image data to be displayed on the display panel 12 . Specifically, the display unit 19 refers to the first skeleton information generated by the skeleton extraction unit 16, and superimposes the skeleton indicated by the first skeleton information on the target person in the moving image acquired by the acquisition unit 15. to display on the display panel 12. The display unit 19 also refers to the second accuracy information derived by the accuracy derivation unit 18 and displays the accuracy indicated by the second accuracy information on the display panel 12 .

(Processing by terminal device 1)
Processing by the terminal device 1 will be described with reference to FIG. FIG. 2 is a sequence diagram showing the flow of processing by the terminal device 1 according to this embodiment.

(Step S12)
In step S<b>12 , the acquiring unit 15 starts acquiring a moving image including the target person as a subject via the imaging unit 13 . For example, when the acquiring unit 15 acquires information indicating that shooting is to be started from an operation accepting unit (not shown) that accepts an operation from the user of the terminal device 1, the acquiring unit 15 starts acquiring moving images via the imaging unit 13. do.

(Step S13)
In step S13, the skeleton extraction unit 16 extracts the skeleton of the subject using the first algorithm from the moving image acquired by the acquisition unit 15 in step S12. The skeleton extraction unit 16 supplies first skeleton information indicating the extracted skeleton to the display unit 19 . Further, in step S13, the skeleton extracting unit 16 may generate first accuracy information indicating the likelihood of the extracted skeleton and error information indicating frames for which skeleton extraction has failed. In this case, the skeleton extraction unit 16 supplies the first accuracy information to the display unit 19 and supplies the error information to the accuracy derivation unit 18 .

(Step S14)
In step S14, the display unit 19 supplies to the display panel 12 moving image data representing a moving image in which the skeleton indicated by the first skeleton information supplied by the skeleton extracting unit 16 in step S13 is superimposed on the subject in the moving image. . The display panel 12 displays the moving image indicated by the acquired moving image data.

(Example 1 of moving image displayed by display unit 19)
An example of a moving image displayed on the display panel 12 by the display unit 19 will be described with reference to FIG. FIG. 3 is a diagram showing an example of a moving image displayed on the display panel 12 by the display unit 19 according to this embodiment.

The display unit 19 superimposes the joints of the skeleton indicated by the first skeleton information on the subject and displays them on the display panel 12 . For example, as shown in FIG. 3, the display unit 19 superimposes points JT1, JT2, and JT3 indicating joints on the subject TP and displays them on the display panel 12 . In addition, the display unit 19 connects adjacent joints (for example, points JT1 and JT2 in FIG. 3) with a line and displays them on the display panel 12 while superimposing them on the subject TP. In other words, the display unit 19 does not display the unextracted skeleton on the display panel 12 in skeleton extraction using the first algorithm. With this configuration, the terminal device 1 can notify the user who is shooting the target person whether or not the moving image being shot is suitable for extracting the skeleton of the target person.

Here, in order to notify the user whether or not the skeleton of the target person TP included in the captured moving image has been extracted, the faster the skeleton extraction process, the better. Therefore, as described above, the first algorithm for extracting the skeleton indicated by the first skeleton information is preferably an algorithm with faster extraction processing than the second algorithm. With this configuration, the terminal device 1 can notify the user who is shooting whether or not the moving image being shot is suitable for extracting the skeleton of the target person.

(Example 2 of moving image displayed by display unit 19)
When the skeleton extracting unit 16 supplies the first accuracy information to the display unit 19, the display unit 19 acquires the first accuracy information and, according to the acquired first accuracy information, extracts a skeleton superimposed on the subject. may be configured to change the display mode of. An example of a moving image displayed on the display panel 12 by the display unit 19 in this configuration will be described with reference to FIG. FIG. 4 is a diagram showing another example of a moving image displayed on the display panel 12 by the display unit 19 according to this embodiment.

As shown in FIG. 4, the display unit 19 superimposes points JT2, JT3, and JT4 indicating joints on the subject TP and displays them on the display panel 12, as in FIG. Here, if the accuracy indicated by the first accuracy information for the left knee of the subject TP is lower than a predetermined value, the display unit 19 changes the aspect of the point JT4 indicating the left knee to another aspect, as shown in FIG. The display mode (circular shape) showing the joints is changed to a different mode (triangular shape) and displayed. With this configuration, the terminal device 1 can notify the user who is shooting which skeleton of the target person in the moving image being shot has been extracted with high accuracy.

In addition to the configuration in which the display unit 19 has different display modes, for example,
- The color of the skeleton display is changed according to the first accuracy information. - The display color of the skeleton is changed in shade according to the first accuracy information. - The skeleton is shown according to the first accuracy information. Examples include, but are not limited to, different display sizes.

(Example 3 of moving image displayed by display unit 19)
When the skeleton extraction unit 16 supplies the first accuracy information to the display unit 19, and the accuracy indicated by the first accuracy information is lower than a predetermined value, the display unit 19 shoots the subject so as to increase the accuracy. The configuration may be such that a display prompting the user to do so is displayed. An example of a moving image displayed on the display panel 12 by the display unit 19 in this configuration will be described with reference to FIG. FIG. 5 is a diagram showing still another example of the moving image displayed on the display panel 12 by the display unit 19 according to this embodiment.

For example, if the accuracy indicated by the first accuracy information is low for all or part of the skeleton because the target person included in the moving image is small, the display unit 19 displays the accuracy as shown in FIG. A display DP1 prompting to photograph the subject is displayed. The display DP1 includes the text "Please take a picture so that the target person appears a little larger". With this configuration, the terminal device 1 can notify the user who is shooting how to shoot to increase the accuracy of extracting the skeleton of the target person.

As an example of a display prompting you to shoot the subject so that the accuracy is high,
・Because the target person included in the moving image is small, if the accuracy indicated by the first accuracy information is low for all skeletons, “Please narrow the angle of view” is displayed. ・The target person included in the moving image is on the left side. Therefore, when the accuracy indicated by the first accuracy information for the skeleton on the left side of the subject is low, a message such as "Please shoot the left side a little more" is displayed, but is not limited to these.

Further, even if the display unit 19 is configured to display a display prompting the subject to be photographed so as to increase the accuracy when the number of body parts extracted by the skeleton extraction unit 16 is smaller than a predetermined number. good. More specifically, the display unit 19 refers to the parts indicated by the first skeleton information supplied by the skeleton extraction unit 16, and if the number of parts is smaller than a predetermined number, the accuracy is increased. A display prompting the user to photograph the subject may be provided.

With reference to FIG. 2 again, the processing after step S15 of the terminal device 1 will be described.

(Step S15)
In step S<b>15 , the acquisition unit 15 finishes acquisition of moving images via the imaging unit 13 . For example, when the acquiring unit 15 acquires information indicating that the shooting is to end from an operation accepting unit (not shown) that accepts an operation from the user of the terminal device 1, the acquisition unit 15 acquires the moving image via the imaging unit 13. finish.

(Step S16)
In step S16, the skeleton extraction unit 16 extracts the skeleton of the subject from the moving image acquired by the acquisition unit 15 in steps S12 to S15 using one or more second algorithms different from the first algorithm. do. In step S16, the skeleton extraction unit 16 may select one or a plurality of second algorithms from among the plurality of algorithm candidates, and extract the skeleton of the subject using the selected one or a plurality of second algorithms. . If the skeleton extraction unit 16 does not generate error information in step S13, the skeleton extraction unit 16 may generate error information in step S16.

(Example of Second Algorithm Selection Processing by Skeleton Extraction Unit 16)
A second example of algorithm selection processing by the skeleton extraction unit 16 will be described below.

In a configuration in which the skeleton extraction unit 16 selects one or a plurality of second algorithms from a plurality of algorithm candidates, the plurality of algorithm candidates have different characteristics such as a site to be extracted, extraction accuracy, and extraction speed. good too. In this case, the skeleton extraction unit 16 may refer to the skeleton of the subject extracted using the first algorithm and select an algorithm according to the characteristics.

As an example, when a plurality of algorithm candidates extract different skeletal parts, the skeleton extracting unit 16 selects one or more second algorithms from the plurality of algorithm candidates according to the skeletal parts to be extracted. In this case, the skeleton extraction unit 16 refers to the skeleton of the subject extracted by the first algorithm, and selects an algorithm capable of extracting the parts of the skeleton extracted by the first algorithm from among the plurality of algorithm candidates. select. For example, when the leg of the subject is extracted by the first algorithm, the skeleton extraction unit 16 selects an algorithm capable of extracting the leg from a plurality of algorithm candidates.

Further, the skeleton extraction unit 16 may select one or more second algorithms from multiple algorithm candidates depending on the period. For example, when the subject's legs are extracted by the first algorithm in a certain period, and the subject's arms are extracted by the first algorithm in a period different from the certain period, the skeleton extraction unit 16 extracts a certain An algorithm capable of extracting a leg is selected from a plurality of candidate algorithms for a period, and an algorithm capable of extracting an arm is selected from a plurality of candidate algorithms for a different period.

Further, the skeleton extraction unit 16 may select one or a plurality of second algorithms from a plurality of algorithm candidates with reference to instructions from the user. For example, if the moving image includes a period during which the walking speed is accelerated and a period during which the walking speed is decelerated, the user can specify an acceleration period for each of the period during which the walking speed is accelerated and the period during which the walking speed is decelerated. An annotation indicating that there is a deceleration period and an annotation indicating that it is a deceleration period are added to the moving image. The skeleton extraction unit 16 refers to the annotation, selects one or a plurality of second algorithms suitable for the movement in which the walking speed accelerates during the period when the walking speed accelerates, and during the period when the walking speed slows down, It may be configured to select one or a plurality of second algorithms that are suitable for motions in which the walking speed slows down. Alternatively, the skeleton extraction unit 16 may be configured such that the user directly instructs which one or a plurality of second algorithms to select for the period and region.

By selecting the second algorithm by the skeleton extracting unit 16 in this manner, the skeleton of the subject can be extracted with high accuracy.

(Step S17)
In step S17, the parameter deriving unit 17 refers to the subject's skeleton extracted in step S16, and derives parameters relating to the body motion of the subject.

(Example of parameter derivation processing by parameter derivation unit 17)
An example of parameter derivation processing by the parameter derivation unit 17 will be described below.

For example, the parameter deriving unit 17 refers to the skeleton of the subject who is walking, which is extracted using the second algorithm, and derives parameters relating to walking of the subject. More specifically, the second skeleton information acquired from the skeleton extraction unit 16 indicates that the subject's right and left legs have been extracted, and the right and left legs indicate that the subject is walking. , the parameter deriving unit 17 calculates the walking speed of the subject. A parameter derivation unit 17 derives a parameter according to the calculated speed. For example, when the subject's walking speed is higher than a predetermined value, the parameter deriving unit 17 sets the walking speed parameter to a parameter indicating that the walking speed is fast. On the other hand, when the subject's walking speed is lower than the predetermined value, the parameter deriving unit 17 sets the walking speed parameter to a parameter indicating that the walking speed is slow.

As another example, when the second skeleton information indicates that the subject's right and left legs have been extracted, and the right and left legs indicate that the subject is walking, the parameter deriving unit 17 calculates the speed at which the subject's right and left legs respectively move. A parameter derivation unit 17 derives a parameter according to the calculated speed. For example, if the difference between the movement speed of the subject's right leg and the movement speed of the subject's left leg is smaller than a predetermined value, the parameter derivation unit 17 sets the left-right difference parameter to indicate that there is no left-right difference. Set to parameter. On the other hand, when the difference between the movement speed of the subject's right leg and the movement speed of the subject's left leg is greater than the predetermined value, the parameter derivation unit 17 sets the left-right difference parameter to indicate that there is a left-right difference. Set to parameter.

In this way, the terminal device 1 refers to the second skeleton information indicating the skeleton extracted by the second algorithm, and derives the parameters related to the motion of the subject's body. That is, the skeleton extracted by the second algorithm is used for deriving parameters related to the motion of the subject's body. It is preferable that the accuracy is higher than With this configuration, the terminal device 1 refers not to the first skeleton information indicating the skeleton extracted by the first algorithm, but to the second skeleton information indicating the skeleton extracted by the second algorithm, so that the subject person It is possible to improve the accuracy of the parameters related to the movement of the body.

(Step S18)
In step S18, the accuracy derivation unit 18 refers to the error information generated by the skeleton extraction unit 16, and obtains the second accuracy information indicating the accuracy of the parameter derived in step S17 from among the plurality of frames included in the moving image. , is derived according to the frame for which the skeleton extraction by the skeleton extraction process has failed.

(Example of accuracy derivation processing by accuracy derivation unit 18)
An example of accuracy derivation processing by the accuracy derivation unit 18 will be described below.

By referring to the error information generated by the skeleton extraction unit 16, the accuracy derivation unit 18 identifies frames for which skeleton extraction by the skeleton extraction process has failed, and derives second accuracy information according to the identified frames.

As an example, when the error information indicates the number of frames for which skeleton extraction by skeleton extraction processing has failed, the accuracy deriving unit 18 derives second accuracy information according to the number of frames for which skeleton extraction by skeleton extraction processing has failed. do. For example, when the number of frames for which skeleton extraction by the skeleton extraction process fails is greater than a predetermined value, the accuracy deriving unit 18 derives second accuracy information indicating that the accuracy is low. . On the other hand, if the number of frames for which skeleton extraction by the skeleton extraction process has failed is less than the predetermined value, the accuracy derivation unit 18 derives second accuracy information indicating that the accuracy is high as second accuracy information. . With this configuration, the terminal device 1 can notify the user that the accuracy of the parameter is low when there are a large number of frames for which skeleton extraction by skeleton extraction processing has failed.

As another example, when the error information indicates the temporal position of the frame for which the skeleton extraction by the skeleton extraction process has failed, the accuracy derivation unit 18 calculates the time position of the frame for which the skeleton extraction by the skeleton extraction process has failed. to derive the second accuracy information. For example, if the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails is included in the period during which the target person is moving (for example, if the target person is walking video, the target person is walking period), the accuracy deriving unit 18 derives second accuracy information indicating that the accuracy is low as the second accuracy information. On the other hand, if the temporal position of the frame for which skeleton extraction by the skeleton extraction process has failed is not included in the period during which the target person is moving (for example, if the target person is walking video, before the target person walks ( or after walking), the accuracy deriving unit 18 derives second accuracy information indicating that the accuracy is high as the second accuracy information. With this configuration, the terminal device 1 can notify the user that the accuracy of the parameters is low when skeleton extraction fails while the subject is in motion.

As yet another example, if the error information indicates a region for which no skeleton was extracted in a frame in which skeleton extraction by skeleton extraction processing failed, the accuracy deriving unit 18 may, in the frame in which skeleton extraction by skeleton extraction processing failed, Second accuracy information is derived according to the part where the skeleton has not been extracted. For example, in a frame for which skeleton extraction by the skeleton extraction process has failed, the part for which the skeleton was to be extracted is an essential part for the target's movement (for example, if the target's walking video, the target's leg part), the accuracy deriving unit 18 derives second accuracy information indicating that the accuracy is low as the second accuracy information. On the other hand, in the frames for which the skeleton extraction by the skeleton extraction process failed, the parts whose skeletons were to be extracted are parts that are not essential for the movement of the subject (for example, if the subject's walking video is neck), the accuracy derivation unit 18 derives second accuracy information indicating that the accuracy is high as the second accuracy information. With this configuration, the terminal device 1 can notify the user that the accuracy of the parameters is low when extraction of the skeleton of the body part essential for the movement of the subject fails.

Referring to FIG. 2 again, step S19 of the processing by the terminal device 1 will be described.

(Step S19)
In step S19, the display unit 19 displays the moving image data including the parameters derived by the parameter deriving unit 17 in step S17 and the accuracy indicated by the second accuracy information derived by the accuracy deriving unit 18 in step S19 on the display panel 12. supply. The display panel 12 displays the moving image indicated by the acquired moving image data.

(Example 4 of moving image displayed by display unit 19)
An example of a moving image displayed on the display panel 12 by the display unit 19 will be described with reference to FIG. FIG. 6 is a diagram showing still another example of the moving image displayed on the display panel 12 by the display unit 19 according to this embodiment.

As shown in FIG. 6, the display unit 19 displays on the display panel 12 a moving image including the parameter PM1 derived by the parameter deriving unit 17 and the accuracy AC1 indicated by the second accuracy information derived by the accuracy deriving unit 18. . Moreover, the display unit 19 may display a moving image including the parameter PM2 for each evaluation item on the display panel 12 . With this configuration, the terminal device 1 can notify the user of the accuracy of the displayed parameter PM1.

Further, the display unit 19 may be configured to display the accuracy indicated by the second accuracy information for each evaluation item. In this configuration, the accuracy deriving unit 18 derives the second accuracy information for each evaluation item. For example, if the moving image includes a motion of the target person walking and a motion of the target person standing still and raising their arms, the skeleton extraction process fails during the period when the target person is performing the walking motion. If the number of frames is greater than the predetermined number, the accuracy deriving unit 18 derives the second accuracy information so that the accuracy of the evaluation item "speed" evaluated in the walking motion of the subject is low.

(Example 5 of moving image displayed by display unit 19)
The display unit 19 may be configured to change the mode of displaying the accuracy according to the accuracy indicated by the second accuracy information derived by the accuracy deriving unit 18 . An example of a moving image displayed on the display panel 12 by the display unit 19 in this configuration will be described with reference to FIG. FIG. 7 is a diagram showing still another example of the moving image displayed on the display panel 12 by the display unit 19 according to this embodiment.

As shown in FIG. 7, when the accuracy indicated by the second accuracy information is lower than a predetermined value, the display unit 19 again displays on the display panel 12 an image including a message DP2 prompting to shoot a moving image. With this configuration, the terminal device 1 can notify the user that it is necessary to capture moving images again in order to derive parameters with high accuracy.

Further, as shown in FIG. 7, the display unit 19 displays the message DP2 prompting to shoot a moving image, the parameter PM3 derived by the parameter derivation unit 17, the parameter PM4 for each evaluation item, and the accuracy derivation unit 18. The accuracy indicated by the derived second accuracy information may be displayed.

In addition to the moving image shown in FIG. 7, the display unit 19 may change the display mode as follows depending on whether the accuracy indicated by the second accuracy information is higher or lower than a predetermined value. good.
・Change the color of the text of the accuracy indicated by the second accuracy information ・Change the size of the text of the accuracy indicated by the second accuracy information ・When the accuracy indicated by the second accuracy information is higher than a predetermined value, Inform the user that they do not need to take video again (e.g. display the text "Measurement successfully completed")
However, these do not limit this embodiment.

In this way, the terminal device 1 superimposes and displays the target person's skeleton extracted using the first algorithm from the moving image including the target person as the subject on the target person in the moving image, and displays one or more A second algorithm is used to refer to the subject's skeleton extracted from the moving image to derive parameters relating to the subject's bodily motion. Therefore, the terminal device 1 superimposes the target person's skeleton extracted using the first algorithm on the target person, while referring to the target person's skeleton extracted using the second algorithm. Since the parameters related to the motion of the person's body are derived, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been shot.

In addition, the terminal device 1 executes a skeleton extraction process for extracting the skeleton of the target person from a moving image containing the target person as a subject, and refers to the result of the skeleton extraction process to derive parameters related to the movement of the body of the target person. Then, accuracy information indicating the accuracy of the parameter is derived according to a frame for which skeleton extraction by the skeleton extraction process has failed among the plurality of frames included in the moving image. Therefore, since the terminal device 1 derives the accuracy of the parameter according to the frame for which skeleton extraction by the skeleton extraction process has failed, the reliability of the parameter can be notified to the user.

[Embodiment 2]
Other embodiments of the invention are described below. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and description thereof will not be repeated.

(Information processing system 100)
An outline of an information processing system 100 according to the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram showing an overview of the information processing system 100 according to this embodiment. As shown in FIG. 8, the information processing system 100 includes a terminal device 1A and an information processing device 3. As shown in FIG.

As shown in FIG. 8, the terminal device 1A and the information processing device 3 are connected via a network N so as to be able to communicate with each other. Here, examples of the terminal device 1A include a smart phone, a PC (Personal Computer), and a tablet, and an example of the information processing device 3 includes a server, but these do not limit the present embodiment.

Further, the specific configuration of the network N does not limit the present embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, Or a combination of these networks can be used.

In the information processing system 100 according to the present embodiment, the terminal device 1A displays the target person's skeleton extracted from the moving image of the target person captured by the user of the terminal device 1A so as to be superimposed on the target person in the moving image. Then, the information processing device 3 acquires the moving image from the terminal device 1A, refers to the skeleton of the subject person extracted from the moving image, and derives parameters related to the movement of the body of the subject person. Further, the information processing device 3 derives accuracy information indicating the accuracy of the derived parameter according to the frame for which the skeleton extraction by the skeleton extraction process has failed.

In the present embodiment, as in the first embodiment, a cared person is an example of the target person, but this does not limit the present embodiment. Further, the user of the terminal device 1A may be a caregiver who takes care of the care recipient, but this does not limit the present embodiment.

(Terminal device 1A)
A configuration example of the terminal device 1A according to the present embodiment will be described with reference to FIG. FIG. 9 is a block diagram showing a configuration example of the terminal device 1A and the information processing device 3 included in the information processing system 100 according to this embodiment. As shown in FIG. 9, the terminal device 1A includes a control unit 10A instead of the control unit 10 in the terminal device 1 of the first embodiment, and further includes a communication unit 11. FIG.

The communication unit 11 is a communication module for performing communication via the network N. FIG. The communication unit 11 outputs data obtained from the control unit 10A to the information processing device 3 via the network N, and supplies data obtained from the information processing device 3 via the network N to the control unit 10A. do.

(Control section 10A)
The control unit 10A controls each component of the terminal device 1A. The control unit 10A also functions as an acquisition unit 15, a skeleton extraction unit 16, a display unit 19, and an output unit 20. Since the function of the display unit 19 is the same as that of the display unit 19 in the first embodiment, description thereof will be omitted.

The acquisition unit 15 acquires data from the information processing device 3 via the communication unit 11 in addition to the functions of the acquisition unit 15 in the first embodiment. Examples of data acquired by the acquisition unit 15 include parameters and second accuracy information, but are not limited to these.

The skeleton extraction unit 16 uses the first algorithm to execute skeleton extraction processing for extracting the skeleton of the target person from the moving image containing the target person as the subject. The skeleton of the subject extracted by the first algorithm is referred to in real time by the display unit 19, which will be described later, as first skeleton information indicating the skeleton. Also, the first algorithm generates first accuracy information indicating the accuracy of the extracted skeleton. The first accuracy information is referred to by the display section 19, which will be described later. As in the first embodiment, the first algorithm is preferably an algorithm with faster skeleton extraction processing than the second algorithm.

Further, the skeleton extracting unit 16 may generate error information indicating a frame for which skeleton extraction has failed among the frames included in the moving image when extracting the skeleton using the first algorithm. The error information is as described in the first embodiment.

The output unit 20 outputs data to the information processing device 3 via the communication unit 11 . Examples of data output by the output unit 20 include moving images and error information, but are not limited to these.

(Information processing device 3)
As shown in FIG. 9, the information processing device 3 includes a control section 30, a communication section 31, and a storage section .

The communication unit 31 is a communication module for performing communication via the network N. FIG. The communication unit 31 outputs data obtained from the control unit 30 to the terminal device 1 via the network N, and supplies data obtained from the terminal device 1A via the network N to the control unit 30 .

The storage unit 34 stores various data referred to by the control unit 30, which will be described later. As an example, the storage unit 34 stores moving images acquired from the terminal device 1A, derived parameters, and the like.

(control unit 30)
The control unit 30 controls each component of the information processing device 3 . As an example, the control unit 30 acquires a moving image output from the terminal device 1A via the communication unit 31, and outputs data generated via the communication unit 31.

The control unit 30 also functions as an acquisition unit 35, a skeleton extraction unit 36, a parameter derivation unit 37, a probability derivation unit 38, and an output unit 40, as shown in FIG.

The acquisition unit 35 acquires data from the terminal device 1A via the communication unit 31 . An example of the data acquired by the acquisition unit 35 is a moving image, but the data is not limited to this.

The skeleton extracting unit 36 executes a skeleton extracting process of extracting the skeleton of the subject using the second algorithm from the moving image acquired by the acquiring unit 35 . The result of the subject's skeleton extracted by the second algorithm is referred to by the parameter derivation unit 37, which will be described later, as second skeleton information indicating the skeleton.

Further, the skeleton extraction unit 36 may generate error information indicating a frame for which skeleton extraction has failed among the frames included in the moving image when extracting the skeleton using the second algorithm.

The parameter derivation unit 37 derives parameters related to the movement of the subject's body by referring to the results of the skeleton extraction processing executed by the skeleton extraction unit 36, similar to the parameter derivation unit 17 described above. The parameters related to the motion of the subject's body are as described in the first embodiment.

Like the accuracy derivation unit 18 described above, the accuracy derivation unit 38 extracts the second accuracy information indicating the accuracy of the parameter derived by the parameter derivation unit 37 from among a plurality of frames included in the moving image by skeleton extraction processing. is derived according to the failed frame.

The output unit 40 outputs data to the terminal device 1A via the communication unit 31. FIG. Examples of data output by the output unit 40 include, but are not limited to, parameters and second accuracy information.

(Processing in information processing system 100)
Processing in the information processing system 100 will be described with reference to FIG. 10 . FIG. 4 is a sequence diagram showing the flow of processing in the information processing system 100 according to this embodiment.

(Step S12 to Step S15)
The processing from step S12 to step S15 from when the terminal device 1A starts acquiring the moving image until the acquisition of the moving image ends is the same as the above-described processing, so the description thereof is omitted.

(Step S20)
In step S<b>20 , the output unit 20 of the terminal device 1</b>A outputs the moving image to the information processing device 3 via the communication unit 11 . Here, if the skeleton extraction unit 16 has generated at least one of the first accuracy information and error information, the output unit 20 outputs at least one of the generated first accuracy information and error information together with the moving image. can be output. The output unit 20 may also output the camera parameters of the imaging unit 13 and the image size of the moving image for parameter derivation processing by the parameter derivation unit 37, which will be described later.

(Step S21)
In step S<b>21 , the acquisition unit 35 of the information processing device 3 acquires the moving image from the terminal device 1</b>A via the communication unit 31 .

(Step S22)
In step S<b>22 , the skeleton extraction unit 36 executes skeleton extraction processing for extracting the skeleton of the subject from the moving image acquired by the acquisition unit 35 using the second algorithm. More specifically, the skeleton extraction unit 36 extracts the skeleton of the subject from the moving image using one or more second algorithms, and generates second skeleton information indicating the extracted skeleton. Further, when the first accuracy information is output from the terminal device 1A, the skeleton extraction unit 36 refers to the first accuracy information and selects one or a plurality of second algorithms from a plurality of algorithm candidates. , the skeleton of the subject may be extracted using one or more selected second algorithms. The details of the process of selecting one or more second algorithms from multiple algorithm candidates are as described above. Also, in step S22, the skeleton extraction unit 36 may generate error information.

(Step S23)
In step S23, the parameter derivation unit 37 refers to the result of the skeleton extraction process executed in step S22 to derive parameters related to the motion of the subject's body. More specifically, the parameter deriving unit 37 refers to the second skeleton information generated by the skeleton extracting unit 36 and derives parameters relating to the motion of the subject's body. The details of the process of deriving the parameters by the parameter derivation unit 37 are as described above.

(Step S24)
In step S24, the accuracy derivation unit 38 calculates the second accuracy information indicating the accuracy of the parameter derived in step S23 as a frame for which skeleton extraction by the skeleton extraction process has failed, among the plurality of frames included in the moving image. derived according to More specifically, the accuracy derivation unit 38 refers to at least one of the error information output from the terminal device 1A in step S20 and the error information generated in step S22 to derive the second accuracy information. The details of the process of deriving the second accuracy information by the accuracy derivation unit 38 are as described above.

(Step S25)
In step S25, the output unit 40 outputs the parameter derived in step S23 and the second accuracy information derived in step S24 to the terminal device 1A via the communication unit 31.

(Step S26)
In step S<b>26 , the acquisition unit 15 of the terminal device 1</b>A acquires the parameters and the second accuracy information output from the information processing device 3 via the communication unit 11 .

(Step S27)
In step S<b>27 , the display unit 19 displays on the display panel 12 the moving image including the parameters and the second accuracy information acquired by the acquisition unit 15 in step S<b>26 . Examples of images displayed by the display unit 19 in step S27 include the images shown in FIGS. 6 and 7 described above.

In this way, the terminal device 1A in the information processing system 100 displays the skeleton of the target person extracted from the moving image including the target person as the subject using the first algorithm, superimposed on the target person in the moving image. . Further, the information processing device 3 acquires a moving image including the subject as a subject, refers to the skeleton extracted from the acquired moving image using the second algorithm, and derives parameters related to the movement of the subject's body. . Therefore, in the information processing system 100, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

Further, the information processing device 3 in the information processing system 100 executes a skeleton extraction process for extracting the skeleton of the target person from a moving image including the target person as a subject, refers to the result of the skeleton extraction process, and extracts the target person's body. parameters for the operation of Further, the information processing device 3 derives accuracy information indicating the accuracy of the derived parameters according to a frame for which skeleton extraction by the skeleton extraction process has failed, among the plurality of frames included in the moving image. Therefore, the information processing device 3 derives the accuracy of the parameter according to the frame for which skeleton extraction by the skeleton extraction process fails, so that the reliability of the parameter can be notified to the user.

(Physical Configuration of Terminal Device 1A)
FIG. 11 is a block diagram illustrating the physical configuration of a computer used as the terminal device 1A(1) and the information processing device 3. As shown in FIG. The terminal device 1A can be configured by a computer having a bus 110, a processor 101, a main memory 102, an auxiliary memory 103, a communication interface 104, a camera 105, and a display panel 106, as shown in FIG. be. Processor 101 , main memory 102 , auxiliary memory 103 , communication interface 104 , camera 105 and display panel 106 are interconnected via bus 110 .

As the processor 101, for example, a CPU (Central Processing Unit), a microprocessor, a digital signal processor, a microcontroller, or a combination thereof is used.

As the main memory 102, for example, a semiconductor RAM (random access memory) or the like is used.

As the auxiliary memory 103, for example, a flash memory, a HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination thereof is used. The auxiliary memory 103 stores a program for causing the processor 101 to execute the operation of the terminal device 1A described above. The processor 101 expands the program stored in the auxiliary memory 103 onto the main memory 102 and executes each instruction included in the expanded program.

A communication interface 104 is an interface that connects to the network N. FIG.

The camera 105 is an example of a hardware element that implements the imaging unit 13 .

Display panel 106 is an example of a hardware element that implements display panel 12 .

In this example, processor 101 is an example of a hardware element that implements control unit 10A. Also, the main memory 102 and the auxiliary memory 103 are examples of hardware elements that implement the storage unit 14 . Also, the communication interface 104 is an example of a hardware element that implements the communication unit 11 .

(Physical Configuration of Information Processing Device 3)
The information processing device 3 can be configured by a computer having a bus 310, a processor 301, a main memory 302, an auxiliary memory 303, and a communication interface 304, as shown in FIG. Processor 301 , main memory 302 , auxiliary memory 303 and communication interface 104 are interconnected via bus 110 .

In this example, processor 301 is an example of a hardware element that implements control unit 30 . Also, the main memory 302 and the auxiliary memory 303 are examples of hardware elements that implement the storage unit 34 . Also, the communication interface 304 is an example of a hardware element that implements the communication unit 31 .

[Example of realization by software]
The functions of the terminal devices 1 and 1A and the information processing device 3 (hereinafter collectively referred to as "apparatus") are programs for causing a computer to function as the device, and each control block (especially the control block) of the device. Each unit included in units 10, 10A, and 30) can be realized by a program for causing a computer to function.

In this case, the apparatus comprises a computer having at least one control device (eg processor) and at least one storage device (eg memory) as hardware for executing the program. Each function described in each of the above embodiments is realized by executing the above program using the control device and the storage device.

The program may be recorded on one or more computer-readable recording media, not temporary. The recording medium may or may not be included in the device. In the latter case, the program may be supplied to the device via any transmission medium, wired or wireless.

Also, part or all of the functions of the above control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits functioning as the control blocks described above are formed are also included in the scope of the present invention. In addition, it is also possible to implement the functions of the control blocks described above by, for example, a quantum computer.

Further, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may operate on the control device, or may operate on another device (for example, an edge computer or a cloud server).

An embodiment of the invention is described below.

A graph showing the relationship between the measured walking speed of the subject measured using a three-dimensional motion analysis device and the estimated walking speed estimated using gait analysis AI from the moving image of the subject walking. , as shown in FIG. FIG. 12 is a graph showing the relationship between estimated values and measured values in Example 1;

◯, Δ, and × in FIG. 12 respectively indicate the following moving images.
△: A video image with frames for which skeleton extraction failed during the period from the start of the video image to the start of the motion, or from the end of the motion to the end of the video image ×: Frame extraction failed in the video image during motion Video with failed frames ○: Video with no frames for which skeleton extraction failed It was found that the degree is low (the difference between the estimated value and the measured value is large). On the other hand, in the period from the start of the video image to the start of the motion, or from the end of the motion to the end of the video image, if there are frames for which skeleton extraction has failed, or if there are no frames for which skeleton extraction has failed. , the estimated value of walking speed was found to be highly reliable (the difference between the estimated value and the measured value was small).

Other embodiments of the invention are described below.

Graph showing the relationship between the measured value of head sway measured using a 3D motion analysis device and the estimated value of head sway estimated using gait analysis AI from moving images of the subject walking is shown in FIG. FIG. 13 is a graph showing the relationship between estimated values and measured values in Example 2;

◯, Δ, and × in FIG. 13 also indicate the following moving images.
△: A video image with frames for which skeleton extraction failed during the period from the start of the video image to the start of the motion, or from the end of the motion to the end of the video image ×: Frame extraction failed in the video image during motion Video image with failed frames ○: Video image with no frames for which skeleton extraction has failed was found to have low reliability (large discrepancy between estimated values and measured values). On the other hand, in the period from the start of the video image to the start of the motion, or from the end of the motion to the end of the video image, if there are frames for which skeleton extraction has failed, or if there are no frames for which skeleton extraction has failed. , the estimated value of head sway was found to be highly reliable (the difference between the estimated value and the measured value was small).

[summary]
A terminal device according to one aspect of each embodiment displays a skeleton of a target person extracted from a moving image including the target person as a subject using a first algorithm, superimposed on the target person in the moving image. and deriving parameters related to the movement of the subject's body by referring to the skeleton of the subject extracted from the moving image using one or more second algorithms different from the first algorithm. and

According to the above configuration, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

In the terminal device according to one aspect of each embodiment, it is preferable that the skeleton extraction processing by the one or more second algorithms has higher accuracy than the skeleton extraction processing by the first algorithm.

According to the above configuration, it is possible to evaluate the motion of the subject's body with higher accuracy.

In the terminal device according to one aspect of each embodiment, the display unit acquires accuracy information indicating the likelihood of the skeleton extracted using the first algorithm, and according to the acquired accuracy information, the subject person It is preferable to change the display mode of the skeleton superimposed on the .

According to the above configuration, it is possible to notify the user which skeleton has been extracted with high accuracy from among the skeletons of the target person in the captured moving image.

In the terminal device according to one aspect of each embodiment, it is preferable that, when the accuracy indicated by the accuracy information is lower than a predetermined value, the display unit displays a message prompting the subject to be photographed so as to increase the accuracy. .

According to the above configuration, it is possible to notify the user of how to take an image to increase the accuracy of extracting the skeleton of the target person.

The terminal device according to one aspect of each embodiment preferably further includes a skeleton extraction unit that executes skeleton extraction processing using the first algorithm and the one or more second algorithms.

According to the above configuration, the skeleton extraction process can be executed.

In the terminal device according to one aspect of each embodiment, the skeleton extracting unit selects the one or more second algorithms from a plurality of second algorithm candidates, and selects the selected one or more second algorithms is preferably used to extract the skeleton of the subject.

According to the above configuration, it is possible to extract the skeleton of the subject with higher accuracy.

In the terminal device according to one aspect of each embodiment, the skeleton extraction unit refers to the skeleton of the subject extracted using the first algorithm, and selects the one or more second algorithms. is preferred.

According to the above configuration, it is possible to extract the skeleton of the subject with higher accuracy.

In the terminal device according to one aspect of each embodiment, the derivation unit refers to the skeleton of the subject during walking, which is extracted using the one or more second algorithms, and It is preferable to derive the parameters.

According to the above configuration, it is possible to derive the parameters related to the walking of the subject who is walking.

An information processing method according to an aspect of each embodiment displays a skeleton of a target person extracted from a moving image including a target person as a subject using a first algorithm, superimposed on the target person in the moving image. and referring to the skeleton of the subject extracted from the moving image using one or more second algorithms different from the first algorithm, and deriving parameters related to the movement of the body of the subject. and a step.

According to the above configuration, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

An information processing system according to one aspect of each embodiment is an information processing system that includes a terminal device and an information processing device, wherein the terminal device uses a first algorithm from a moving image including a target person as a subject. a display unit for superimposing and displaying the extracted skeleton of the target person on the target person in the moving image; and an output unit for outputting the moving image to the information processing device, wherein the information processing device comprises: with reference to the skeleton of the subject extracted from the moving image using an acquisition unit that acquires the moving image from the terminal device and one or more second algorithms different from the first algorithm; and a deriving unit for deriving parameters related to the motion of the person's body.

According to the above configuration, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

A program according to one aspect of each embodiment is a program for causing a computer to function as the above-described terminal device, and causes the computer to function as the display unit and the derivation unit.

According to the above configuration, it is possible to accurately evaluate the motion of the subject's body while notifying the user who is shooting the subject of whether or not an appropriate moving image has been captured.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

Reference Signs List 1, 1A terminal device 3 information processing device 10, 10A, 30 control unit 11, 31 communication unit 12 display panel 13 imaging unit 14, 34 storage unit 15, 35 acquisition unit 16, 36 skeleton extraction unit 17, 37 parameter derivation unit 18 , 38 accuracy derivation unit 19 display unit 20, 40 output unit 100 information processing system

Claims (11)

a display unit for superimposing and displaying the skeleton of the target person extracted using a first algorithm from a moving image including the target person as a subject on the target person in the moving image;
a deriving unit that refers to the skeleton of the subject extracted from the moving image using one or more second algorithms different from the first algorithm, and derives parameters related to the movement of the body of the subject;
terminal device. 2. The terminal device according to claim 1, wherein skeleton extraction processing by said one or more second algorithms has higher accuracy than skeleton extraction processing by said first algorithm. The display unit
Acquiring accuracy information indicating the accuracy of the skeleton extracted using the first algorithm,
3. The terminal device according to claim 1, wherein the display mode of the skeleton superimposed on the subject is changed according to the acquired accuracy information. 4. The terminal device according to claim 3, wherein, when the accuracy indicated by the accuracy information is lower than a predetermined value, the display section displays a display prompting the subject to be photographed so as to increase the accuracy. The terminal device according to any one of claims 1 to 4, further comprising a skeleton extracting unit that executes skeleton extraction processing using the first algorithm and the one or more second algorithms. wherein said skeleton extracting unit selects said one or more second algorithms from a plurality of second algorithm candidates, and extracts said subject's skeleton using said one or more selected second algorithms. Item 6. The terminal device according to item 5. 6. The terminal device according to claim 5, wherein the skeleton extraction unit selects the one or more second algorithms by referring to the skeleton of the subject extracted using the first algorithm. 8. The derivation unit refers to the skeleton of the subject during walking, which is extracted using the one or more second algorithms, and derives parameters related to walking of the subject. The terminal device according to item 1. a step of superimposing and displaying the skeleton of the target person extracted using a first algorithm from a moving image including the target person as a subject on the target person in the moving image;
referring to the skeleton of the subject extracted from the moving image using one or more second algorithms different from the first algorithm, and deriving parameters relating to the movement of the subject's body;
Information processing method including. An information processing system including a terminal device and an information processing device,
The terminal device
a display unit for superimposing and displaying the skeleton of the target person extracted using a first algorithm from a moving image including the target person as a subject on the target person in the moving image;
an output unit that outputs the moving image to the information processing device;
with
The information processing device is
an acquisition unit that acquires the moving image from the terminal device;
a deriving unit that refers to the skeleton of the subject extracted from the moving image using one or more second algorithms different from the first algorithm, and derives parameters related to the movement of the body of the subject;
An information processing system comprising A program for causing a computer to function as the terminal device according to claim 1, the program for causing the computer to function as the display section and the derivation section.

Images