JP7042956B1 - Information processing methods, information processing devices, and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To notify a user of the reliability of evaluation of a subject's physical movement. SOLUTION: The information processing method executes a skeleton extraction process for extracting a skeleton of a target person from a moving image including the target person, and derives a parameter related to the movement of the body of the target person by referring to the result of the skeleton extraction process. It includes a step of deriving the accuracy information indicating the accuracy of the derived parameter according to the frame in which the skeleton extraction by the skeleton extraction process fails. [Selection diagram] Fig. 1

Description

The present invention relates to an information processing method, an information processing apparatus, and a program.

In the technique of analyzing the gait of a subject from a moving image including a walking subject, a technique aimed at improving the accuracy of the analysis is known. Patent Document 1 discloses a terminal that acquires situation information including information indicating a shooting situation which is an internal situation of a shooting device at the time of shooting, and outputs a composite image including a situation image showing the situation information in the shot image. ing.

Japanese Patent No. 6895669

By the way, a technique for evaluating the walking of a subject by performing gait analysis is also known. In such a technique, there is no known technique for notifying the user of the reliability of the evaluation.

One aspect of the present invention is to notify the user of the reliability of the evaluation of the body movement of the subject.

In order to solve the above problems, the information processing method according to one aspect of the present invention executes a skeleton extraction process for extracting the skeleton of the subject from a moving image including the subject as a subject, and results in the skeleton extraction process. Of the plurality of frames included in the moving image, the parameter derivation step for deriving the parameters related to the movement of the body of the subject and the accuracy information indicating the accuracy of the parameters derived in the parameter derivation step are obtained with reference to. An information processing method including a probability derivation step of deriving according to a frame in which skeleton extraction by the skeleton extraction process fails.

According to one aspect of the present invention, it is possible to notify the user of the reliability of the evaluation of the body movement of the subject.

It is a block diagram which shows the structural example of the terminal apparatus which concerns on Embodiment 1 of this invention. It is a sequence diagram which shows the flow of the process by the terminal apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows an example of the moving image which the display part which concerns on Embodiment 1 of this invention displays on a display panel. It is a figure which shows the other example of the moving image which the display part which concerns on Embodiment 1 of this invention displays on a display panel. It is a figure which shows the further example of the moving image which the display part which concerns on Embodiment 1 of this invention displays on a display panel. It is a figure which shows the further example of the moving image which the display part which concerns on Embodiment 1 of this invention displays on a display panel. It is a figure which shows the further example of the moving image which the display part which concerns on Embodiment 1 of this invention displays on a display panel. It is a schematic diagram which shows the outline of the information processing system which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structural example of the terminal apparatus and information processing apparatus included in the information processing system which concerns on Embodiment 2 of this invention. It is a sequence diagram which shows the flow of processing in the information processing system which concerns on Embodiment 2 of this invention. It is a block diagram which exemplifies the physical composition of the computer used as a terminal device and an information processing device. It is a graph which shows the relationship between the estimated value and the measured value in Example 1. FIG. It is a graph which shows the relationship between the estimated value and the measured value in Example 2. FIG.

[Embodiment 1]
Hereinafter, an embodiment of the present invention will be described.

(Overview of terminal device 1)
A configuration example of the terminal device 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration example of the terminal device 1 according to the present embodiment.

The terminal device 1 according to the present embodiment captures a moving image including a target person as a subject, refers to the moving image, and performs the following processing.
・ Extract the skeleton of the target person and display the extracted skeleton of the target person superimposed on the target person in the moving image. Here, a care recipient can be mentioned as an example of the above-mentioned subject, but this does not limit the present embodiment. Further, as a user of the terminal device 1, a care worker who cares for the person to be cared for can be mentioned, but this does not limit the present embodiment.

(Configuration of terminal device 1)
As shown in FIG. 1, the terminal device 1 includes a control unit 10, a display panel 12, an image pickup unit 13, and a storage unit 14.

The display panel 12 is a display that displays data supplied from the control unit 10 described later. As an example, the display panel 12 displays a moving image in which the skeleton of the subject is superimposed on the subject.

The image pickup unit 13 is a camera capable of capturing a moving image. As an example, the image pickup unit 13 is directed by the user so that the subject is included in the angle of view of the image pickup unit 13. Then, the imaging unit 13 captures a state in which the subject is performing a predetermined motion (walking, arm raising, etc.), and generates a moving image including the subject as a subject. The image pickup unit 13 supplies the moving image obtained by the image pickup to the control unit 10 described later.

The storage unit 14 stores various data referred to by the control unit 10 described later. As an example, the storage unit 14 stores a moving image captured by the imaging unit 13, skeleton information indicating the skeleton of the subject described later, and the like.

(Control unit 10)
The control unit 10 controls each component of the terminal device 1. As an example, the control unit 10 acquires a moving image captured by the imaging unit 13 and supplies moving image data to be displayed on the display panel 12.

Further, as shown in FIG. 1, the control unit 10 also functions as an acquisition unit 15, a skeleton extraction unit 16, a parameter derivation unit 17 (derivation unit), an accuracy derivation unit 18, and a display unit 19.

The acquisition unit 15 acquires a moving image captured by the image pickup unit 13. The acquisition unit 15 stores the acquired moving image in the storage unit 14.

The skeleton extraction unit 16 executes a skeleton extraction process for extracting the skeleton of the target person from a moving image including the target person as a subject by using the first algorithm and one or a plurality of second algorithms.

Here, the first algorithm and the second algorithm are algorithms for extracting the skeleton of the subject. The skeleton of the subject extracted by the first algorithm is referred to in real time by the display unit 19 described later as the first skeleton information indicating the skeleton. Therefore, it is preferable that the first algorithm is an algorithm having a faster skeleton extraction process than the second algorithm described later. More specifically, the first algorithm is preferably faster than the video refresh rate (30 fps or 60 fps). In addition, the first algorithm generates first accuracy information indicating the accuracy of the extracted skeleton. The first accuracy information is referred to by the display unit 19 described later.

The result of the skeleton extraction process of the subject extracted by the second algorithm is referred to by the parameter derivation unit 17 described later as the second skeleton information indicating the skeleton. Although the details will be described later, it is preferable that the skeleton extraction process by one or a plurality of second algorithms has higher accuracy than the skeleton extraction process by the first algorithm.

Further, the skeleton extraction unit 16 is configured to select one or a plurality of second algorithms from a plurality of algorithm candidates and extract the skeleton of the subject by using the selected one or a plurality of second algorithms. May be good. For example, the skeleton extraction unit 16 may select one or a plurality of second algorithms with reference to the skeleton of the subject extracted using the first algorithm. A specific example of the process in which the skeleton extraction unit 16 selects one or a plurality of second algorithms will be described later.

Further, the skeleton extraction unit 16 outputs error information indicating a frame in which the skeleton extraction has failed among the frames included in the moving image when the skeleton is extracted using at least one of the first algorithm and the second algorithm. Generate.

The frame in which the skeleton extraction fails is a frame in which the skeleton is not extracted or a frame in which the skeleton is not properly extracted in the skeleton extraction using the first algorithm or the second algorithm. .. As a concrete example
-The return value of the skeleton extraction process could not be obtained because the skeleton extraction process was not completed.-The return value of the skeleton extraction process was completed, but the return value of the skeleton extraction process was out of the specified range (outlier value). The frames that have been used are, but are not limited to these.

As an example of error information
-Information indicating the number of frames for which skeleton extraction failed-Indicates the temporal position of the frame for which skeleton extraction failed (such as the date and time when the frame for which skeleton extraction failed was taken, or the elapsed time from the start of moving image shooting). Information ・ In the frame where the skeleton extraction failed, information indicating the part where the skeleton could not be extracted can be mentioned, but the present invention is not limited to these.

The parameter derivation unit 17 derives parameters related to the movement of the body of the subject with reference to the result of the skeleton extraction process executed by the skeleton extraction unit 16. More specifically, the parameter derivation unit 17 refers to the second skeleton information indicating the skeleton of the subject extracted from the moving image by the skeleton extraction unit 16 using the second algorithm, and the movement of the body of the subject. Derives the parameters for. The parameter relating to the movement of the body of the subject is a parameter indicating how much the movement can be performed when the subject performs the movement. Examples of parameters related to the movement of the subject's body are when the subject performs movements such as walking, raising arms, and standing up, the speed at which the movement is performed, the rhythm of the movement, and the wobbling of the body when performing the movement. The difference between the left and right of the operation can be mentioned, but the present invention is not limited to these. A specific example of the process of deriving the parameter by the parameter deriving unit 17 will be described later.

The accuracy derivation unit 18 derives the second accuracy information indicating the accuracy of the parameter derived by the parameter derivation unit 17 according to the frame in which the skeleton extraction by the skeleton extraction process fails among the plurality of frames included in the moving image. .. More specifically, the accuracy derivation unit 18 derives the second accuracy information indicating the accuracy of the parameter derived by the parameter derivation unit 17 with reference to the error information generated by the skeleton extraction unit 16. The accuracy of the parameter is the degree of certainty of the parameter, and the higher the accuracy of the parameter, the higher the reliability of the parameter. A specific example of the process in which the accuracy derivation unit 18 derives the second accuracy information will be described later.

The display unit 19 generates moving image data to be displayed on the display panel 12. Specifically, the display unit 19 refers to the first skeleton information generated by the skeleton extraction unit 16, and superimposes the skeleton indicated by the first skeleton information on the target person in the moving image acquired by the acquisition unit 15. And display it on the display panel 12. Further, the display unit 19 refers to the second accuracy information derived by the accuracy derivation unit 18, and displays the accuracy indicated by the second accuracy information on the display panel 12.

(Processing by terminal device 1)
The processing by the terminal device 1 will be described with reference to FIG. FIG. 2 is a sequence diagram showing a flow of processing by the terminal device 1 according to the present embodiment.

(Step S12)
In step S12, the acquisition unit 15 starts acquiring a moving image including the subject as a subject via the image pickup unit 13. For example, when the acquisition unit 15 acquires information indicating that shooting is to be started from an operation reception unit (not shown) that accepts an operation from the user of the terminal device 1, the acquisition unit 15 starts acquisition of a moving image via the image pickup unit 13. do.

(Step S13)
In step S13, the skeleton extraction unit 16 extracts the skeleton of the subject using the first algorithm from the moving image acquired by the acquisition unit 15 in step S12. The skeleton extraction unit 16 supplies the display unit 19 with the first skeleton information indicating the extracted skeleton. Further, in step S13, the skeleton extraction unit 16 may generate the first accuracy information indicating the certainty of the extracted skeleton and the error information indicating the frame in which the skeleton extraction has failed. In this case, the skeleton extraction unit 16 supplies the first accuracy information to the display unit 19, and supplies the error information to the accuracy derivation unit 18.

(Step S14)
In step S14, the display unit 19 supplies the display panel 12 with moving image data indicating a moving image in which the skeleton indicated by the first skeleton information supplied by the skeleton extracting unit 16 in step S13 is superimposed on the target person in the moving image. .. The display panel 12 displays the moving image indicated by the acquired moving image data.

(Example 1 of a moving image displayed by the display unit 19)
An example of a moving image displayed on the display panel 12 by the display unit 19 will be described with reference to FIG. FIG. 3 is a diagram showing an example of a moving image displayed on the display panel 12 by the display unit 19 according to the present embodiment.

The display unit 19 superimposes the joints on the subject among the skeletons indicated by the first skeleton information and displays them on the display panel 12. For example, as shown in FIG. 3, the display unit 19 superimposes the points JT1, the points JT2, the points JT3, and the like indicating the joints on the subject TP and displays them on the display panel 12. Further, the display unit 19 connects adjacent joints (for example, points JT1 and JT2 in FIG. 3) with a line, superimposes them on the subject TP, and displays them on the display panel 12. In other words, the display unit 19 does not display the unextracted skeleton on the display panel 12 in the skeleton extraction using the first algorithm. With this configuration, the terminal device 1 can notify the user who is photographing the target person whether or not the moving image being photographed is a moving image suitable for extracting the skeleton of the target person.

Here, in order to notify the user whether or not the skeleton of the subject TP included in the moving image to be captured has been extracted, it is preferable that the skeleton extraction process is fast. Therefore, as described above, the first algorithm for extracting the skeleton indicated by the first skeleton information is preferably an algorithm having a faster extraction process than the second algorithm. With this configuration, the terminal device 1 can notify the user during shooting whether or not the moving image being shot is a moving image suitable for extracting the skeleton of the subject.

(Example 2 of a moving image displayed by the display unit 19)
When the skeleton extraction unit 16 supplies the first accuracy information to the display unit 19, the display unit 19 acquires the first accuracy information, and the skeleton superimposed on the target person according to the acquired first accuracy information. It may be configured to change the display mode of. An example of a moving image displayed on the display panel 12 by the display unit 19 in this configuration will be described with reference to FIG. FIG. 4 is a diagram showing another example of a moving image displayed on the display panel 12 by the display unit 19 according to the present embodiment.

As shown in FIG. 4, the display unit 19 superimposes the point JT2, the point JT3, the point JT4, etc. indicating the joints on the subject TP and displays them on the display panel 12, as in FIG. 3 described above. Here, when the accuracy indicated by the first accuracy information in the left knee of the subject TP is lower than a predetermined value, the display unit 19 displays another aspect of the point JT4 indicating the left knee, as shown in FIG. The mode (shape is triangle) different from the display mode (shape is circle) indicating the joint is changed and displayed. With this configuration, the terminal device 1 can notify the user during shooting which of the skeletons of the target person in the moving image being shot can be extracted with high accuracy.

The configuration in which the display unit 19 has a different display mode is, for example, other than the configuration in which the shape is different.
-The color of the display indicating the skeleton is different according to the first accuracy information.-The shade of the display indicating the skeleton is different according to the first accuracy information.-The skeleton is indicated according to the first accuracy information. The size of the display may be different, but the display size is not limited to these.

(Example 3 of a moving image displayed by the display unit 19)
When the skeleton extraction unit 16 supplies the first accuracy information to the display unit 19, the display unit 19 photographs the subject so that the accuracy is high when the accuracy indicated by the first accuracy information is lower than a predetermined value. It may be configured to display a message prompting the user to do so. An example of a moving image displayed on the display panel 12 by the display unit 19 in this configuration will be described with reference to FIG. FIG. 5 is a diagram showing still another example of the moving image displayed on the display panel 12 by the display unit 19 according to the present embodiment.

For example, when the subject included in the moving image is small and the accuracy indicated by the first accuracy information is low in all or a part of the skeleton, the display unit 19 has a high accuracy as shown in FIG. Display DP1 that prompts the subject to be photographed is displayed. The display DP1 contains the text "Please shoot so that the subject looks a little bigger". With this configuration, the terminal device 1 can notify the user during shooting how to shoot to increase the accuracy of skeleton extraction of the subject.

As an example of a display that encourages you to shoot a subject with high accuracy,
・ Since the target person included in the moving image is small, if the accuracy indicated by the first accuracy information is low in all skeletons, "Please narrow the angle of view" is displayed. ・ The target person included in the moving image is on the left side. If the accuracy indicated by the first accuracy information in the skeleton on the left side of the subject is low, the message "Please take a picture of the left side a little more" may be displayed, but the present invention is not limited to these.

Further, even if the display unit 19 is configured to display a display prompting the subject to be photographed with high accuracy when the number of parts extracted by the skeleton extraction unit 16 is smaller than a predetermined number. good. More specifically, the display unit 19 refers to a portion indicated by the first skeleton information supplied by the skeleton extraction unit 16, and if the number of the portions is smaller than a predetermined number, the accuracy is high. A display prompting the subject to be photographed may be displayed.

With reference to FIG. 2 again, steps S15 and subsequent steps of the process by the terminal device 1 will be described.

(Step S15)
In step S15, the acquisition unit 15 ends the acquisition of the moving image via the image pickup unit 13. For example, when the acquisition unit 15 acquires information indicating the end of shooting from the operation reception unit (not shown) that accepts an operation from the user of the terminal device 1, the acquisition unit 15 acquires a moving image via the image pickup unit 13. finish.

(Step S16)
In step S16, the skeleton extraction unit 16 extracts the skeleton of the subject from the moving images acquired by the acquisition unit 15 in steps S12 to S15 using one or a plurality of second algorithms different from the first algorithm. do. In step S16, the skeleton extraction unit 16 may select one or a plurality of second algorithms from a plurality of algorithm candidates and extract the skeleton of the subject using the selected one or a plurality of second algorithms. .. If the skeleton extraction unit 16 does not generate error information in step S13, the skeleton extraction unit 16 may generate error information in step S16.

(Example of second algorithm selection process by the skeleton extraction unit 16)
An example of the second algorithm selection process by the skeleton extraction unit 16 will be described below.

In the configuration in which the skeleton extraction unit 16 selects one or a plurality of second algorithms from a plurality of algorithm candidates, the plurality of algorithm candidates have different characteristics such as extraction site, extraction accuracy, and extraction speed. May be good. In this case, the skeleton extraction unit 16 may refer to the skeleton of the subject extracted using the first algorithm and select an algorithm according to the characteristics.

As an example, when a plurality of algorithm candidates have different parts of the skeleton to be extracted, the skeleton extraction unit 16 selects one or a plurality of second algorithms from the plurality of algorithm candidates according to the parts of the skeleton to be extracted. In this case, the skeleton extraction unit 16 refers to the skeleton of the subject extracted by the first algorithm, and selects an algorithm from a plurality of algorithm candidates capable of extracting the skeleton portion extracted by the first algorithm. select. For example, when the leg of the subject is extracted by the first algorithm, the skeleton extraction unit 16 selects an algorithm capable of extracting the leg from a plurality of algorithm candidates.

Further, the skeleton extraction unit 16 may select one or a plurality of second algorithms from a plurality of algorithm candidates depending on the period. For example, if the leg of the subject is extracted by the first algorithm in a certain period and the arm of the subject is extracted by the first algorithm in a period different from the certain period, the skeleton extraction unit 16 is present. An algorithm capable of extracting the legs in a period is selected from a plurality of algorithm candidates, and an algorithm capable of extracting an arm in a different period is selected from a plurality of algorithm candidates.

Further, the skeleton extraction unit 16 may select one or a plurality of second algorithms from a plurality of algorithm candidates with reference to instructions from the user. For example, if the moving image includes a period in which the walking speed accelerates and a period in which the walking speed decreases, the user can use the acceleration period for each of the period in which the walking speed accelerates and the period in which the walking speed decreases. An annotation indicating that there is and an annotation indicating that the deceleration period is in effect are added to the moving image. The skeleton extraction unit 16 refers to the annotation and selects one or a plurality of second algorithms suitable for the movement in which the walking speed is accelerated during the period in which the walking speed is accelerated, and in the period in which the walking speed is decelerated. It may be configured to select one or more second algorithms suitable for the movement in which the walking speed is decelerated. Alternatively, the skeleton extraction unit 16 may be configured to directly instruct the user to select any one or a plurality of second algorithms for the period and the site.

In this way, the skeleton extraction unit 16 can extract the skeleton of the subject with high accuracy by selecting the second algorithm.

(Step S17)
In step S17, the parameter derivation unit 17 refers to the skeleton of the subject extracted in step S16, and derives parameters related to the movement of the body of the subject.

(Example of parameter derivation processing by parameter derivation unit 17)
An example of parameter derivation processing by the parameter derivation unit 17 will be described below.

For example, the parameter derivation unit 17 refers to the skeleton of the walking subject extracted by using the second algorithm, and derives the parameters related to the walking of the subject. More specifically, the second skeleton information acquired from the skeleton extraction unit 16 indicates that the right leg and the left leg of the subject have been extracted, and the right leg and the left leg are the subject walking. In the case of, the parameter derivation unit 17 calculates the walking speed of the target person. The parameter derivation unit 17 derives parameters according to the calculated speed. For example, when the walking speed of the subject is greater than a predetermined value, the parameter derivation unit 17 sets the parameter of the walking speed to a parameter indicating that the speed is high. On the other hand, when the walking speed of the subject is smaller than a predetermined value, the parameter derivation unit 17 sets the parameter of the walking speed to a parameter indicating that the speed is slow.

As another example, when the second skeletal information indicates that the right leg and the left leg of the subject have been extracted, and the right leg and the left leg indicate that the subject is walking, the parameter derivation unit 17 Calculates the speed at which the subject's right and left legs move, respectively. The parameter derivation unit 17 derives parameters according to the calculated speed. For example, when the difference between the speed at which the right leg of the subject moves and the speed at which the left leg of the subject moves is smaller than a predetermined value, the parameter derivation unit 17 indicates that there is no difference between the left and right parameters. Set in the parameter. On the other hand, when the difference between the speed at which the right leg of the subject moves and the speed at which the left leg of the subject moves is larger than a predetermined value, the parameter derivation unit 17 indicates that there is a difference between the left and right parameters. Set in the parameter.

In this way, the terminal device 1 derives parameters related to the movement of the body of the subject with reference to the second skeleton information indicating the skeleton extracted by the second algorithm. That is, since the skeleton extracted by the second algorithm is used for deriving parameters related to the movement of the body of the subject, as described above, the skeleton extraction process of the second algorithm is the skeleton extraction process by the first algorithm. It is preferable that the accuracy is higher than that. With this configuration, the terminal device 1 refers to the second skeleton information indicating the skeleton extracted by the second algorithm instead of the first skeleton information indicating the skeleton extracted by the first algorithm. The accuracy of the parameters related to the movement of the body can be improved.

(Step S18)
In step S18, the accuracy derivation unit 18 refers to the error information generated by the skeleton extraction unit 16, and obtains the second accuracy information indicating the accuracy of the parameter derived in step S17 among the plurality of frames included in the moving image. , Derived according to the frame in which the skeleton extraction by the skeleton extraction process failed.

(Example of accuracy derivation process by accuracy derivation unit 18)
An example of the accuracy derivation process by the accuracy derivation unit 18 will be described below.

The accuracy derivation unit 18 identifies the frame in which the skeleton extraction by the skeleton extraction process has failed by referring to the error information generated by the skeleton extraction unit 16, and derives the second accuracy information according to the specified frame.

As an example, when the error information indicates the number of frames in which the skeleton extraction by the skeleton extraction process has failed, the accuracy derivation unit 18 derives the second accuracy information according to the number of frames in which the skeleton extraction by the skeleton extraction process has failed. do. For example, when the number of frames for which skeleton extraction by the skeleton extraction process has failed is larger than a predetermined value, the accuracy derivation unit 18 derives the second accuracy information indicating that the accuracy is low as the second accuracy information. .. On the other hand, when the number of frames in which the skeleton extraction by the skeleton extraction process has failed is less than a predetermined value, the accuracy derivation unit 18 derives the second accuracy information indicating that the accuracy is high as the second accuracy information. .. With this configuration, the terminal device 1 can notify the user that the accuracy of the parameter is low when the number of frames in which the skeleton extraction by the skeleton extraction process fails is large.

As another example, when the error information indicates the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails, the accuracy derivation unit 18 responds to the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails. The second accuracy information is derived. For example, when the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails is included in the period during which the subject operates (for example, in the case of the walking video of the subject, the subject is walking). (When included in the period), the accuracy derivation unit 18 derives the second accuracy information indicating that the accuracy is low as the second accuracy information. On the other hand, when the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails is not included in the period during which the subject operates (for example, in the case of a walking video of the subject, before the subject walks (for example). Or after walking)), the accuracy derivation unit 18 derives the second accuracy information indicating that the accuracy is high as the second accuracy information. With this configuration, the terminal device 1 can notify the user that the accuracy of the parameter is low when the skeleton extraction fails during the period when the target person is operating.

As yet another example, when the error information indicates a site where the skeleton was not extracted in the frame in which the skeleton extraction by the skeleton extraction process failed, the accuracy derivation unit 18 is in the frame in which the skeleton extraction by the skeleton extraction process failed. The second accuracy information is derived according to the site where the skeleton was not extracted. For example, in a frame in which skeleton extraction by skeleton extraction processing fails, the part where the skeleton was desired to be extracted is a part indispensable for the subject's movement (for example, in the case of a walking video of the subject, the subject's leg. In the case of the unit), the accuracy derivation unit 18 derives the second accuracy information indicating that the accuracy is low as the second accuracy information. On the other hand, in the frame in which the skeleton extraction by the skeleton extraction process fails, the part where the skeleton was desired to be extracted is not indispensable for the subject's movement (for example, in the case of the subject's walking video, the subject's walking video). In the case of the neck), the accuracy derivation unit 18 derives the second accuracy information indicating that the accuracy is high as the second accuracy information. With this configuration, the terminal device 1 can notify the user that the accuracy of the parameter is low when the skeleton extraction of the portion essential for the operation of the target person fails.

With reference to FIG. 2 again, step S19 of the process by the terminal device 1 will be described.

(Step S19)
In step S19, the display unit 19 displays the moving image data including the parameters derived by the parameter derivation unit 17 in step S17 and the accuracy indicated by the second accuracy information derived by the accuracy derivation unit 18 in step S19 on the display panel 12. Supply. The display panel 12 displays the moving image indicated by the acquired moving image data.

(Example 4 of a moving image displayed by the display unit 19)
An example of a moving image displayed on the display panel 12 by the display unit 19 will be described with reference to FIG. FIG. 6 is a diagram showing still another example of the moving image displayed on the display panel 12 by the display unit 19 according to the present embodiment.

As shown in FIG. 6, the display unit 19 displays a moving image including the parameter PM1 derived by the parameter derivation unit 17 and the accuracy AC1 indicated by the second accuracy information derived by the accuracy derivation unit 18 on the display panel 12. .. Further, the display unit 19 may display a moving image including the parameter PM2 for each evaluation item on the display panel 12. With this configuration, the terminal device 1 can notify the user of the accuracy of the displayed parameter PM1.

Further, the display unit 19 may be configured to display the accuracy indicated by the second accuracy information for each evaluation item. In this configuration, the accuracy derivation unit 18 derives the second accuracy information for each evaluation item. For example, when the moving image includes the movement of the subject walking and the movement of the subject stopping and raising the arm, the skeletal extraction by the skeletal extraction process failed during the period when the subject was walking. When the number of frames is larger than a predetermined number, the accuracy derivation unit 18 derives the second accuracy information so that the accuracy of the evaluation item “speed” evaluated in the walking motion of the subject becomes low.

(Example 5 of a moving image displayed by the display unit 19)
The display unit 19 may be configured to change the mode of displaying the accuracy according to the accuracy indicated by the second accuracy information derived by the accuracy derivation unit 18. An example of a moving image displayed on the display panel 12 by the display unit 19 in this configuration will be described with reference to FIG. 7. FIG. 7 is a diagram showing still another example of the moving image displayed on the display panel 12 by the display unit 19 according to the present embodiment.

As shown in FIG. 7, when the accuracy indicated by the second accuracy information is lower than a predetermined value, the display unit 19 displays an image including the message DP2 prompting to take a moving image again on the display panel 12. With this configuration, the terminal device 1 can notify the user that it is necessary to take a moving image again in order to derive a parameter with high accuracy.

Further, as shown in FIG. 7, in addition to the message DP2 prompting to take a moving image, the display unit 19 includes a parameter PM3 derived by the parameter derivation unit 17, a parameter PM4 for each evaluation item, and an accuracy derivation unit 18. The accuracy indicated by the derived second accuracy information may be displayed.

In addition to the moving image shown in FIG. 7, the display unit 19 may be configured to change the display mode as follows depending on whether the accuracy indicated by the second accuracy information is higher or lower than a predetermined value. good.
-Change the color of the accuracy text indicated by the second accuracy information-Change the size of the accuracy text indicated by the second accuracy information-If the accuracy indicated by the second accuracy information is higher than the predetermined value Notify the user that it is not necessary to take another moving image (for example, display the text "Measurement completed properly")
However, these are not limited to this embodiment.

As described above, the terminal device 1 superimposes and displays the skeleton of the target person extracted from the moving image including the target person as the subject by using the first algorithm on the target person in the moving image, and displays one or a plurality of them. Using the second algorithm, the skeleton of the subject extracted from the moving image is referred to, and the parameters related to the movement of the subject's body are derived. Therefore, the terminal device 1 superimposes the skeleton of the target person extracted by the first algorithm on the target person and displays the skeleton of the target person by referring to the skeleton of the target person extracted by the second algorithm. Since the parameters related to the movement of the body of the subject are derived, it is possible to evaluate the movement of the subject's body with high accuracy while notifying the user who shoots the subject whether or not an appropriate moving image can be shot.

Further, the terminal device 1 executes a skeleton extraction process for extracting the skeleton of the target person from a moving image including the target person as a subject, and derives a parameter related to the movement of the body of the target person by referring to the result of the skeleton extraction process. Then, the accuracy information indicating the accuracy of the parameter is derived according to the frame in which the skeleton extraction by the skeleton extraction process fails among the plurality of frames included in the moving image. Therefore, since the terminal device 1 derives the accuracy of the parameter according to the frame in which the skeleton extraction by the skeleton extraction process fails, the reliability of the parameter can be notified to the user.

[Embodiment 2]
Other embodiments of the present invention will be described below. For convenience of explanation, the same reference numerals are given to the members having the same functions as the members described in the above-described embodiment, and the description thereof will not be repeated.

(Information processing system 100)
An outline of the information processing system 100 according to the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram showing an outline of the information processing system 100 according to the present embodiment. As shown in FIG. 8, the information processing system 100 includes a terminal device 1A and an information processing device 3.

As shown in FIG. 8, the terminal device 1A and the information processing device 3 are connected to each other so as to be able to communicate with each other via the network N. Here, examples of the terminal device 1A include a smartphone, a PC (Personal Computer), and a tablet, and examples of the information processing device 3 include a server, but these are not limited to this embodiment.

The specific configuration of the network N is not limited to this embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, etc. Alternatively, a combination of these networks can be used.

In the information processing system 100 according to the present embodiment, the terminal device 1A superimposes and displays the skeleton of the target person extracted from the moving image captured by the user of the terminal device 1A on the moving image. Then, the information processing device 3 acquires a moving image from the terminal device 1A, refers to the skeleton of the target person extracted from the moving image, and derives parameters related to the movement of the body of the target person. Further, the information processing apparatus 3 derives the accuracy information indicating the accuracy of the derived parameters according to the frame in which the skeleton extraction by the skeleton extraction process fails.

In the present embodiment as well, as in the first embodiment, the care recipient can be mentioned as an example of the subject, but this is not limited to the present embodiment. Further, as a user of the terminal device 1A, a care worker who cares for the person to be cared for can be mentioned, but this does not limit the present embodiment.

(Terminal device 1A)
A configuration example of the terminal device 1A according to the present embodiment will be described with reference to FIG. 9. FIG. 9 is a block diagram showing a configuration example of the terminal device 1A and the information processing device 3 included in the information processing system 100 according to the present embodiment. As shown in FIG. 9, the terminal device 1A includes a control unit 10A instead of the control unit 10 in the terminal device 1 in the first embodiment, and further includes a communication unit 11.

The communication unit 11 is a communication module for performing communication via the network N. The communication unit 11 outputs the data acquired from the control unit 10A to the information processing device 3 via the network N, and supplies the data acquired from the information processing device 3 via the network N to the control unit 10A. do.

(Control unit 10A)
The control unit 10A controls each component of the terminal device 1A. The control unit 10A also functions as an acquisition unit 15, a skeleton extraction unit 16, a display unit 19, and an output unit 20. Since the function of the display unit 19 is the same as that of the display unit 19 in the first embodiment, the description thereof will be omitted.

In addition to the function of the acquisition unit 15 in the first embodiment, the acquisition unit 15 acquires data from the information processing device 3 via the communication unit 11. Examples of the data acquired by the acquisition unit 15 include, but are not limited to, parameters and second accuracy information.

The skeleton extraction unit 16 executes a skeleton extraction process for extracting the skeleton of the target person from a moving image including the target person as a subject by using the first algorithm. The skeleton of the subject extracted by the first algorithm is referred to in real time by the display unit 19 described later as the first skeleton information indicating the skeleton. In addition, the first algorithm generates first accuracy information indicating the accuracy of the extracted skeleton. The first accuracy information is referred to by the display unit 19 described later. As in the first embodiment, it is preferable that the first algorithm is an algorithm having a faster skeleton extraction process than the second algorithm.

Further, the skeleton extraction unit 16 may generate error information indicating a frame in which the skeleton extraction has failed among the frames included in the moving image during the skeleton extraction using the first algorithm. The error information is as described in the first embodiment.

The output unit 20 outputs data to the information processing device 3 via the communication unit 11. Examples of the data output by the output unit 20 include, but are not limited to, moving images and error information.

(Information processing device 3)
As shown in FIG. 9, the information processing apparatus 3 includes a control unit 30, a communication unit 31, and a storage unit 34.

The communication unit 31 is a communication module for performing communication via the network N. The communication unit 31 outputs the data acquired from the control unit 30 to the terminal device 1 via the network N, and supplies the data acquired from the terminal device 1A via the network N to the control unit 30.

The storage unit 34 stores various data referred to by the control unit 30 described later. As an example, the storage unit 34 stores a moving image acquired from the terminal device 1A, a derived parameter, and the like.

(Control unit 30)
The control unit 30 controls each component of the information processing apparatus 3. As an example, the control unit 30 acquires a moving image output from the terminal device 1A via the communication unit 31 or outputs data generated via the communication unit 31.

Further, as shown in FIG. 9, the control unit 30 also functions as an acquisition unit 35, a skeleton extraction unit 36, a parameter derivation unit 37, an accuracy derivation unit 38, and an output unit 40.

The acquisition unit 35 acquires data from the terminal device 1A via the communication unit 31. An example of the data acquired by the acquisition unit 35 is, but is not limited to, a moving image.

The skeleton extraction unit 36 executes a skeleton extraction process for extracting the skeleton of the target person from the moving image acquired by the acquisition unit 35 by using the second algorithm. The result of the subject's skeleton extracted by the second algorithm is referred to by the parameter derivation unit 37 described later as the second skeleton information indicating the skeleton.

Further, the skeleton extraction unit 36 may generate error information indicating a frame in which the skeleton extraction has failed among the frames included in the moving image during the skeleton extraction using the second algorithm.

Similar to the parameter derivation unit 17 described above, the parameter derivation unit 37 derives parameters related to the movement of the body of the subject with reference to the result of the skeleton extraction process executed by the skeleton extraction unit 36. The parameters related to the movement of the body of the subject are as described in the first embodiment.

Similar to the accuracy derivation unit 18 described above, the accuracy derivation unit 38 extracts the second accuracy information indicating the accuracy of the parameters derived by the parameter derivation unit 37 from the plurality of frames included in the moving image by skeleton extraction processing. Derives according to the failed frame.

The output unit 40 outputs data to the terminal device 1A via the communication unit 31. Examples of the data output by the output unit 40 include, but are not limited to, parameters and second accuracy information.

(Processing in the information processing system 100)
The processing in the information processing system 100 will be described with reference to FIG. FIG. 4 is a sequence diagram showing a processing flow in the information processing system 100 according to the present embodiment.

(Step S12 to Step S15)
Since the processes of steps S12 to S15 from the start of the acquisition of the moving image to the end of the acquisition of the moving image by the terminal device 1A are the same as the above-mentioned processes, the description thereof will be omitted.

(Step S20)
In step S20, the output unit 20 of the terminal device 1A outputs the moving image to the information processing device 3 via the communication unit 11. Here, when the skeleton extraction unit 16 generates at least one of the first accuracy information and the error information, the output unit 20 receives at least one of the generated first accuracy information and the error information together with the moving image. May be output. Further, the output unit 20 may also output the camera parameters of the image pickup unit 13, the image size of the moving image, and the like for the parameter derivation process by the parameter derivation unit 37 described later.

(Step S21)
In step S21, the acquisition unit 35 of the information processing device 3 acquires a moving image from the terminal device 1A via the communication unit 31.

(Step S22)
In step S22, the skeleton extraction unit 36 executes a skeleton extraction process for extracting the skeleton of the subject from the moving image acquired by the acquisition unit 35 by using the second algorithm. More specifically, the skeleton extraction unit 36 extracts the skeleton of the subject from the moving image by using one or a plurality of second algorithms, and generates the second skeleton information indicating the extracted skeleton. Further, when the first accuracy information is output from the terminal device 1A, the skeleton extraction unit 36 selects one or a plurality of second algorithms from a plurality of algorithm candidates with reference to the first accuracy information. , The skeleton of the subject may be extracted using the selected one or a plurality of second algorithms. The details of the process of selecting one or a plurality of second algorithms from a plurality of algorithm candidates are as described above. Further, in step S22, the skeleton extraction unit 36 may generate error information.

(Step S23)
In step S23, the parameter derivation unit 37 derives parameters related to the movement of the body of the subject with reference to the result of the skeleton extraction process executed in step S22. More specifically, the parameter derivation unit 37 refers to the second skeleton information generated by the skeleton extraction unit 36, and derives parameters related to the movement of the body of the subject. The details of the process of deriving the parameters by the parameter deriving unit 37 are as described above.

(Step S24)
In step S24, the accuracy derivation unit 38 uses the second accuracy information indicating the accuracy of the parameter derived in step S23 as the frame in which the skeleton extraction by the skeleton extraction process fails among the plurality of frames included in the moving image. Derived according to. More specifically, the accuracy derivation unit 38 refers to at least one of the error information output from the terminal device 1A in step S20 and the error information generated in step S22, and derives the second accuracy information. The details of the process by which the accuracy derivation unit 38 derives the second accuracy information are as described above.

(Step S25)
In step S25, the output unit 40 outputs the parameter derived in step S23 and the second accuracy information derived in step S24 to the terminal device 1A via the communication unit 31.

(Step S26)
In step S26, the acquisition unit 15 of the terminal device 1A acquires the parameters output from the information processing device 3 and the second accuracy information via the communication unit 11.

(Step S27)
In step S27, the display unit 19 displays a moving image including the parameters acquired by the acquisition unit 15 in step S26 and the second accuracy information on the display panel 12. As an example of the image displayed by the display unit 19 in step S27, the above-mentioned images of FIGS. 6 and 7 can be mentioned.

As described above, the terminal device 1A in the information processing system 100 displays the skeleton of the target person extracted from the moving image including the target person as the subject by using the first algorithm, superimposed on the target person in the moving image. .. Further, the information processing apparatus 3 acquires a moving image including the target person as a subject, refers to a skeleton extracted from the acquired moving image using the second algorithm, and derives parameters related to the movement of the target person's body. .. Therefore, the information processing system 100 can accurately evaluate the movement of the body of the target person while notifying the user who shoots the target person whether or not an appropriate moving image can be shot.

Further, the information processing apparatus 3 in the information processing system 100 executes a skeleton extraction process for extracting the skeleton of the target person from a moving image including the target person as a subject, and refers to the result of the skeleton extraction process to refer to the body of the target person. Derivation of parameters related to the operation of. Further, the information processing apparatus 3 derives the accuracy information indicating the accuracy of the derived parameters according to the frame in which the skeleton extraction by the skeleton extraction process fails among the plurality of frames included in the moving image. Therefore, the information processing apparatus 3 derives the accuracy of the parameter according to the frame in which the skeleton extraction by the skeleton extraction process fails, so that the reliability of the parameter can be notified to the user.

(Physical configuration of terminal device 1A)
FIG. 11 is a block diagram illustrating the physical configuration of a computer used as the terminal device 1A (1) and the information processing device 3. As shown in FIG. 11, the terminal device 1A can be configured by a computer including a bus 110, a processor 101, a main memory 102, an auxiliary memory 103, a communication interface 104, a camera 105, and a display panel 106. be. The processor 101, the main memory 102, the auxiliary memory 103, the communication interface 104, the camera 105, and the display panel 106 are connected to each other via the bus 110.

As the processor 101, for example, a CPU (Central Processing Unit), a microprocessor, a digital signal processor, a microcontroller, or a combination thereof or the like is used.

As the main memory 102, for example, a semiconductor RAM (random access memory) or the like is used.

As the auxiliary memory 103, for example, a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination thereof or the like is used. The auxiliary memory 103 stores a program for causing the processor 101 to execute the operation of the terminal device 1A described above. The processor 101 expands the program stored in the auxiliary memory 103 on the main memory 102, and executes each instruction included in the expanded program.

The communication interface 104 is an interface connected to the network N.

The camera 105 is an example of a hardware element that realizes the image pickup unit 13.

The display panel 106 is an example of hardware elements that realize the display panel 12.

In this example, the processor 101 is an example of a hardware element that realizes the control unit 10A. Further, the main memory 102 and the auxiliary memory 103 are examples of hardware elements that realize the storage unit 14. Further, the communication interface 104 is an example of a hardware element that realizes the communication unit 11.

(Physical configuration of information processing device 3)
As shown in FIG. 11, the information processing apparatus 3 can be configured by a computer including a bus 310, a processor 301, a main memory 302, an auxiliary memory 303, and a communication interface 304. The processor 301, the main memory 302, the auxiliary memory 303, and the communication interface 104 are connected to each other via the bus 110.

In this example, the processor 301 is an example of a hardware element that realizes the control unit 30. Further, the main memory 302 and the auxiliary memory 303 are examples of hardware elements that realize the storage unit 34. Further, the communication interface 304 is an example of a hardware element that realizes the communication unit 31.

[Example of implementation by software]
The functions of the terminal devices 1, 1A, and the information processing device 3 (hereinafter collectively referred to as "devices") are programs for operating the computer as the device, and each control block (particularly control) of the device is used. Each part included in parts 10, 10A, and 30) can be realized by a program for operating a computer.

In this case, the device includes a computer having at least one control device (for example, a processor) and at least one storage device (for example, memory) as hardware for executing the program. By executing the above program by the control device and the storage device, each function described in each of the above embodiments is realized.

The program may be recorded on one or more computer-readable recording media rather than transiently. This recording medium may or may not be provided in the above-mentioned apparatus. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

Further, some or all of the functions of the above control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit functioning as each of the above control blocks is formed is also included in the category of the present invention. In addition to this, it is also possible to realize the function of each of the above control blocks by, for example, a quantum computer.

Further, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may be operated by the above-mentioned control device, or may be operated by another device (for example, an edge computer or a cloud server).

An embodiment of the present invention will be described below.

A graph showing the relationship between the measured value of the walking speed of the subject measured using the three-dimensional motion analysis device and the estimated value of the walking speed estimated using the gait analysis AI from the moving image of the subject walking. , FIG. 12. FIG. 12 is a graph showing the relationship between the estimated value and the actually measured value in Example 1.

〇, Δ, and × in FIG. 12 indicate the following moving images, respectively.
Δ: There is a frame in which skeleton extraction failed during the period from the start of the moving image to the start of the motion, or from the end of the motion to the end of the motion image. Moving image with failed frames 〇: Moving image with no frame that failed to extract the skeleton As shown in Fig. 12, if there is a frame that failed to extract the skeleton in the moving image during operation, the estimated walking speed is reliable. It was found that the degree was low (the deviation of the estimated value from the measured value was large). On the other hand, when there is a frame in which the skeleton extraction fails in the period from the start of the moving image to the start of the motion, or from the end of the motion to the end of the moving image, or when there is no frame in which the skeleton extraction fails. It was found that the estimated walking speed was highly reliable (the deviation of the estimated value from the measured value was small).

Other embodiments of the present invention will be described below.

A graph showing the relationship between the measured value of head wobbling measured using a three-dimensional motion analyzer and the estimated value of head wobbling estimated using gait analysis AI from a moving image of a subject walking. Is shown in FIG. FIG. 13 is a graph showing the relationship between the estimated value and the actually measured value in Example 2.

〇, Δ, and × in FIG. 13 also show the following moving images, respectively.
Δ: There is a frame in which skeleton extraction failed during the period from the start of the moving image to the start of the motion, or from the end of the motion to the end of the motion image. Moving image with failed frames 〇: Moving image with no frame that failed to extract the skeleton As shown in Fig. 13, if there is a frame that failed to extract the skeleton in the moving image during operation, the estimated value of head wobbling. Was found to have low reliability (large deviation of the estimated value from the measured value). On the other hand, when there is a frame in which the skeleton extraction fails in the period from the start of the moving image to the start of the operation, or from the end of the operation to the end of the moving image, or when there is no frame in which the skeleton extraction fails. It was found that the estimated value of head wobbling is highly reliable (the deviation of the estimated value from the measured value is small).

[summary]
In the information processing method according to one embodiment of each embodiment, a skeleton extraction process for extracting the skeleton of the subject from a moving image including the subject as a subject is executed, and the result of the skeleton extraction process is referred to to the subject. Of the plurality of frames included in the moving image, the skeleton extraction by the skeleton extraction process can obtain the accuracy information indicating the accuracy of the parameter derived in the parameter derivation step and the parameter derivation step for deriving the parameters related to the movement of the body. Includes an accuracy derivation step, which derives according to the failed frame.

According to the above configuration, it is possible to notify the user of the reliability of the evaluation of the body movement of the subject.

In the information processing method according to one embodiment of each embodiment, in the accuracy derivation step, it is preferable to derive the accuracy information according to the number of frames in which the skeleton extraction by the skeleton extraction process has failed.

According to the above configuration, when there are many frames in which the skeleton extraction fails, it is possible to notify the user that the reliability is low.

In the information processing method according to one embodiment of each embodiment, in the accuracy derivation step, it is preferable to derive the accuracy information according to the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails.

According to the above configuration, if there is a frame in which the skeleton extraction fails during the operation of the target person, it is possible to notify the user that the reliability is low.

In the information processing method according to one embodiment of each embodiment, in the accuracy derivation step, in the frame in which the skeleton extraction by the skeleton extraction process fails, the accuracy information is derived according to the portion where the skeleton was not extracted. Is preferable.

According to the above configuration, when there is a frame in which the skeleton extraction of the portion related to the movement of the target person fails, it is possible to notify the user that the reliability is low.

It is preferable that the information processing method according to one embodiment of each embodiment further includes a display step for displaying the accuracy indicated by the accuracy information derived in the accuracy derivation step.

According to the above configuration, it is possible to notify the user of the reliability of the evaluation of the body movement of the subject.

In the information processing method according to one aspect of each embodiment, in the display step, it is preferable to change the mode of displaying the accuracy according to the accuracy indicated by the accuracy information derived in the accuracy derivation step.

According to the above configuration, when the reliability of the evaluation of the body movement of the subject is low, the user can be appropriately notified.

In the information processing method according to one embodiment of each embodiment, in the display step, when the accuracy indicated by the accuracy information derived in the accuracy derivation step is lower than a predetermined value, a message prompting the user to take a moving image again is displayed. It is preferable to display it.

According to the above configuration, if the reliability of the evaluation of the body movement of the subject is low, the user can be prompted to take a picture again.

The information processing apparatus according to one embodiment of each embodiment executes a skeleton extraction process for extracting the skeleton of the subject from a moving image including the subject as a subject, and refers to the result of the skeleton extraction process to refer to the subject. Of the plurality of frames included in the moving image, the skeleton extraction by the skeleton extraction process can obtain the accuracy information indicating the accuracy of the parameters derived by the parameter derivation unit and the parameter derivation unit for deriving the parameters related to the movement of the body. It is provided with an accuracy derivation unit for deriving according to a failed frame.

According to the above configuration, it is possible to notify the user of the reliability of the evaluation of the body movement of the subject.

The program according to one aspect of each embodiment is a program for operating a computer as the above-mentioned information processing apparatus, and causes the computer to function as the parameter derivation unit and the accuracy derivation unit.

According to the above configuration, it is possible to notify the user of the reliability of the evaluation of the body movement of the subject.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1, 1A Terminal device 3 Information processing device 10, 10A, 30 Control unit 11, 31 Communication unit 12 Display panel 13 Imaging unit 14, 34 Storage unit 15, 35 Acquisition unit 16, 36 Skeleton extraction unit 17, 37 Parameter derivation unit 18 , 38 Accuracy derivation unit 19 Display unit 20, 40 Output unit 100 Information processing system

Claims (9)

A parameter derivation step of executing a skeleton extraction process for extracting the skeleton of the target person from a moving image including the target person as a subject and deriving parameters related to the body movement of the target person by referring to the result of the skeleton extraction process. ,
The accuracy derivation step of deriving the accuracy information indicating the accuracy of the parameters derived in the parameter derivation step according to the frame in which the skeleton extraction by the skeleton extraction process fails among the plurality of frames included in the moving image.
Including
In the parameter derivation step, the parameter is derived by at least referring to the result of the skeleton extraction process in the frame in which the skeleton extraction by the skeleton extraction process did not fail among the plurality of frames included in the moving image. Information processing method. The information processing method according to claim 1, wherein in the accuracy derivation step, the accuracy information is derived according to the number of frames in which the skeleton extraction by the skeleton extraction process fails. The information processing method according to claim 1 or 2, wherein in the accuracy derivation step, the accuracy information is derived according to the temporal position of the frame in which the skeleton extraction by the skeleton extraction process fails. The information according to any one of claims 1 to 3 for deriving the accuracy information according to the portion where the skeleton was not extracted in the frame in which the skeleton extraction by the skeleton extraction process failed in the accuracy derivation step. Processing method. The information processing method according to any one of claims 1 to 4, further comprising a display step for displaying the accuracy indicated by the accuracy information derived in the accuracy derivation step. The information processing method according to claim 5, wherein in the display step, the mode of displaying the accuracy is changed according to the accuracy indicated by the accuracy information derived in the accuracy derivation step. The information processing method according to claim 6, wherein in the display step, when the accuracy indicated by the accuracy information derived in the accuracy derivation step is lower than a predetermined value, a message prompting the user to take a moving image again is displayed. A parameter derivation unit that executes a skeleton extraction process that extracts the skeleton of the target person from a moving image including the target person as a subject, and derives parameters related to the body movement of the target person by referring to the result of the skeleton extraction process. ,
An accuracy derivation unit that derives accuracy information indicating the accuracy of the parameters derived by the parameter derivation unit according to a frame in which the skeleton extraction by the skeleton extraction process fails among a plurality of frames included in the moving image.
Equipped with
The parameter derivation unit derives the parameter by at least referring to the result of the skeleton extraction process in the frame in which the skeleton extraction by the skeleton extraction process did not fail among the plurality of frames included in the moving image. Information processing device. The program for operating a computer as the information processing apparatus according to claim 8, wherein the computer functions as the parameter derivation unit and the accuracy derivation unit.

Images