JP2021071773A - Motion evaluation device, motion evaluation method, and motion evaluation system - Google Patents

Abstract

To provide a motion evaluation device capable of realizing the same evaluation as an instructor evaluation using the dynamic image photographing a training state.SOLUTION: A motion evaluation method to evaluate a trainee motion comprises a step to detect a partial dynamic image corresponding to each of a plurality of partial motions from whole dynamic image photographing the whole motion including a series of the partial motions, using a detecting model generated by machine learning, and comprises a step to evaluate at least one of the whole motion and the partial motion in reference to the plurality of detected partial dynamic images.SELECTED DRAWING: Figure 3

Description

The present invention relates to an operation evaluation device, an operation evaluation method, and an operation evaluation system.

Streamline learning by shooting videos of training in various fields (for example, nursing care, medical care, sports, etc.) and using the videos to provide guidance by instructors with specialized knowledge. Efforts are being made.

The trainee who conducts the training may learn the video including the state of the training taken by himself / herself when he / she cannot receive the guidance of the instructor. Learning using videos of training is useful from the viewpoint of improving the learning efficiency of trainees.

For example, in Patent Document 1, in the field of sports, a value indicating the operating state of a subject is calculated based on the ratio of the distance between the joints of the subject estimated from the image and the reference length, and , A technique for evaluating the movement of a subject based on a value indicating the operation state is disclosed.

International Publication WO2019082376 Gazette

In actual training, the instructor evaluates a series of movements as a whole, not just individual movements.

However, in Patent Document 1, although the operation is evaluated individually, the series of operations is not evaluated as a whole.
Therefore, the evaluation obtained by Patent Document 1 is significantly different from the evaluation of the leader.

An object of the present invention is to realize an evaluation equivalent to that of an instructor by using a moving image of a state of training.

One aspect of the present invention is
It is an operation evaluation method that evaluates the operation of the trainee.
Using a detection model generated by machine learning, a step is provided to detect a partial moving image corresponding to each of a plurality of partial movements from a whole moving image in which a whole motion including a series of partial motions is shot.
It comprises a step of evaluating at least one of the whole motion and the partial motion with reference to the detected plurality of partial motions.
This is an operation evaluation method.

According to the present invention, it is possible to realize an evaluation equivalent to that of an instructor by using a moving image of a state of training.

It is a block diagram which shows the structure of the operation evaluation system of 1st Embodiment. It is a functional block diagram of the operation evaluation system of FIG. It is explanatory drawing of the outline of 1st Embodiment. It is a figure which shows the data structure of the user information database of this embodiment. It is a figure which shows the data structure of the operation identification information database of this embodiment. It is a figure which shows the data structure of the partial operation information database of this embodiment. It is a figure which shows the data structure of the whole moving image information database of this embodiment. It is a figure which shows the data structure of the evaluation condition data table of this embodiment. It is explanatory drawing of the teacher data of the detection model of 1st Embodiment. It is a sequence diagram of the evaluation process of the operation of 1st Embodiment. It is a detailed flowchart of the evaluation of FIG. It is a figure which shows the screen example displayed in the information processing of FIG. It is explanatory drawing of the detection of the partial moving image of FIG. It is a figure which shows the data structure of the partial evaluation information of 1st Embodiment. It is a sequence diagram of the information processing of the modification 1. It is a figure which shows the screen example displayed in the information processing of FIG. It is explanatory drawing of the outline of the modification 2. It is explanatory drawing of the outline of the modification 3.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. In addition, in the drawing for demonstrating the embodiment, the same components are in principle the same reference numerals, and the repeated description thereof will be omitted.

(1) First Embodiment The first embodiment will be described.

(1-1) Configuration of Operation Evaluation System The configuration of the operation evaluation system of the first embodiment will be described. FIG. 1 is a block diagram showing a configuration of an operation evaluation system according to the first embodiment. FIG. 2 is a functional block diagram of the operation evaluation system of FIG.

As shown in FIG. 1, the operation evaluation system 1 includes a client device 10 and a server 30.
The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

The client device 10 is a computer (an example of an "information processing device") that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is a computer (an example of an "operation evaluation device") that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a web server.

(1-1-1) Configuration of Client Device The configuration of the client device 10 will be described.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, a communication interface 14, and a camera 15.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
-OS (Operating System) program-Program of an application that executes information processing (for example, a shooting application)

The data includes, for example, the following data.
-Database referenced in information processing-Data obtained by executing information processing (that is, the execution result of information processing)

The processor 12 is configured to realize the function of the client device 10 by activating the program stored in the storage device 11. The processor 12 is, for example, a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

The input / output interface 13 is configured to acquire a user's instruction from an input device connected to the client device 10 and output information to an output device connected to the client device 10.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 14 is configured to control communication between the client device 10 and the server 30.

The camera 15 is configured to capture an image (eg, a moving image).

(1-1-2) Server Configuration The configuration of the server 30 will be described.

As shown in FIG. 2, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS program ・ Application program that executes information processing

The data includes, for example, the following data.
・ Database referenced in information processing ・ Execution result of information processing

The processor 32 is configured to realize the function of the server 30 by activating the program stored in the storage device 31. The processor 32 is, for example, a CPU, an ASIC, an FPGA, or a combination thereof.

The input / output interface 33 is configured to acquire a user's instruction from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and the client device 10.

(1-2) Outline of First Embodiment An outline of the first embodiment will be described. FIG. 3 is an explanatory diagram of an outline of the first embodiment.

As shown in FIG. 3, the client device 10 transmits the entire moving image to the server 30. The whole movie is a movie in which the whole movement of the traini is shot. The whole operation includes a series of multiple partial operations.

The server 30 uses the detection model generated by machine learning to detect the partial moving image corresponding to each of the plurality of partial motions from the whole moving image.
The server 30 evaluates at least one of the traini's overall operation and partial operation with reference to the detected plurality of partial moving images.

(1-3) Data structure The data structure of the first embodiment will be described. The following data is stored in the storage device 31.

(1-3-1) User Information Database The user information database of the present embodiment will be described. FIG. 4 is a diagram showing a data structure of the user information database of the present embodiment.

User information is stored in the user information database of FIG. User information is information about the user.
The user information database includes a "user ID" field, a "user name" field, a "user attribute" field, and a "client ID" field. Each field is associated with each other.

User identification information is stored in the "User ID" field. User identification information is information that identifies a user.

The Username field stores information about the username (eg, text).

User attribute information is stored in the "user attribute" field. User attribute information is information related to user attributes. The user's attributes include, for example, at least one of the following:
・ Gender ・ Age ・ Occupation (caregiver, doctor, athlete, etc.)

Client identification information is stored in the "client ID" field. The client identification information is information that identifies the client device 10 of the user.

(1-3-2) Operation type information database The operation type information database of the present embodiment will be described. FIG. 5 is a diagram showing a data structure of the operation identification information database of the present embodiment.

The operation type information database of FIG. 5 stores the operation type information. The operation type information is information related to the type of operation.
The operation type information database includes an "overall operation ID" field, an "overall operation name" field, and a "partial operation ID" field. Each field is associated with each other.

The overall operation identification information is stored in the "overall operation ID" field. The overall operation identification information is information that identifies the overall operation.

The "Overall Action Name" field stores information about the overall action name (eg, text). The overall operation includes, for example, at least one of the following.
・ Nursing care-related actions (for example, hand washing, standing assistance, walking assistance, etc.)
・ Medical movements (for example, suturing, incision, etc.)
・ Sports-related movements (for example, pitching, running, walking, etc.)

The "partial operation ID" field stores partial operation identification information that identifies the partial operation that constitutes the entire operation.

(1-3-3) Partial operation information database The partial operation information database of the present embodiment will be described. FIG. 6 is a diagram showing a data structure of the partial operation information database of the present embodiment.

The partial operation information database of FIG. 6 stores partial operation information. The partial operation information is information related to the partial operation.
The partial operation information database includes a "partial operation ID" field, a "partial operation name" field, and a "model ID" field. Each field is associated with each other.
The partial operation information database is associated with the total operation identification information.

The partial operation identification information is stored in the "partial operation ID" field. The partial motion identification information is information for identifying the partial motions constituting the entire motion.

The "partial action name" field stores information about the partial action name (eg, text). For example, the partial movement of the whole movement "hand washing" includes "washing the palm", "washing the back of the hand", "washing the fingertips", "rinsing" and the like.

Model identification information is stored in the "model ID" field. The model identification information is information for identifying a detection model that detects a frame corresponding to a partial motion (hereinafter referred to as "partial moving image") from the whole moving image.

(1-3-4) Overall video information database The overall video information database of the present embodiment will be described. FIG. 7 is a diagram showing a data structure of the entire moving image information database of the present embodiment.

The whole moving image information database of FIG. 7 stores the whole moving image information. The whole video information is information about the whole video.
The overall video information database includes an "overall video ID" field, an "overall video" field, and an "evaluation result" field. Each field is associated with each other.
The overall video information database is associated with user identification information.

The entire moving image identification information is stored in the "overall moving image ID" field. The whole moving image identification information is information for identifying the whole moving image.

The entire video is stored in the "whole video" field.

Evaluation result information is stored in the "evaluation result" field. The evaluation result information is information regarding the evaluation result for the whole motion and the partial motion included in the whole moving image. The "evaluation result" field includes an "evaluation date and time" field, a "partial evaluation" field, and an "overall evaluation" field.

The "evaluation date and time" field stores information about the evaluation date and time.

Partial evaluation information is stored in the "partial evaluation" field. The partial evaluation information is information regarding the evaluation result of the partial operation.

The overall evaluation information is stored in the "overall evaluation" field. The overall evaluation information is information regarding the evaluation result of the overall operation.

(1-3-5) Evaluation condition data table The evaluation condition data table of the present embodiment will be described. FIG. 8 is a diagram showing a data structure of the evaluation condition data table of the present embodiment.

The evaluation condition information is stored in the evaluation condition data table of FIG. The evaluation condition information is information on the evaluation conditions referred to when evaluating the overall operation and the partial operation.
The evaluation condition data table includes an "evaluation index" field and an "evaluation score" field. Each field is associated with each other.
The evaluation condition data table is associated with the partial operation identification information. That is, the evaluation conditions are prepared for each partial operation.

Evaluation index information is stored in the "evaluation index" field. The evaluation index information is information related to the evaluation index. The evaluation index includes, for example, the following.
・ "Presence" ... Presence or absence of partial video ・ "Order" ... Order of partial video in the whole video ・ "Period" ... Length of the partial video ・ "Start time" ... From the start time of the whole video to the start time of the partial video elapsed time

In the "evaluation score" field, conditions related to the evaluation score for each evaluation index are stored.
For example, when there is no partial motion (that is, the partial motion is not executed), the evaluation score of the partial motion is "0". When there is a partial moving image (that is, the partial motion is executed), the evaluation score of the partial motion is "1".
If the order of the partial moving images is incorrect (that is, the partial movements are not executed according to the predetermined order), the evaluation score of the partial movements is "0". When the order of the partial moving images is correct (that is, the partial movements are executed according to a predetermined order), the evaluation score of the partial movements is "1".
When the length of the partial moving image is less than the predetermined period threshold value Pth (that is, the partial operation is not executed for the predetermined period), the evaluation score of the partial operation is “0”. When the length of the partial moving image is equal to or greater than the period threshold value Pth (that is, the partial motion is executed for a predetermined period of time), the evaluation score of the partial motion is “1”.
When the start time ST of the partial moving image is less than the predetermined first start time threshold value Sth1 (that is, the partial operation is executed by the predetermined timing), the evaluation score of the partial operation is “0”. When the start time ST of the partial moving image is equal to or more than the first start time threshold value Sth1 and less than the second start time threshold value Sth2 (> Sth1) (that is, the partial operation is executed within a predetermined period), The evaluation score of the partial movement is "1".

(1-4) Information Processing The information processing of the first embodiment will be described.

(1-4-1) Detection Model Generation Process The detection model generation process of the first embodiment will be described. FIG. 9 is an explanatory diagram of teacher data of the detection model of the first embodiment.

In order to execute the generation process of the detection model of the present embodiment, the teacher data, the machine learning model, and the sample partial moving image are prepared.

As shown in FIG. 9, the teacher data is a combination of a plurality of learning videos and a plurality of correct answer data.
The learning moving image is a random combination of the target moving image Y and the non-target moving image Xn different from the target moving image.
The correct answer data is time-series data indicating the position of the target moving image Y in each learning moving image.

The detection model is generated by letting the machine learning model machine-learn the teacher data. Machine learning is executed by adjusting the parameters of the machine learning model so that the correct answer data is output when the learning video is input. As a result, when the learning moving image is input, a general-purpose model configured to output the likelihood of the target moving image Y in time series is generated.

The general-purpose model is transferred and learned so that the target moving image Y becomes a partial moving image. As a result, a detection model for detecting a partial moving image from the whole moving image is generated.

(1-4-2) Operation evaluation process The operation evaluation process of the first embodiment will be described. FIG. 10 is a sequence diagram of the operation evaluation process of the first embodiment. FIG. 11 is a detailed flowchart of the evaluation of FIG. FIG. 12 is a diagram showing an example of a screen displayed in the information processing of FIG. FIG. 13 is an explanatory diagram of detection of the partial moving image of FIG. FIG. 14 is a diagram showing a data structure of the partial evaluation information of the first embodiment.

In the following example, the user of the client device 10 is a trainer (eg, a caregiver, a doctor, an athlete, etc.) who performs motion training.
The process of FIG. 10 is started by invoking the photographing application using the user identification information that identifies the user.

As shown in FIG. 10, the client device 10 executes the reception (S111) for designating the operation type.
Specifically, the processor 12 displays the screen G10 (FIG. 12) on the display.

The screen G10 includes operation objects B10a to B10d.
The operation objects B10a to B10c are objects for receiving a user instruction for designating the overall operation names "hand washing", "standing assistance", or "walking assistance", respectively. The operation objects B11a to B11c are assigned overall operation identification information of the overall operation names "hand washing", "standing assistance", and "walking assistance", respectively. When the user operates any of the operation objects B10a to B10c, the processor 12 converts the overall operation identification information assigned to any of the operated operation objects B10a to B10c into the overall operation identification information to be evaluated (hereinafter referred to as the overall operation identification information). Accepted as "object-wide operation identification information").
The operation object B10d is an object for receiving a user instruction for starting shooting of a moving image. When the operation object B10d is operated, the processor 12 causes the camera 15 to start shooting a moving image.

After step S111, the client device 10 executes shooting (S112) of the entire operation.
Specifically, when the user operates the operation objects B10a and B10d, the processor 12 causes the camera 15 to start shooting a moving image.
When the user gives a user instruction to end the video recording to the client device 10 (for example, when operating an operation object (not shown) that accepts the user instruction to end the video recording), the processor 12 moves the entire processor 12. The moving image is stored in the storage device 11, and the screen G11 (FIG. 12) is displayed on the display.

The screen G11 includes a moving object MOV11 and an operation object B11.
The moving image object MOV 11 is an object of thumbnail images of the entire moving image stored in the storage device 11.
The operation object B11 is an object that receives a user instruction for uploading the entire moving image to the server 30. When the user operates the operation object B11, the processor 12 uploads the entire moving image stored in the storage device 11 to the server 30.

After step S112, the client device 10 executes the evaluation request (S113).
Specifically, when the user operates the operation object B11, the processor 12 transmits the evaluation request data to the server 30. The evaluation request data includes the following information.
-User identification information specified when starting the shooting application-Overall operation identification information of the target received in step S111-Overall moving image stored in the storage device 11 in step S112

After step S113, the server 30 executes the evaluation (S131) according to the flow of FIG.

As shown in FIG. 11, the server 30 executes the determination of the detection model (S1311).
Specifically, the processor 32 refers to the operation type information database (FIG. 5), and the information in the "partial operation ID" field associated with the target-wide operation identification information included in the evaluation request data (that is, the user Specify the partial operation identification information (hereinafter referred to as "target partial operation identification information") of the partial operation that constitutes the entire operation of the specified operation type.
The processor 32 refers to the partial operation information database (FIG. 6) and refers to the information in the "model ID" field associated with each target partial operation identification information (that is, the partial operation corresponding to the target partial operation identification information (hereinafter, "" The model identification information) of the detection model for detecting (referred to as "target partial motion") is specified.

For example, in the case of the target overall motion identification information "T01" (overall motion name "hand wash"), the target partial motion identification information "P11" to "P14" (partial motion names "wash the palm", "wash the back of the hand", " "Wash your fingertips" and "rinse"), and model identification information "M11" to "M14" are identified (FIG. 6).

After step S1311, the server 30 executes partial moving image detection (S1312).
Specifically, the processor 32 inputs the entire moving image included in the evaluation request data for each detection model identified in step S1311. The processor 32 detects each partial moving image based on the output of each detection model.

More specifically, in the case of the target overall operation identification information "T01" (overall operation name "hand wash"), the processor 32 performs the entire detection model corresponding to the model identification information "M11" to "M14". Enter the video.
As shown in FIG. 13, each detection model (model identification information “M11” to “M14”) constitutes a partial operation corresponding to the target overall operation identification information “T01” (overall operation name “hand wash”). The likelihoods of the partial moving images P11 to P14 corresponding to (“washing the palm”, “washing the back of the hand”, “washing the fingertips”, and “rinsing”) are output in chronological order.
The processor 32 specifies the start times ST1 to ST4 and the end times ET1 to ET4 of each partial moving image P11 to P14 based on the output likelihood. As an example, the processor 32 identifies the start times ST1 to ST4 and the end times ET1 to ET4 by comparing the likelihood with a predetermined likelihood threshold.
The processor 32 stores in the storage device 31 a combination of the target partial motion identification information of the partial moving images P11 to P14, the start times ST1 to ST4 of the partial moving images P11 to P14, and the end times ET1 to ET4.

After step S1312, the server 30 executes the partial operation evaluation (S1313).
Specifically, the processor 32 identifies the evaluation condition data table (FIG. 8) associated with each target partial operation identification information stored in the storage device 31 in step S1312.
The processor 32 refers to the specified evaluation condition data table and refers to the information in the "evaluation index" field (for example, the presence / absence of partial moving images including each partial operation, the order of each partial moving image, the period of each partial moving image, and each of them. Identify the information (ie, the evaluation score) in the "evaluation score" field that corresponds to the "start time of the partial movement in the partial video". As a result, partial evaluation information (FIG. 14) is obtained.

After step S1313, the server 30 executes an overall operation evaluation (S1314).

In the first example of step S1314, the processor 32 determines the evaluation score of the overall operation (that is, the overall evaluation information) with reference to the partial evaluation information obtained in step S1313.
More specifically, the processor 32 determines at least one of the following as an evaluation score for overall operation.
-Total value of evaluation score of partial movement-Average value of evaluation score of partial movement-The total value or the index corresponding to the average value (for example, rank)

As an example, when the operation type is "wash the palm", the evaluation condition data table of FIG. 8 is referred to.
In this case, the processor 32 detects the partial moving image P11, the order of the partial moving images P11 is the same as the order shown in the evaluation conditions, the period of the partial moving image P11 is Pth or more, and the start time ST is the first start time. When it is included between the threshold value Sth1 and the second start time threshold value Sth2, the evaluation score of the partial moving image P11 is determined to be "4".

In the second example of step S1314, the processor 32 determines the evaluation score of the overall operation according to the number of partial moving images detected in step S1312.
In this case, step S1313 can be omitted.

After step S131, the server 30 executes the database update (S132).
Specifically, the processor 32 adds a new record to the overall moving image information database (FIG. 7) associated with the user identification information included in the evaluation request data. The following information is stored in each field of the new record.
-"Overall video ID" field ... New overall video identification information- "Overall video" field ... Overall video included in the evaluation request data- "Evaluation date and time" field ... Information about the execution date and time of step S131-"Partial evaluation" field ... Partial evaluation information obtained in step S1313 ・ "Overall evaluation" field ... Overall evaluation information obtained in step S1314

After step S132, the server 30 executes the evaluation response (S133).
Specifically, the processor 32 transmits the evaluation response data to the client device 10. The evaluation response data includes the following information.
-Information regarding the execution result of step S1312 (that is, the detection result of the partial moving image) -The execution result of step S1313 (that is, the partial evaluation information)
-Execution result of step S1314 (that is, overall evaluation information)

After step S133, the client device 10 executes the display of the evaluation result (S114).
Specifically, the processor 12 displays the screen G12 (FIG. 12) on the display.

Information included in the evaluation response data is displayed on the screen G12. The screen G12 includes display objects A12a to A12b, an image object IMG12, and a moving object MOV11.
Partial evaluation information is displayed on the display object A12a.
Overall evaluation information is displayed on the display object A12b.
The image object IMG12 is an object that shows the detection result of the partial moving image in time series.

In step S133, the evaluation response data does not have to include the partial evaluation information. In this case, in step S114, the display of the partial evaluation information is omitted.
Further, in step S133, the evaluation response data does not have to include the overall evaluation information. In this case, in step S114, the display of the overall evaluation information is omitted.

According to the first embodiment, the partial moving image is detected from the whole moving image, and the whole operation is evaluated by using the evaluation conditions related to the partial moving image. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

Further, according to the first embodiment, the partial moving image is detected by using the detection model for each partial operation. As a result, partial motion can be detected appropriately.

Further, according to the first embodiment, at least one of the total motion and the partial motion is evaluated according to the number of detected partial moving images. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

Further, according to the first embodiment, at least one of the total operation and the partial operation is referred to by referring to at least one of the detected order of the partial moving images, the period of each partial moving image, and the start time of each partial operation. To evaluate. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

Further, according to the first embodiment, at least one of the total operation and the partial operation is evaluated according to the type of the traini operation specified in advance. Thereby, more accurate evaluation can be realized.

Further, according to the first embodiment, the partial moving image is detected by using the detection model according to the type of the detected partial motion. As a result, the partial moving image can be detected appropriately.

Further, according to the first embodiment, the partial moving image is evaluated using the evaluation conditions according to the type of operation. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

(2) Second Embodiment The second embodiment will be described. The second embodiment is an example of evaluating a partial motion using a partial motion evaluation model corresponding to each of the plurality of partial motions. The same description as in the above embodiment will be omitted.

(2-1) Operation Evaluation Process The operation evaluation process of the second embodiment will be described.

The server 30 executes the determination of the detection model (S1311) to the detection of the partial moving image (S1312) in the same manner as in the first embodiment (FIG. 11).

After step S1312, the server 30 executes the partial operation evaluation (S1313).
Specifically, the storage device 31 stores a partial motion evaluation model for each partial motion. Each partial motion evaluation model is associated with partial motion identification information.
In the partial motion evaluation model, evaluation conditions for each element motion (for example, posture, facial expression, and at least one of utterances) included in each partial motion are described. The partial motion evaluation model is, for example, a predetermined rule or a trained model generated by machine learning.

The processor 32 refers to the partial motion evaluation model associated with the target partial motion identification information, and determines the evaluation score of the element motion included in the partial motion corresponding to each partial motion detected in step S1312.
The processor 32 determines the partial motion evaluation information of the partial motion by referring to the evaluation score of each element motion.
More specifically, the processor 32 determines at least one of the following as an evaluation score for partial operation.
-Total value of the evaluation score of the element movement-Average value of the evaluation score of the element movement-The total value or the index corresponding to the average value (for example, rank)
・ Presence / absence of partial video ・ Order of partial video ・ Period of partial video ・ Start time of partial operation in each partial video

As an example, the motion type is "walking assistance", the partial motion is "take the hand of the person being assisted", the element motion to be evaluated is "facial expression", and the evaluation condition of the element motion. When is "seeing the eyes of the person being assisted with a smile", the processor 32 identifies the traini's facial expression (eg, the level of smile) by analyzing the partial moving image.
The processor 32 calculates the evaluation score of the elemental motion "facial expression" based on the specific result (for example, the level of smile).

After step S1313, the server 30 executes the evaluation of the overall operation (S1314) as in the first embodiment (FIG. 10).

After step S1314, the server 30 executes the database update (S132) of FIG.
Specifically, the processor 32 adds a new record to the overall moving image information database (FIG. 7) associated with the user identification information included in the evaluation request data. The following information is stored in each field of the new record.
-"Overall video ID" field ... New overall video identification information- "Overall video" field ... Overall video included in the evaluation request data- "Evaluation date and time" field ... Information about the execution date and time of step S131-"Partial evaluation" field ... Partial evaluation information obtained in step S1313 ・ "Overall evaluation" field ... Overall evaluation information obtained in step S1314

After step S2311, the server 30 executes the evaluation response (S133) as in the first embodiment (FIG. 10).
After step S133, the client device 10 executes the display of the evaluation result (S114) as in the first embodiment.

According to the second embodiment, each of the plurality of elemental motions constituting the partial motion is evaluated by using the partial motion evaluation model corresponding to each of the plurality of partial motions, and the evaluation result of each elemental motion is referred to. Then, at least one of the total movement and the partial movement is evaluated. Thereby, at least one of the total operation and the partial operation can be evaluated in more detail.

(3) Modification example A modification of the present embodiment will be described.

(3-1) Modification 1
A modification 1 of the present embodiment will be described. Modification 1 is an example in which the server 30 determines the operation type.
FIG. 15 is a sequence diagram of information processing of the first modification. FIG. 16 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 15, the client device 10 executes shooting (S111) of the entire operation.
Specifically, the processor 12 displays the screen G20 (FIG. 16) on the display. The screen G20 includes the operation object B10d.

After step S111, the client device 10 executes the evaluation request (S112).
Specifically, the processor 12 transmits the evaluation request data to the server 30. The evaluation request data includes the following information.
-User identification information specified when starting the shooting application-The entire moving image stored in the storage device 11 in step S112

After step S112, the server 30 executes the operation type identification (S331).
Specifically, the storage device 31 stores an operation type specific model. In the operation type specific model, the correlation between the feature amount of the moving image and the operation type is described.
The processor 32 specifies the operation type of the entire moving image (that is, the type of the overall operation) by inputting the whole moving image included in the evaluation request data into the operation type specifying model.

After step S331, the server 30 executes the evaluation (S131) to the evaluation response (S133) in the same manner as in the first embodiment (FIG. 10) or the second embodiment.

After step S133, the client device 10 executes the display of the evaluation result (S114) in the same manner as in the present embodiment (FIG. 10).

The operation type specification (S331) may be executed after the detection of the partial moving image (S1312).
In this case, the processor 32 specifies the operation type of the entire moving image by inputting at least one of the plurality of partial moving images detected in step S1312 into the operation type specifying model.

According to the first modification, the type of operation is specified without being specified by the user. This makes it possible to reduce the burden on the user in evaluating the operation.

(3-2) Modification 2
A modification 2 of the present embodiment will be described. The second modification is an example of evaluating the movement of the traini in consideration of the movement of a person other than the traini (hereinafter referred to as “non-traini”) included in the whole moving image.

(3-2-1) Outline of Modification Example 2 An outline of Modification 2 of the present embodiment will be described. FIG. 17 is an explanatory diagram of an outline of the modified example 2.

As shown in FIG. 17, in the second modification, the non-trainer's motion (at least one of the total motion and the partial motion) and the trainee's motion (at least one of the total motion and the partial motion) included in the whole moving image are shown. Refer to it to evaluate the behavior of the trainee.
Non-traini includes, for example, at least one of the following:
・ Patients when the traini is a caregiver ・ Patients when the traini is a doctor ・ Opponents when the traini is an athlete

As shown in FIG. 17, in the second modification, the client device 10 transmits the entire moving image to the server 30. The whole moving image corresponds to a combination of traini movements (for example, whole movements including a series of partial movements) and non-traini movements (for example, whole movements including a series of partial movements).

The server 30 uses the detection model generated by machine learning to detect the partial moving image corresponding to each of the plurality of partial motions from the whole moving image.
The server 30 evaluates at least one of the total motion and the partial motion by referring to the plurality of detected partial moving images (that is, the non-traini motion and at least one combination of the traini's total motion and the partial motion). To do.

(3-2-2) Information processing Information processing of the second modification of the present embodiment will be described.

The detection model of the second modification is configured to detect a combination of a traini partial motion and a non-traini partial motion according to the type of moving image.

The storage device 31 contains an evaluation condition table relating to the correlation between the traini's posture and the non-traini's posture, an evaluation condition table regarding the correlation between the traini's facial expression and the non-traini's facial expression, and the correlation between the traini's utterance and the non-traini's utterance. At least one of the evaluation condition tables related to the relationship is stored.

In step S1312 (FIG. 11), the processor 32 uses the rule or machine learning model stored in the storage device 11 to display the image of the traini's posture, the image of the traini's facial expression, and the traini's utterance from the partial moving image. Extract at least one of the images or sounds.

The processor 32 refers to the evaluation condition table regarding the correlation between the posture of the traini and the posture of the non-traini, and the extracted images of the posture of the traini and the posture of the non-traini, and at least one of the partial motion and the total motion. Calculate the evaluation score (that is, the evaluation score for posture).
The processor 32 refers to the evaluation condition table regarding the correlation between the facial expression of the traini and the facial expression of the non-traini, and the extracted images of the facial expression of the traini and the facial expression of the non-traini, and at least one of the partial motion and the total motion. Calculate the evaluation score (that is, the evaluation score for facial expressions).
The processor 32 refers to the evaluation condition table regarding the correlation between the traini's utterance and the non-traini's utterance, and the extracted traini's utterance and the non-traini's utterance voice, and at least one of the partial operation and the whole operation. Calculate the evaluation score (that is, the evaluation score for speech).

According to the second modification, the correlation between the non-trainer's motion and the trainee's motion is considered in the evaluation of the motion. This makes it possible to evaluate the operation of the trainee under conditions close to the actual operating environment.

(3-3) Modification 3
A modified example 3 of this embodiment will be described. Modification 3 is an example of evaluating the movement of the traini in consideration of the movement of the object included in the whole moving image.

(3-3-1) Outline of Modified Example 3 An outline of Modified Example 3 of the present embodiment will be described. FIG. 18 is an explanatory diagram of an outline of the modified example 3.

As shown in FIG. 18, in the modification 3, the motion of the object (at least one of the total motion and the partial motion) and the motion of the traini (at least one of the total motion and the partial motion) included in the whole moving image are referred to. Then, evaluate the operation of the traini.
The object includes, for example, at least one of the following:
・ Nursing care products when the trainee is a caregiver (for example, a movable nursing bed)
・ Medical equipment when the trainee is a doctor (for example, surgical instruments)
・ Sports equipment when the trainer is an athlete (for example, balls, bats, rackets, golf clubs, etc.)

The server 30 uses the detection model generated by machine learning to detect the partial moving image corresponding to each of the plurality of partial motions from the whole moving image.
The server 30 evaluates at least one of the total motion and the partial motion by referring to the plurality of detected partial moving images (that is, the motion of the object and at least one combination of the traini's total motion and the partial motion). ..

(3-3-2) Information processing The information processing of the third modification of the present embodiment will be described.

The detection model of the third modification is configured to detect a combination of a traini's partial motion and an object's partial motion according to the type of moving image.

The storage device 31 stores at least one of an evaluation condition table regarding the correlation between the posture of the traini and the posture of the object, and an evaluation condition table regarding the correlation between the utterance of the traini and the utterance of the object.

In step S1312 (FIG. 11 or 16), the processor 32 uses the rule or machine learning model stored in the storage device 11 to perform a partial moving image, an image of the traini's posture, and an image or voice of the traini's utterance. At least one of them is extracted.

The processor 32 refers to the evaluation condition table regarding the correlation between the posture of the traini and the posture of the object and the extracted image of the posture of the traini and the posture of the object, and refers to at least one evaluation score of the partial motion and the total motion. (That is, the evaluation score for posture) is calculated.
The processor 32 refers to the evaluation condition table regarding the correlation between the traini's utterance and the motion sound of the object, and the extracted voice of the traini's utterance and the motion sound of the object, and at least one of the partial motion and the entire motion. Calculate the evaluation score (that is, the evaluation score for speech).

After step S1312, the server 30 executes database update (S132) to evaluation response (S133) in the same manner as in the first embodiment (FIG. 10) or the second embodiment.
After step S133, the client device 10 executes the display of the evaluation result (S114) as in the first embodiment or the second embodiment.

According to the third modification, the correlation between the motion of the object and the motion of the trainee is considered in the evaluation of the motion. This makes it possible to evaluate the operation of the trainee under conditions close to the actual operating environment.

(4) Summary of the present embodiment A summary of the present embodiment will be given.

The first aspect of this embodiment is
It is an operation evaluation method that evaluates the operation of the trainee.
Using the detection model generated by machine learning, a step (for example, step S1312) of detecting a partial moving image corresponding to each of a plurality of partial movements from a whole moving image including a series of partial movements is performed. Prepare,
A step (for example, steps S1313 to S1314) for evaluating at least one of the total motion and the partial motion is provided with reference to the plurality of detected partial moving images.
This is an operation evaluation method.

According to the first aspect, a partial moving image is detected from the whole moving image, and at least one of the whole motion and the partial motion is evaluated by using the evaluation conditions related to the partial moving image. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

The second aspect of this embodiment is
The steps to detect are
The likelihood of each partial motion is calculated using the detection model corresponding to each of the multiple partial motions.
Detect each partial video based on the likelihood of each partial motion,
This is an operation evaluation method.

According to the second aspect, the partial moving image is detected by using the detection model for each partial motion. As a result, the partial moving image can be detected appropriately.

The third aspect of this embodiment is
The evaluation step is to evaluate the overall operation by referring to the number of detected partial videos.
This is an operation evaluation method.

According to the third aspect, the evaluation is performed according to the number of partial moving images. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

The fourth aspect of this embodiment is
The step to evaluate refers to at least one of the presence / absence of the partial motion corresponding to each partial motion, the order of each partial motion, the period of each partial motion, and the start time of the partial motion in each partial motion, and the overall operation. And evaluate at least one of the partial movements,
This is an operation evaluation method.

According to the fourth aspect, the evaluation is performed according to the order and duration of the partial moving images and at least one of the start times of the partial movements. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

A fifth aspect of this embodiment is
A step (for example, step S1311) for acquiring type information regarding the type of traini operation is provided.
A step (S1311) for specifying the type of the entire moving image is provided with reference to the acquired type information.
This is an operation evaluation method.

According to the fifth aspect, the evaluation is executed according to the type of operation of the traini specified in advance. Thereby, more accurate evaluation can be realized.

The sixth aspect of this embodiment is
A step (for example, step S331) for specifying the type of traini operation is provided with reference to at least one of the whole moving image and the partial moving image.
This is an operation evaluation method.

According to the sixth aspect, the type of operation is specified without being specified by the user. This makes it possible to reduce the burden on the user in evaluating the operation.

The seventh aspect of this embodiment is
The detection step uses a detection model according to the specified type.
This is an operation evaluation method.

According to the seventh aspect, the same effect as that of the sixth aspect can be obtained.

The eighth aspect of this embodiment is
The evaluation step is to evaluate each partial video by referring to the evaluation conditions according to the specified type.
This is an operation evaluation method.

According to the eighth aspect, the partial moving image is evaluated using the evaluation conditions according to the type of motion. As a result, it is possible to realize an evaluation equivalent to that of the instructor by using a moving image of the training.

The ninth aspect of this embodiment is
The evaluation step evaluates at least one of the total movement and the partial movement by referring to the movement of a person other than the traini included in the whole video and at least one combination of the traini's total movement and the partial movement.
This is an operation evaluation method.

According to the ninth aspect, the correlation between the non-traini movement and the traini movement is considered in the evaluation of the movement. This makes it possible to evaluate the operation of the trainee under conditions close to the actual operating environment.

The tenth aspect of this embodiment is
The evaluation step evaluates at least one of the total motion and the partial motion by referring to the appearance of the object other than the traini included in the whole movie and at least one combination of the total motion and the partial motion of the traini.
This is an operation evaluation method.

According to the tenth aspect, the correlation between the motion of the object and the motion of the trainee is considered in the evaluation of the motion. This makes it possible to evaluate the operation of the trainee under conditions close to the actual operating environment.

The eleventh aspect of this embodiment is
The steps to evaluate are
Evaluate each of the multiple elemental actions that make up a partial action,
Evaluate at least one of the total motion and the partial motion by referring to the evaluation result of each element motion.
This is an operation evaluation method.

According to the eleventh aspect, each of the plurality of elemental motions constituting the partial motion is evaluated by using the partial motion evaluation model corresponding to each of the plurality of partial motions, and the evaluation result of each elemental motion is referred to. At least one of the total motion and the partial motion is evaluated. Thereby, at least one of the total operation and the partial operation can be evaluated in more detail.

The twelfth aspect of this embodiment is
An operation evaluation device (for example, server 30) that evaluates the operation of the trainee.
Using the detection model generated by machine learning, a means for detecting a partial moving image corresponding to each partial motion from the entire moving motion including a series of partial motions (for example, a processor 32 for executing step S1312). )
A means (for example, a processor 32 that executes steps S1313 to S1314) for evaluating at least one of the total operation and the partial operation with reference to the detected plurality of partial moving images is provided.
It is an operation evaluation device (for example, a server 30).

According to the twelfth aspect, the same effect as that of the first aspect can be obtained.

The thirteenth aspect of this embodiment is
It is an operation evaluation system 1 that evaluates the operation of the trainee.
A client device 10 and a server 30 are provided.
The client device 10
A means (for example, a processor 12 for executing step S113) for transmitting an entire moving image in which an entire operation including a series of partial operations is captured is provided to the server 30.
The server 30
Using the detection model generated by machine learning, a means for detecting a partial moving image corresponding to each partial motion from the entire moving motion including a series of partial motions (for example, a processor 32 for executing step S1312). )
A means (for example, a processor 32 that executes steps S1313 to S1314) for evaluating at least one of the total operation and the partial operation with reference to the detected plurality of partial moving images is provided.
This is an operation evaluation system 1.

According to the thirteenth aspect, the same effect as that of the first aspect can be obtained.

(5) Other Modification Examples Other modification examples will be described.

The storage device 11 may be connected to the client device 10 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30.
For example, when the client device 10 can execute all the steps of the above information processing, the client device 10 functions as an information processing device that operates standalone without sending a request to the server 30.

In the present embodiment, an example in which the client device 10 shoots the entire moving image is shown, but the scope of the present embodiment is not limited to this. This embodiment can be applied to any of the following cases.
The entire moving image stored in the storage device 11 before the execution of step S111 (that is, the entire moving image taken before the execution of the process of FIG. 10).
An entire moving image taken by a camera other than the camera 15 of the client device 10 (for example, an external camera capable of communicating with the client device 10 that executes step S111).

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Further, the above-described embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Moreover, the above-described embodiment and modification can be combined.

1: Operation evaluation system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Camera 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface

Claims (13)

It is an operation evaluation method that evaluates the operation of the trainee.
Using a detection model generated by machine learning, a step is provided to detect a partial moving image corresponding to each of a plurality of partial movements from a whole moving image in which a whole motion including a series of partial motions is shot.
It comprises a step of evaluating at least one of the whole motion and the partial motion with reference to the detected plurality of partial motions.
Operation evaluation method. The step to detect is
The likelihood of each partial motion is calculated using the detection model corresponding to each of the plurality of partial motions.
Detect each partial video based on the likelihood of each partial motion,
The operation evaluation method according to claim 1. The evaluation step evaluates the overall operation with reference to the number of detected partial moving images.
The operation evaluation method according to claim 1 or 2. The evaluation step refers to at least one of the presence / absence of a partial motion corresponding to each partial motion, the order of each partial motion, the period of each partial motion, and the start time of the partial motion in each partial motion. Evaluate at least one of the overall motion and the partial motion,
The operation evaluation method according to any one of claims 1 to 3. A step of acquiring type information regarding the type of operation of the trainee is provided.
A step of specifying the type of the entire moving image by referring to the acquired type information is provided.
The operation evaluation method according to claim 4. A step of identifying the type of operation of the traini with reference to at least one of the whole moving image and the partial moving image is provided.
The operation evaluation method according to claim 4. The detection step uses a detection model according to the specified type.
The operation evaluation method according to claim 5 or 6. The evaluation step evaluates each partial moving image with reference to the evaluation conditions according to the specified type.
The operation evaluation method according to any one of claims 5 to 7. The evaluation step refers to the movement of a person other than the traini included in the whole moving image, and at least one combination of the whole movement and the partial movement of the traini, and at least one of the whole movement and the partial movement. Evaluate one,
The operation evaluation method according to any one of claims 1 to 8. The evaluation step refers to the appearance of an object other than the traini included in the overall moving image, and at least one combination of the overall motion and the partial motion of the traini, and at least one of the overall motion and the partial motion. Evaluate one,
The operation evaluation method according to any one of claims 1 to 9. The evaluation step is
Each of the plurality of elemental movements constituting the partial movement is evaluated, and
With reference to the evaluation result of each element operation, at least one of the whole operation and the partial operation is evaluated.
The operation evaluation method according to any one of claims 1 to 10. It is an operation evaluation device that evaluates the operation of the trainee.
Using a detection model generated by machine learning, a means for detecting a partial moving image corresponding to each partial motion from a whole moving image including a series of partial motions is provided.
A means for evaluating at least one of the whole motion and the partial motion is provided with reference to the detected plurality of partial moving images.
Operation evaluation device. It is an operation evaluation system that evaluates the operation of the trainee.
It is equipped with a client device and a server.
The client device
A means for transmitting an entire moving image of a whole motion including a series of partial motions to the server is provided.
The server
A means for detecting a partial moving image corresponding to each partial motion from the whole moving image using a detection model generated by machine learning is provided.
A means for evaluating at least one of the whole motion and the partial motion is provided with reference to the detected plurality of partial moving images.
Operation evaluation system.

Images