JP2021026527A - Information processing device and program - Google Patents

Abstract

To provide an environment which can improve the learning effect of a trainee and the teaching efficiency of a trainer in coaching.SOLUTION: An information processing device comprises: means which extracts point cloud data for a plurality of feature points of a subject from a subject image of at least one subject; means which generates a model image of a subject model by referring to the point cloud data; means which generates a first composite image by overlapping the model image on the subject image; means which displays the first composite image on a display; means which acquires a user instruction corresponding to the user operation to the model image contained in the first composite image displayed on the display; means which changes the model image by referring to the user instruction; means which generates a second composite image by overlapping the model image after the change on the subject image; and means which displays the second composite image on the display.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device and a program.

In recent years, the needs for coaching have been increasing. For example, in the field of long-term care, there is an increasing need for trainers (for example, experienced caregivers) to coach trainees (for example, beginners) in long-term care movements in order to increase the number of caregivers.

For example, Patent Document 1 discloses a method of superimposing a moving image of a traini and a moving image of a model prepared in advance as a method of efficiently learning a sports form.

JP-A-2006-230630

In general, it is useful for the trainer to correct the trainee's movements in order to improve the learning effect.

However, as in Patent Document 1, the trainer cannot provide correction advice only by superimposing the moving image of the trainer and the moving image of the model. Therefore, the learning effect of the trainer and the teaching efficiency of the trainer in coaching are insufficient.

An object of the present invention is to provide an environment in which the learning effect of a trainer in coaching and the teaching efficiency of a trainer can be improved.

One aspect of the present invention is
A means for extracting point cloud data relating to a plurality of feature points of the subject from a subject image of at least one subject is provided.
A means for generating a model image of a subject model with reference to the point cloud data is provided.
A means for generating a first composite image by superimposing the model image on the subject image is provided.
A means for displaying the first composite image on a display is provided.
A means for acquiring a user instruction corresponding to a user operation on a model image included in the first composite image displayed on the display is provided.
A means for changing the model image with reference to the user instruction is provided.
A means for generating a second composite image by superimposing the changed model image on the subject image is provided.
A means for displaying the second composite image on the display is provided.
It is an information processing device.

According to the present invention, it is possible to provide an environment in which the learning effect of the trainer in coaching and the teaching efficiency of the trainer can be improved.

It is a block diagram which shows the structure of the information processing system of this embodiment. It is a functional block diagram of the information processing system of FIG. It is explanatory drawing of the outline of this embodiment. It is a figure which shows the data structure of the traini information database of this embodiment. It is a figure which shows the data structure of the trainer information database of this embodiment. It is a figure which shows the data structure of the synthetic image information database of this embodiment. It is a sequence diagram of the process of generating the composite image of this embodiment. It is a figure which shows the screen example displayed in the information processing of FIG. 7. It is a figure which shows the screen example displayed in the information processing of FIG. 7. It is a figure which shows the screen example displayed in the information processing of FIG. 7. It is a sequence diagram of the process of displaying the composite image of this embodiment. It is a figure which shows the screen example displayed in the information processing of FIG. It is a detailed flowchart of the first example of the change of the subject model of the modification 3. It is explanatory drawing of the information processing of FIG. It is a detailed flowchart of the 2nd example of the change of the subject model of the modification 3. It is a figure which shows the screen example of the modification 4.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. In addition, in the drawing for demonstrating the embodiment, the same components are in principle the same reference numerals, and the repeated description thereof will be omitted.

(1) Configuration of information processing system The configuration of the information processing system will be described. FIG. 1 is a block diagram showing a configuration of an information processing system of the present embodiment. FIG. 2 is a functional block diagram of the information processing system of FIG.

As shown in FIG. 1, the information processing system 1 includes a first client device 10, a second client device 20, and a server 30.
The first client device 10, the second client device 20, and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

The first client device 10 and the second client device 20 are computers (an example of an "information processing device") that transmit a request to the server 30. The first client device 10 and the second client device 20 are, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is a computer that provides the first client device 10 and the second client device 20 with a response in response to a request transmitted from the first client device 10 and the second client device 20 (an example of an "information processing device"). Is. The server 30 is, for example, a web server.

(1-1) Configuration of First Client Device The configuration of the first client device 10 will be described.

As shown in FIG. 2, the first client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS (Operating System) program ・ Application (for example, web browser) program that executes information processing

The data includes, for example, the following data.
-Database referenced in information processing-Data obtained by executing information processing (that is, the execution result of information processing)

The processor 12 is configured to realize the function of the first client device 10 by activating the program stored in the storage device 11. The processor 12 is, for example, a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

The input / output interface 13 is configured to acquire a user's instruction from an input device connected to the first client device 10 and output information to an output device connected to the first client device 10.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display (for example, a touch panel display).

The communication interface 14 is configured to control communication between the second client device 20 and the server 30.

(1-2) Configuration of Second Client Device The configuration of the second client device 20 will be described.

As shown in FIG. 2, the second client device 20 includes a storage device 21, a processor 22, an input / output interface 23, and a communication interface 24.

The storage device 21 is configured to store programs and data. The storage device 21 is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
-OS program-Program of an application (for example, a web browser) that executes information processing

The data includes, for example, the following data.
-Database referenced in information processing-Data obtained by executing information processing (that is, the execution result of information processing)

The processor 22 is configured to realize the function of the second client device 20 by activating the program stored in the storage device 21. The processor 22 is, for example, a CPU, an ASIC, an FPGA, or a combination thereof.

The input / output interface 23 is configured to acquire a user's instruction from an input device connected to the second client device 20 and output information to an output device connected to the second client device 20.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display (for example, a touch panel display).

The communication interface 24 is configured to control communication between the second client device 20 and the server 30.

(1-3) Server Configuration The configuration of the server 30 will be described.

As shown in FIG. 2, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

The program includes, for example, the following program.
・ OS program ・ Application program that executes information processing

The data includes, for example, the following data.
・ Database referenced in information processing ・ Execution result of information processing

The processor 32 is configured to realize the function of the server 30 by activating the program stored in the storage device 31. The processor 32 is, for example, a CPU, an ASIC, an FPGA, or a combination thereof.

The input / output interface 33 is configured to acquire a user's instruction from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and the first client device 10 and the second client device 20.

(2) Outline of the Embodiment The outline of the present embodiment will be described. FIG. 3 is an explanatory diagram of an outline of the present embodiment.

As shown in FIG. 3, the trainee uses the first client device 10. The trainer uses the second client device 20.

The second client device 20 has the following functions.
-A function of acquiring a subject image (still image or moving image) of at least one subject from the first client device 10-A function of extracting point group data relating to a plurality of feature points of the subject from the subject image-Point group data Function to generate a model image of the subject model by reference ・ Function to generate the first composite image by superimposing the model image on the subject image ・ Function to display the first composite image on the display ・ Displayed on the display A function to acquire user instructions corresponding to user operations on the model image included in the first composite image-A function to generate a changed model image by changing the model image with reference to the user instruction-For the subject image A function to generate a second composite image by superimposing the changed model image-A function to display the second composite image on the display of the first client device 10.

(3) Database The database of the present embodiment will be described. The following database is stored in the storage device 31.

(3-1) Traini Information Database The traini information database of the present embodiment will be described. FIG. 4 is a diagram showing a data structure of the traini information database of the present embodiment.

The traini information database of FIG. 4 stores traini information related to the traini.
The traini information database contains the following fields: Each field is associated with each other.
-"Traini ID" field- "Traini name" field- "Client ID" field

The trainee identification information is stored in the "traini ID" field. The traini identification information is information that identifies the traini.

The "Traini Name" field stores information (eg, text) about the traini's name (eg, name or nickname).

In the "client ID" field, client identification information that identifies the first client device 10 used by the trainee is stored.

(3-2) Trainer information database The trainer information database of the present embodiment will be described. FIG. 5 is a diagram showing a data structure of the trainer information database of the present embodiment.

Trainer information about the trainer is stored in the trainer information database of FIG.
The trainer information database contains the following fields: Each field is associated with each other.
-"Trainer ID" field- "Trainer name" field- "Client ID" field

Trainer identification information is stored in the "Trainer ID" field. The trainer identification information is information that identifies the trainer.

The Trainer Name field contains information (eg, text) about the trainer's name (eg, name or nickname).

In the "client ID" field, client identification information that identifies the second client device 20 used by the trainer is stored.

(3-3) Synthetic image information database The synthetic image information database of the present embodiment will be described. FIG. 6 is a diagram showing a data structure of the composite image information database of the present embodiment.

The composite image information database of FIG. 6 includes an "image ID" field, a "traini ID" field, a "trainer ID" field, and a "user comment" field.

Image identification information is stored in the "image ID" field. The image identification information is information for identifying the second composite image generated by the second client device 20.

In the "traini ID" field, traini identification information of the traini corresponding to the subject image that is the basis of the second composite image is stored.

The "Trainer ID" field stores the trainer identification information of the trainer that has given the instruction to change the model image that is the basis of the second composite image.

User comment information about the user comment is stored in the "user comment" field.

(4) Information processing The information processing of the present embodiment will be described.

(4-1) Process for generating a composite image The process for generating a composite image of the present embodiment will be described. FIG. 7 is a sequence diagram of the process of generating the composite image of the present embodiment. FIG. 8 is a diagram showing an example of a screen displayed in the information processing of FIG. 7. FIG. 9 is a diagram showing an example of a screen displayed in the information processing of FIG. 7. FIG. 10 is a diagram showing an example of a screen displayed in the information processing of FIG. 7.

In the example of FIG. 7, the trainee uses the first client device 10. The trainer uses the second client device 20.

As shown in FIG. 7, the first client device 10 executes a coaching request (S110).
Specifically, the processor 12 displays the screen P10 (FIG. 8) on the display.

The screen P10 includes an operation object B10 and field objects F10a to F10c.
The field object F10a is an object that accepts the designation of an image to be coached (hereinafter referred to as “subject image”) from the images stored in the storage device 11. The subject image includes an image of a trainee (hereinafter referred to as "subject image") when a coaching target motion (for example, a nursing care motion) to be coached is performed. The subject image is a still image or a moving image.
The field object F10b is an object that accepts the designation of trainer identification information of the trainer in charge of coaching.
The field object F10c is an object that accepts the designation of trainee identification information of the trainee (for example, the trainee using the first client device 10) to be coached.
The operation object B10 is an object that receives a user instruction for transmitting a subject image to the second client device 20 via the server 30.

The trainer specifies an arbitrary image from a plurality of images displayed on the field object F10a, specifies the trainer identification information on the field object F10b, specifies the trainer identification information on the field object F10c, and sets the operation object B10. Upon operation, the processor 12 transmits coaching request data to the server 30. The coaching request data includes the following information.
-Image specified in the field object F10a-Trainer identification information specified in the field object F10b-Trainer identification information specified in the field object F10c

After step S110, the server 30 executes the coaching notification (S130).
Specifically, the processor 32 refers to the traini information database (FIG. 4) and refers to the information in the “traini name” field associated with the traini identification information included in the coaching request data in step S110 (ie, the trainer name). Information) and the information in the "Client Identification Information" field (that is, the client identification information of the first client device 10) is specified.
The processor 32 refers to the trainer information database (FIG. 5) and refers to the information in the "trainer name" field (that is, information about the trainer name) and the "client" associated with the trainer identification information included in the coaching request data in step S110. The information in the "Identification Information" field (that is, the client identification information of the second client device 20) is specified.
The processor 32 transmits coaching notification data to the second client device 20 corresponding to the specified client identification information. The coaching notification data includes the following information.
-Subject image included in coaching request data-Information about the identified trainee name

After step S130, the second client device 20 executes the display of the subject image (S120).
Specifically, the processor 22 displays the screen P20 (FIG. 9) on the display.

The screen P20 includes display objects A20a to A20b and operation objects B20.
The display object A20a displays information about the trainee name included in the coaching notification data.
The image IMG20 is displayed on the display object A20b. The image IMG 20 is a subject image included in the coaching notification data.
The operation object B20 is an object that receives an execution instruction for generating a subject model (S121).

After step S120, the second client device 20 executes subject model generation (S121).
Specifically, the storage device 21 stores a model generation algorithm. The model generation algorithm describes rules for generating a subject model. The subject model is a model that represents the shape of the skeleton of the subject (person or object). The subject model is, for example, a model represented only by the skeleton of the subject, or a model represented by a combination of the skeleton of the subject and a portion fleshed out in the skeleton. The model generation algorithm includes a person model generation algorithm and an object model generation algorithm.
In the human model generation algorithm, rules for identifying feature points of the skeleton when the subject is a person are described.
In the object model generation algorithm, rules for identifying feature points of the skeleton when the subject is an object are described.

When the user operates the operation object B20, the processor 22 extracts the point cloud data from the image IMG 20 by using the model generation algorithm. The point cloud data is data relating to a plurality of feature points of the subject. The point cloud data includes the two-dimensional coordinates of each feature point. The two-dimensional coordinates are defined by a coordinate system (hereinafter referred to as "reference coordinate system") in the image space of the subject image.
The processor 22 generates a feature line connecting each feature point.
The processor 22 generates a model image by synthesizing a plurality of feature points, a plurality of feature lines, and two-dimensional coordinates of each feature point.

After step S121, the second client device 20 executes the generation of the first composite image (S122).
Specifically, the processor 22 generates the first composite image by superimposing the model image obtained in step S121 on the subject image included in the coaching notification data.

After step S122, the second client device 20 executes the subject model change (S123).
Specifically, the processor 22 displays the screen P21 (FIG. 9) on the display.

The constraint model is stored in the storage device 21. In the constraint model, constraints on the movement of the subject (person or object) are described. The constraint model includes a person constraint model and an object constraint model.

In the person constraint model, constraint conditions related to the movement of a person are described. In the person constraint model, for example, the movement of a node not specified by the user is described in response to the movement of the node specified by the user with respect to the change in the posture of the person and the position of the person. The constraint condition includes, for example, at least one of the following.
・ Limitation of the movable angle of the person's joints ・ Limitation of the orientation of the person's joints ・ The person's feet do not leave the ground ・ The person's center of gravity is placed above the feet

In the object constraint model, constraint conditions related to the operation of the object are described. In the object constraint model, the movement of the node not specified by the user is described in accordance with the movement of the node specified by the user with respect to the change in the shape of the object and the position of the object. The constraint condition includes, for example, at least one of the following.
・ Limit of range of motion of object ・ Limit of orientation of object ・ Bottom of object does not move away from the ground ・ Limit of inclination of object

The screen P21 includes display objects A20a and A21 and operation objects B21.
The image IMG21 is displayed on the display object A21. The image IMG 21 is a first composite image obtained in step S122 (that is, an image in which a model image is superimposed on a subject image). The image IMG 21 includes a node object FN corresponding to the feature point and a link object FL corresponding to the feature line. The node object FN is configured to be movable within the range of the constraint conditions described in the person constraint model or the object constraint model. That is, the processor 22 acquires the user instruction within the range of the constraint according to the subject.
The operation object B21 is an object that receives an execution instruction for changing the subject model.

In the first example of step S123, the trainer operates a touch panel (for example, swipe, pinch in, pinch out, and pinch out) to change the positions of all node objects FN with respect to the image IMG21 on the screen P21 (FIG. 10). Given at least one of the slides), the processor 22 acquires user instructions for changing the coordinates of the feature points corresponding to all node objects FN.
This user instruction corresponds to an instruction for changing the position of the model image.

In the second example of step S123, when the trainer gives the image IMG21 on the screen P21 (FIG. 10) a touch panel operation for changing the position of some node objects FNa, the processor 22 causes the node object FNa. Acquires a user instruction for changing the coordinates of the feature points corresponding to.
This user instruction corresponds to an instruction for changing the posture of the model image when the subject is a person.
This user instruction corresponds to an instruction for changing the shape of the model image when the subject is an object.

The processor 22 changes the two-dimensional coordinates assigned to the feature points corresponding to the node object FN according to the position of the node object FN corresponding to the acquired user instruction.
The processor 22 changes the link object FL connected to the node object FN according to the position of the node object FN corresponding to the user instruction.
When the subject is a person, at least one of the positions and orientations of the skeleton model in the image space of the subject image is changed.
When the subject is an object, at least one of the positions and shapes of the skeleton model in the image space of the subject image is changed.

As a result, a subject model (hereinafter referred to as "changed subject model") corresponding to the combination of the changed node object FN and the link object FL can be obtained.

After step S123, the second client device 20 executes the generation of the second composite image (S124).
Specifically, the processor 22 generates a second composite image by superimposing the changed subject model obtained in step S123 on the subject image included in the coaching notification data.

After step S124, the second client device 20 executes the reception of user comments (S125).
Specifically, the processor 22 displays the screen P22 (FIG. 10) on the display.

The screen P22 includes display objects A20a and A22, an operation object B22, and a field object F22.
The image IMG 22 is displayed on the display object A22. The image IMG 22 is the second composite image obtained in step S124.
The field object F22 is an object that receives a user instruction regarding a trainer's user comment. The trainer's user comments are, for example, advice from the trainer to the trainee.
The operation object B22 is an object that receives an execution instruction for transmitting the second composite image data (S126).

After step S125, the second client device 20 executes the transmission of the second composite image data (S126).
Specifically, when the trainer inputs a user comment to the field object F22 and operates the operation object B22, the processor 22 transmits the second composite image data to the server 30. The second composite image data includes the following information.
-Second composite image obtained in step S124-User comment information regarding the trainer user comment input to the field object F22.

After step S126, the server 30 executes the saving of the second composite image (S131).
Specifically, the processor 32 adds a new record to the composite image information database (FIG. 6). The following information is stored in each field of the new record.
-New image identification information is stored in the "image ID" field.
-The trainee identification information included in the coaching request data is stored in the "traini ID" field.
-The trainer identification information included in the coaching request data is stored in the "Trainer ID" field.
-The user comment information included in the second composite image data is stored in the "user comment" field.

(4-2) Processing of Displaying Composite Image The processing of displaying the composite image of the present embodiment will be described. FIG. 11 is a sequence diagram of processing for displaying the composite image of the present embodiment. FIG. 12 is a diagram showing an example of a screen displayed in the information processing of FIG.

As shown in FIG. 11, the first client device 10 executes the display request (S111).
Specifically, the processor 12 displays the screen P11 (FIG. 12) on the display.

The screen P11 includes an operation object B11 and a field object F11.
The field object F11 displays a list of file names of the second composite image that can be displayed. Image identification information is assigned to each file name included in the list. The field object F11 is an object that receives a user instruction for designating a second composite image to be displayed (hereinafter referred to as “display target image”) from the second composite image included in the list.
The operation object B11 is an object that receives a user instruction for determining the display target image specified in the field object F11.

When the user specifies a file name of an arbitrary second composite image from the second composite image included in the list displayed on the field object F11 and operates the operation object B11, the processor 12 outputs the display request data. Send to server 30. The display request data includes the following information.
-Image identification information assigned to the file name specified by the field object F11

After step S111, the server 30 executes the display response (S132).
Specifically, the processor 32 refers to the composite image information database (FIG. 6) to identify trainer identification information and user comment information associated with the image identification information included in the display request data.
The processor 32 refers to the trainer information database (FIG. 5) to identify information about the trainer name associated with the identified trainer identification information.
The processor 32 transmits the display response data to the first client device 10. The display response data includes the following information.
-Information about the specified trainer name-Second composite image to be displayed-Specified user comment information

After step S132, the first client device 10 executes the display of the second composite image (S112).
Specifically, the processor 12 displays the screen P12 (FIG. 10) on the display.

The screen P12 includes display objects A12a to A12c.

The screen P12 includes display objects A12a to A12c.
Information about the trainer name included in the display response data is displayed on the display object A12a.
The image IMG22 is displayed on the display object A12b. The image IMG 22 is a second composite image included in the display response data.
The display object A12c displays the user comment included in the user comment information included in the display response data.

According to this embodiment, the second client device 20 used by the trainer generates the second composite image. The second composite image is an image in which the subject model and the subject image according to the instruction of the trainer are superimposed. As a result, the training efficiency of the trainer can be improved.
In addition, the first client device 10 used by the traini presents the second composite image to the traini. Thereby, the learning effect of the traini can be improved.
In this way, it is possible to provide an environment in which the learning effect of the trainer in coaching and the teaching efficiency of the trainer can be improved.

Further, according to the present embodiment, when the subject is a person, a subject model in which at least one of the positions and postures of the skeleton model of the person is changed is generated. As a result, the learning effect of the trainer and the teaching efficiency of the trainer in coaching about a person can be further improved.

Further, according to the present embodiment, when the subject is an object, a subject model in which at least one of the positions and shapes of the skeleton model of the object is changed is generated. As a result, the learning effect of the trainer and the teaching efficiency of the trainer in coaching about the object can be further improved.

Further, according to the present embodiment, a user instruction for changing the subject model is received via the touch panel operation. This allows the trainer to intuitively change the subject model. As a result, the training efficiency of the trainer is further improved.

Further, according to the present embodiment, the subject model is changed within the range of the constraint condition. As a result, the learning effect of the trainer and the teaching efficiency of the trainer can be further improved.

(5) Modification Example A modification of the present embodiment will be described.

(5-1) Modification 1
Modification 1 will be described. Modification 1 is an example of generating a model image including three-dimensional information.

The storage device 21 of the first modification stores a three-dimensional coordinate estimation algorithm. In the three-dimensional coordinate estimation algorithm, rules for converting the two-dimensional coordinates of the feature points of the image into the three-dimensional coordinates are described based on the feature amount of the image.

In step S121 (FIG. 7), the processor 22 uses a model generation algorithm to extract point cloud data from the image IMG 20.
The processor 22 uses a three-dimensional coordinate estimation algorithm to convert the two-dimensional coordinates of each feature point included in the extracted point cloud data into three-dimensional coordinates. The three-dimensional coordinates are defined by the reference coordinate system.
The processor 22 generates a feature line connecting each feature point.
The processor 22 generates a model image by synthesizing a plurality of feature points, a plurality of feature lines, and two-dimensional coordinates of each feature point.

In step S123 (FIG. 7), when the trainer gives the image IMG21 on the screen P21 (FIG. 10) a touch panel operation for changing the position of some node objects FNa, the processor 22 causes the node object FNa. Acquire a user instruction for changing the three-dimensional coordinates of the feature point corresponding to.
This user instruction corresponds to an instruction for rotating the model image.

In step S123 (FIG. 7), the processor 22 changes the three-dimensional coordinates assigned to the feature points corresponding to the node object FNa according to the position of the node object FNa corresponding to the user instruction obtained in step S123. ..
The processor 22 changes the link object FL connected to the node object FN according to the position of the node object FNa corresponding to the user instruction.
As a result, the angle of the skeleton model in the image space of the subject image is changed.

According to the first modification, a model image in the three-dimensional space is generated according to the instruction of the trainer. This makes it possible to provide coaching in a three-dimensional space (for example, coaching related to rotation). As a result, it is possible to provide an environment in which the learning effect of the trainer in coaching and the teaching efficiency of the trainer can be further improved.

(5-2) Modification 2
Modification 2 will be described. Modification 2 is an example in which the subject image includes a plurality of subjects.

In the first example of the second modification, in step S121 (FIG. 7), the processor 22 generates a model image corresponding to all the subjects included in the subject image.

In step S123 (FIG. 7), the processor 22 acquires user instructions for a model image corresponding to at least one subject (eg, a subject arbitrarily designated by the user).

In the second example of the second modification, in step S121 (FIG. 7), the processor 22 generates a model image corresponding to at least one subject arbitrarily designated by the user among the plurality of subjects included in the subject image. ..

According to the second modification, the same effect as that of the present embodiment can be obtained even when the subject image includes a plurality of subject images.

(5-3) Modification 3
Modification 3 will be described. Modification 3 is an example in which the subject image is a moving image (that is, an image composed of a plurality of subject image frames).

In the third modification, after step S120 (FIG. 7), the second client device 20 executes subject model generation (S121).
Specifically, the processor 22 generates a frame of a model image for each subject image frame (hereinafter referred to as "model image frame"). As a result, a model moving image composed of a plurality of model image frames can be obtained.

After step S121, the second client device 20 executes the generation of the first composite image (S122).
Specifically, the processor 22 generates the first composite moving image by superimposing the model moving image obtained in step S121 on the subject image included in the coaching notification data.

(5-3-1) First Example of Changing the Subject Model The first example of changing the subject model of the modified example 3 will be described. The first example is an example of accepting a change instruction to the subject model while keeping the subject image stationary.
FIG. 13 is a detailed flowchart of the first example of changing the subject model of the modified example 3. FIG. 14 is an explanatory diagram of information processing in FIG.

As shown in FIG. 13, the second client device 20 executes the acquisition of the user instruction (S2230) for each model image frame.
Specifically, when the user specifies the time (for example, time T1) assigned to the model image frame to be changed, the processor 22 displays the model image frame (FLAME1) corresponding to the time T1 as the display object A21. It is displayed in (Fig. 9).
When the user gives the model image frame (FLAME1) a touch panel operation for changing the positions of all the node objects FN or some of the node objects FNa, the processor 22 causes the node object corresponding to the user's operation. Acquire a user instruction for changing the coordinates (two-dimensional coordinates or three-dimensional coordinates) of the feature points associated with.

After step S2230, the second client device 20 executes the generation of the subject model (S2231) in response to the user instruction.
Specifically, the processor 22 sets the coordinates assigned to the feature points corresponding to the node object FN of the model image frame (FLAME1) according to the position of the node object FN corresponding to the user instruction obtained in step S2230. change.
The processor 22 changes the link object FL connected to the node object FN according to the position of the node object FN corresponding to the user instruction.
As a result, a moving image of the subject model (hereinafter referred to as "model moving image") can be obtained. The model moving image is composed of a model image frame (hereinafter referred to as "changed model image frame") corresponding to the combination of the changed node object FN and the link object FL.

After S2231, the second client device 20 executes subject model complementation (S2322).
Specifically, the storage device 21 stores the interpolation algorithm. The interpolation algorithm describes the rules for the movement of people and objects.
The processor 22 inserts the interpolation frame between the changed model image frames by applying the interpolation algorithm to the subject model obtained in step S2231. The interpolation frame is a frame that interpolates the movement of the subject corresponding to the plurality of modified model image frames.
As a result, a model moving image (hereinafter referred to as "changed model moving image") composed of a combination of the changed model image frame and the interpolation frame can be obtained.

After step S2232, the second client device 20 executes the generation of the second composite image (S124) of FIG.
Specifically, the processor 22 generates a second composite moving image by superimposing the changed model moving image obtained in step S2232 on the subject image included in the coaching notification data.

(5-3-2) Second Example of Changing the Subject Model A second example of changing the subject model of the modified example 3 will be described. The second example is an example of accepting a change instruction to the subject model while reproducing the subject image.
FIG. 15 is a detailed flowchart of the second example of changing the subject model of the modified example 3.

As shown in FIG. 15, the second client device 20 executes the start of reproduction of the subject image (S3230).
Specifically, when the user gives a touch panel operation (for example, tap) to a predetermined area of the display object A20b (FIG. 9), the processor 22 starts reproducing the subject image.

After step S3230, the second client device 20 executes the acquisition of the user instruction (S2230) for each model image frame in the same manner as in the first example (FIG. 13) of the modification example 3. As a result, a user instruction for the model image frame that was displayed when the user operation is given can be obtained.

After step S2230, the second client device 20 executes the stop of reproduction of the subject image (S3231).
Specifically, when the user operates the operation object B21 (FIG. 10), the processor 22 stops the reproduction of the subject image.

After step S3231, the second client device 20 generates a subject model according to a user instruction (S2231), interpolates motion (S2232), and a second, as in the first example (FIG. 13) of the change example 3. 2 Generate a composite image (S124).

According to the third modification, the subject model corresponding to the moving image is changed. This makes it possible to provide an environment in which the learning effect of the trainer in the coaching of movement and the teaching efficiency of the trainer can be improved.

Further, according to the modification 3, an interpolation frame for interpolating between the changed model image frames is generated. This makes it possible to provide a model moving image with smooth movement.

According to the second example of the modification 3, the subject model corresponding to the moving image is changed while reproducing the moving image. As a result, it is possible to provide an environment in which the training efficiency of the trainer in motion coaching can be improved.

Although the modification 3 shows an example of acquiring user instructions for two model image frames (FLAME1 to FLAME2), the number of model image frames corresponding to the user instructions is not limited to this. Modification 3 is also applicable when user instructions are given for three or more model image frames.

In the second example of the third modification, an example in which the acquisition of the user instruction (S2230) for each model image frame is executed after the start of the reproduction of the subject image (S3230) is shown, but the execution order of steps S3230 and S2230 is Not limited to this. In the second modification, the reproduction of the subject image may be started in step S2230 after acquiring the user instruction for a certain model image frame. In this case, the second client device 20 acquires a user instruction for a model image frame other than the model image frame after step S3230.

(5-4) Modification 4
Modification 4 will be described. Modification 4 is an example of accepting a traini user comment. FIG. 16 is a diagram showing a screen example of the modified example 4.

The processor 12 of the modification 4 displays the screen P41 (FIG. 16) on the display in step S111 (FIG. 7).

The screen P41 includes display objects A41a to A41b, a field object F41, and an operation object B41.
The field object F41 displays the user comment included in the second composite image data. The field object F41 is an object that receives a user instruction regarding a traini's user comment. The traini's user comments are, for example, a traini's question to the trainer and a trainer's response to the trainer's user comment.
The operation object B41 is an object that receives a user instruction for transmitting the trainee user comment given to the field object F41 to the second client device 20.

When the trainee inputs a user comment to the field object F41 and operates the operation object B41, the processor 12 transmits the user comment input to the field object F41 to the server 30.

The server 30 transmits the user comment transmitted from the first client device 10 to the second client device 20.

The processor 22 displays the user comment transmitted from the server 30 on the display (for example, the field object F22 on the screen P22 (FIG. 10)).

According to the modification 4, not only the trainer's user comment but also the trainer's user comment is acquired. This makes it possible to provide an environment in which the learning effect of the trainer in coaching and the teaching efficiency of the trainer can be further improved.

(6) Summary of the present embodiment A summary of the present embodiment will be given.

The first aspect of this embodiment is
A means (for example, a step) for extracting point cloud data relating to a plurality of feature points of the subject from a subject image of at least one subject is provided.
With a means to generate a model image of the subject model by referring to the point cloud data,
A means for generating a first composite image by superimposing a model image on a subject image is provided.
Provided with a means for displaying the first composite image on the display
A means for acquiring a user instruction corresponding to a user operation on a model image included in the first composite image displayed on the display is provided.
With a means to change the model image with reference to the user's instructions,
A means for generating a second composite image by superimposing the changed model image on the subject image is provided.
A means for displaying the second composite image on the display is provided.
An information processing device (for example, a second client device 20 or a server 30).

According to the first aspect, the second composite image is generated by synthesizing the subject image and the model image changed according to the instruction of the trainer, and the second composite image is displayed on the trainee. This makes it possible to provide an environment in which the learning effect of the trainer in coaching and the teaching efficiency of the trainer can be improved.

The second aspect of this embodiment is
The subject is a person
The means for acquiring the user instruction is to acquire the user instruction for changing the coordinates of each feature point.
The means of generating a model image is to generate a model image of a human skeleton model and
The means for changing is to change at least one of the positions and orientations of the skeleton model in the image space of the subject image with reference to the user's instructions.
An information processing device (for example, a second client device 20 or a server 30).

According to the second aspect, a subject model in which at least one of the positions and postures of the skeleton model of the person is changed is generated. As a result, the learning effect of the trainer and the teaching efficiency of the trainer in coaching about a person can be further improved.

The third aspect of this embodiment is
The subject is an object,
The means for acquiring the user instruction is to acquire the user instruction for changing the coordinates of each feature point.
The means for generating a model image is to generate a model image of the skeleton model of the object and
The means for changing is to change at least one of the positions and shapes of the skeleton model in the image space of the subject image with reference to the user's instructions.
An information processing device (for example, a second client device 20 or a server 30).

According to the third aspect, a subject model in which at least one of the positions and shapes of the skeleton model of the object is changed is generated. As a result, the learning effect of the trainer and the teaching efficiency of the trainer in coaching about the object can be further improved.

The fourth aspect of this embodiment is
The means for generating the model image is to estimate the 3D coordinates of each feature point and
The means for acquiring the user instruction is to acquire the user instruction for changing the three-dimensional coordinates of each feature point.
The means for changing is to perform at least one of rotation, position change, and shape change of the subject model in the image space of the subject image with reference to the user instruction.
An information processing device (for example, a second client device 20 or a server 30).

According to the fourth aspect, a model image in the three-dimensional space is generated according to the instruction of the trainer. This makes it possible to provide coaching in a three-dimensional space (for example, coaching related to rotation). As a result, it is possible to provide an environment in which the learning effect of the trainer in coaching and the teaching efficiency of the trainer can be improved.

A fifth aspect of this embodiment is
The means for acquiring the user instruction is to acquire the user instruction in response to the touch panel operation on the model image included in the first composite image or the second composite image displayed on the touch panel display.
An information processing device (for example, a second client device 20 or a server 30).

According to the fifth aspect, the user instruction for changing the subject model is received via the touch panel operation. This allows the trainer to intuitively change the subject model. As a result, the training efficiency of the trainer is further improved.

The sixth aspect of this embodiment is
The means for acquiring the user instruction is to acquire the user instruction within the range of the constraint according to the subject.
An information processing device (for example, a second client device 20 or a server 30).

According to the sixth aspect, the subject model is changed within the range of the constraint condition. As a result, the learning effect of the trainer and the teaching efficiency of the trainer can be further improved.

The seventh aspect of this embodiment is
When the means for acquiring the user instruction includes a plurality of subjects, the means for acquiring the user instruction acquires the user instruction for the model image corresponding to the subject specified by the user.
An information processing device (for example, a second client device 20 or a server 30).

According to the seventh aspect, even when the subject image includes a plurality of subject images, the learning effect of the trainer and the teaching efficiency of the trainer in coaching about a person can be further improved.

The eighth aspect of this embodiment is
The subject image is a moving image,
The means for generating the first composite image is to generate the first composite moving image by superimposing a plurality of model image frames on a plurality of subject image frames.
The means for generating the second composite image is to generate the second composite moving image by superimposing the model image frame after each change on each subject image frame.
An information processing device (for example, a second client device 20 or a server 30).

According to the eighth aspect, the subject model corresponding to the moving image is changed. This makes it possible to provide an environment in which the learning effect of the trainer in the coaching of movement and the teaching efficiency of the trainer can be improved.

The ninth aspect of this embodiment is
The means for generating the second composite image interpolates the movement of the subject model corresponding to the plurality of modified model image frames.
An information processing device (for example, a second client device 20 or a server 30).

According to the ninth aspect, an interpolation frame for interpolating between the changed model image frames is generated. This makes it possible to provide a model moving image with smooth movement.

The tenth aspect of this embodiment is
The means for acquiring the user instruction is to acquire the user instruction corresponding to the user operation during the reproduction of the first composite moving image.
An information processing device (for example, a second client device 20 or a server 30).

According to the tenth aspect, the subject model corresponding to the moving image is changed while reproducing the moving image. As a result, it is possible to provide an environment in which the training efficiency of the trainer in motion coaching can be improved.

The eleventh aspect of this embodiment is
The means for acquiring the user instruction is to acquire the user instruction for the model image frame displayed when the user operation is given among the plurality of model image frames included in the first composite moving image.
The means for changing the model image is to change the subject model image of the model image frame corresponding to the acquired user instruction.
An information processing device (for example, a second client device 20 or a server 30).

According to the eleventh aspect, the subject model corresponding to the moving image is changed while reproducing the moving image. As a result, it is possible to provide an environment in which the training efficiency of the trainer in motion coaching can be improved.

The twelfth aspect of this embodiment is
This is a program for causing a computer (for example, the second client device 20 or the server 30) to function as any of the above means.

(7) Other Modification Examples Other modification examples will be described.

The storage device 11 may be connected to the first client device 10 via the network NW. The storage device 21 may be connected to the second client device 20 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

The server 30 can be omitted. In this case, the first client device 10 and the second client device 20 can communicate without going through the server 30.

The coaching target movement is not limited to the nursing care movement. The coaching target operation includes the following operations.
・ Sports movements ・ Cooking movements ・ Face-to-face sales movements ・ Announcement movements ・ Cosmetologist or barber movements ・ Other movements of occupational movements

In the present embodiment, an example of presenting the second composite image (that is, an image in which the subject image and the changed subject model image are superimposed and displayed) is presented to the trainee, but the second composite image of the present embodiment is this. Not limited to. The present embodiment is also applicable to the case where the subject image, the subject model image before the change, and the second composite image on which the changed subject model image is superimposed and displayed are presented to the trainee.

In the present embodiment, an example of displaying the second composite image in response to the trainee's request is shown, but the present embodiment is not limited to this. This embodiment can also be applied to the case where the second composite image is displayed in response to the trainer's request (for example, when the second client device 20 executes step S111).

In the present embodiment, an example in which the subject model is a combination of the node object FN and the link object FL is shown, but the present embodiment is not limited to this. This embodiment can also be applied to a subject model that does not include the node object FN.

In the present embodiment, an example in which the second client device 20 executes steps S121 to S125 (FIG. 7) is shown, but the present embodiment is not limited to this. This embodiment is also applicable when the server 30 executes steps S121 to S125. In this case, steps S130 and S126 can be omitted.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Further, the above-described embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Moreover, the above-described embodiment and modification can be combined.

1: Information processing system 10: First client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 20: Second client device 21: Storage device 22: Processor 23: Input / output interface 24: Communication interface 30 : Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface

Claims (12)

A means for extracting point cloud data relating to a plurality of feature points of the subject from a subject image of at least one subject is provided.
A means for generating a model image of a subject model with reference to the point cloud data is provided.
A means for generating a first composite image by superimposing the model image on the subject image is provided.
A means for displaying the first composite image on a display is provided.
A means for acquiring a user instruction corresponding to a user operation on a model image included in the first composite image displayed on the display is provided.
A means for changing the model image with reference to the user instruction is provided.
A means for generating a second composite image by superimposing the changed model image on the subject image is provided.
A means for displaying the second composite image on the display is provided.
Information processing device. The subject is a person
The means for acquiring the user instruction acquires a user instruction for changing the coordinates of each feature point, and obtains the user instruction.
The means for generating the model image generates a model image of the skeleton model of the person.
The changing means changes at least one of the positions and orientations of the skeleton model in the image space of the subject image with reference to the user instruction.
The information processing device according to claim 1. The subject is an object
The means for acquiring the user instruction acquires a user instruction for changing the coordinates of each feature point, and obtains the user instruction.
The means for generating the model image generates a model image of the skeleton model of the object.
The changing means performs at least one change in the position and shape of the skeleton model in the image space of the subject image with reference to the user instruction.
The information processing device according to claim 1 or 2. The means for generating the model image estimates the three-dimensional coordinates of each feature point and
The means for acquiring the user instruction acquires a user instruction for changing the three-dimensional coordinates of each feature point, and obtains the user instruction.
The changing means performs at least one of rotation, position change, and shape change of the subject model in the image space of the subject image with reference to the user instruction.
The information processing device according to any one of claims 1 to 3. The means for acquiring the user instruction acquires the user instruction in response to a touch panel operation on the model image included in the first composite image or the second composite image displayed on the touch panel display.
The information processing device according to any one of claims 1 to 4. The means for acquiring the user instruction acquires the user instruction within the range of the constraint according to the subject.
The information processing device according to any one of claims 1 to 5. When the subject image includes a plurality of subjects, the means for acquiring the user instruction acquires the user instruction for the model image corresponding to the subject specified by the user.
The information processing device according to any one of claims 1 to 6. The subject image is a moving image and is
The means for generating the first composite image is to generate a first composite moving image by superimposing a plurality of model image frames on a plurality of subject image frames.
The means for generating the second composite image generates a second composite moving image by superimposing each changed model image frame on each subject image frame.
The information processing device according to any one of claims 1 to 7. The means for generating the second composite image interpolates the movement of the subject model corresponding to the plurality of modified model image frames.
The information processing device according to claim 8. The means for acquiring the user instruction acquires the user instruction corresponding to the user operation during the reproduction of the first composite moving image.
The information processing device according to claim 8 or 9. The means for acquiring the user instruction acquires a user instruction for the model image frame displayed when the user operation is given among the plurality of model image frames included in the first composite moving image.
The means for changing the model image is to change the subject model image of the model image frame corresponding to the acquired user instruction.
The information processing device according to claim 8. A program for causing a computer to function as each means according to any one of claims 1 to 11.