WO2022196059A1

WO2022196059A1 - Information processing device, information processing method, and program

Info

Publication number: WO2022196059A1
Application number: PCT/JP2022/000897
Authority: WO
Inventors: 嘉寧呉; 順横野; 嘉昭岩井; 明香渡辺; 夏子尾崎
Original assignee: ソニーグループ株式会社
Priority date: 2021-03-19
Filing date: 2022-01-13
Publication date: 2022-09-22

Abstract

[Problem] To provide a novel and improved information processing device, information processing method, and program capable of providing personalized feedback according to physique feature information about a user. [Solution] Provided is an information processing device comprising: a posture information acquisition unit that acquires posture information indicating the posture of a user; a calculation unit that calculates physique feature information about the user through a comparison between the posture information about the user and predetermined posture information; a correction unit that corrects posture information prepared as reference movement information according to the physique feature information about the user; and an output unit that outputs guidance information for guiding movements by the user on the basis of the reference movement information corrected by the correction unit.

Description

Information processing device, information processing method and program

The present disclosure relates to an information processing device, an information processing method, and a program.

In recent years, technology has been developed that acquires the user's posture information during exercise using a sensor that acquires the user's posture information, and provides feedback to improve the user's posture according to the posture information.

For example, Patent Literature 1 discloses a technique of evaluating the exercise posture based on the height of the user's leg and the time the user raises the leg, and feeding back the evaluation result to the user in one-leg standing exercise. Further, it is disclosed that personalized information such as height and weight manually input by the user is used when evaluating exercise posture.

JP 2017-38959

However, the technique described in Patent Literature 1 does not recognize physique feature information due to the physique of the user, such as the length of each part of the user and the presence or absence of limbs, and uses the physique feature information that differs for each user as an exercise posture. Not used for evaluation.

Therefore, the present disclosure proposes a new and improved information processing method, information processing device, and program capable of providing personalized feedback according to the user's physique feature information.

According to the present disclosure, a posture information acquisition unit that acquires posture information indicating a posture of a user, and a calculation unit that calculates physique characteristic information of the user by comparing the posture information of the user and predetermined posture information. a correction unit that corrects posture information prepared as reference motion information according to the physique feature information of the user; and guidance information that guides the user's motion based on the reference motion information corrected by the correction unit. and an output unit that outputs.

In addition, according to the present disclosure, obtaining posture information indicating a posture of a user, calculating physical characteristic information of the user by comparing the posture information of the user and predetermined posture information, and referring to correcting posture information prepared as motion information according to the user's physique feature information; and outputting guidance information for guiding the user's motion based on the corrected reference motion information. A computer implemented method for processing information is provided.

Further, according to the present disclosure, a computer has a posture information acquisition function for acquiring posture information indicating a posture of a user; a calculation function for calculating, a correction function for correcting posture information prepared as reference movement information according to the physique characteristic information of the user, and a movement of the user based on the reference movement information corrected by the correction function. A program is provided for realizing an output function of outputting guidance information for guidance.

1 is an explanatory diagram showing an information processing system according to an embodiment of the present disclosure; FIG. FIG. 2 is an explanatory diagram for explaining a functional configuration example of the information processing terminal 10 according to the present disclosure; FIG. 2 is an explanatory diagram for explaining a functional configuration example of a server 20 according to the present disclosure; FIG. FIG. 4 is an explanatory diagram for explaining an example of skeleton data, which is user posture information; FIG. 4 is an explanatory diagram for explaining an example of object detection according to the present disclosure; FIG. 10 is an explanatory diagram for explaining an example of a calculation method of user's physique characteristic information; FIG. 11 is an explanatory diagram for explaining an example of correction of reference motion information; FIG. 10 is an explanatory diagram for explaining Feedback Example 1 according to the present disclosure; FIG. 11 is an explanatory diagram for explaining a feedback example 2 according to the present disclosure; FIG. 11 is an explanatory diagram for explaining a feedback example 3 according to the present disclosure; FIG. 11 is an explanatory diagram for explaining a feedback example 4 according to the present disclosure; FIG. 11 is an explanatory diagram for explaining a feedback example 5 according to the present disclosure; FIG. 4 is an explanatory diagram for explaining an operation processing example related to feedback of the information processing terminal 10 according to the present disclosure; FIG. 10 is an explanatory diagram for explaining an example of operation processing of the information processing terminal 10 when acquiring another user's operation as reference operation information; 2 is a block diagram showing the hardware configuration of the information processing terminal 10; FIG.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

In addition, the "Mode for Carrying Out the Invention" will be explained according to the order of items shown below.
1. Overview of information processing system 2 . Example of functional configuration 2-1. Functional Configuration Example of Information Processing Terminal 2-2. Functional configuration example of server 3 . Details 3-1. Posture Estimation 3-2. Object detection 3-3. Calculation of physique feature information 3-4. Correction of reference motion information 3-5. Feedback example 4 . Operation example 4-1. First operation example of information processing terminal 4-2. Second operation example of information processing terminal 5 . Example of action and effect 6. Hardware configuration7. supplement

<<1. Information processing system overview >>
For the posture information of the user, skeleton data expressed by a skeleton structure indicating the structure of the body, for example, is used in order to visualize information on the movement of a moving body such as a human being or an animal. Skeleton data includes information on parts. Note that the parts in the skeleton structure correspond to, for example, terminal parts and joint parts of the body. Skeleton data may also include bones, which are line segments connecting parts. The bones in the skeleton structure can correspond to, for example, human bones, but the positions and numbers of bones do not necessarily match the actual human skeleton.

The position and posture of each part in the skeleton data can be obtained by sensors that detect the user's movements. For example, technology that detects the position and posture of each part of the body based on time-series data of image data acquired by an imaging sensor, and time-series data acquired by a motion sensor attached to a part of the body. Techniques exist to obtain the position information of the motion sensor based on .

In addition, the uses of skeleton data are diverse. For example, time-series data of skeleton data is used for form improvement in sports, and for applications such as VR (Virtual Reality) or AR (Augmented Reality). In addition, time-series data of skeleton data is used to generate an avatar image imitating a user's movement, and the avatar image is distributed.

In the following, as an embodiment of the present disclosure, information for calculating user's physique feature information from skeleton data calculated from time-series data of movement of the user's whole body, and correcting reference motion information according to the physique feature information A configuration example of the processing system will be described. Note that although humans will be mainly described below as an example of a moving object, the embodiments of the present disclosure are similarly applicable to other moving objects such as animals and robots.

FIG. 1 is an explanatory diagram showing an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, an information processing system according to an embodiment of the present disclosure has an information processing terminal 10 and a server 20. As shown in FIG.

The information processing terminal 10 and the server 20 are connected via the network 1. A network 1 is a wired or wireless transmission path for information transmitted from devices connected to the network 1 . For example, the network 1 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), and the like. The network 1 may also include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).

(Information processing terminal 10)
The information processing terminal 10 is an example of an information processing device. The information processing terminal 10 acquires posture information indicating the posture of the user U, and calculates physical characteristic information of the user U by comparing the posture information with predetermined posture information.

Further, the information processing terminal 10 corrects the posture information prepared as the reference motion information by the server 20 according to the physique characteristic information of the user U, and guides the motion of the user U based on the corrected reference motion information. Output guidance information.

Although FIG. 1 shows a smartphone as the information processing terminal 10, the information processing terminal 10 may be other information processing devices such as a notebook PC (Personal Computer) and a desktop PC.

(Server 20)
The server 20 holds standard skeleton data containing predetermined skeleton information. In addition, the server 20 holds time-series data of posture information prepared as at least one piece of reference motion information.

For example, the server 20 transmits the above-described standard skeleton data or reference motion information to the information processing terminal 10 in response to a request signal received from the information processing terminal 10 .

The outline of the information processing system according to the present disclosure has been described above. Subsequently, functional configuration examples of the information processing terminal 10 and the server 20 according to the present disclosure will be described.

<<2. Functional configuration example >>
<2-1. Example of functional configuration of information processing terminal>
FIG. 2 is an explanatory diagram for explaining a functional configuration example of the information processing terminal 10 according to the present disclosure. As shown in FIG. 2 , the information processing terminal 10 includes a data acquisition section 110 , an operation display section 120 , a communication section 130 , a storage section 140 and a control section 150 .

(Data acquisition unit 110)
The data acquisition unit 110 acquires motion information of a user and an object. For example, the data acquisition unit 110 may be an imaging sensor, a ToF (Time of Flight) sensor, or an antenna that transmits and receives radio waves. In this specification, an imaging sensor will be described as a main example of the data acquisition unit 110 .

(Operation display unit 120)
The operation display unit 120 has a function as a display unit that displays guidance information that guides user's actions under the control of the display control unit 171 . Further, the operation display unit 120 has a function as an operation unit for the user to operate and input physical information and health information, which will be described later.

The operation display unit 120 has a function as an operation unit that allows the user to operate and input sports events, points of interest, and the like. For example, the user inputs or selects "baseball" and "swing" on the operation display unit 120 . In this case, the information processing terminal 10 according to the present disclosure may output guidance information that guides the user's motion based on posture information when the user swings and reference motion information prepared as a swing model. .

The function of the display unit is realized by, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, or an OLED (Organic Light Emitting Diode) device.

Also, the function as an operation unit is realized by, for example, a touch panel, keyboard, or mouse.

In FIG. 1, the information processing terminal 10 has a configuration in which the functions of the display unit and the operation unit are integrated, but may have a configuration in which the functions of the display unit and the operation unit are separated.

(Communication unit 130)
The communication unit 130 communicates various information with the server 20 via the network 1 . For example, the communication unit 130 transmits to the server 20 a request signal requesting standard skeleton data and reference motion information. The communication unit 130 also receives standard skeleton data and reference motion information transmitted from the server 20 in response to a request signal.

(storage unit 140)
The storage unit 140 holds software and various data. For example, the storage unit 140 holds physique feature information of the user.

(control unit 150)
The control unit 150 controls overall operations of the information processing terminal 10 . As shown in FIG. 2, the control unit 150 includes a posture estimation unit 151, an object detection unit 155, a physique feature information calculation unit 159, a conversion unit 163, an action comparison unit 167, and a display control unit 171. .

The posture estimation unit 151 estimates part information indicating the position and posture of each part of the user based on the time-series data acquired by the data acquisition unit 110 .

Then, posture estimation section 151 generates skeleton data including position information and posture information of each part in the skeleton structure based on the part information. Details of the skeleton data will be described later.

The object detection unit 155 detects object information and object motion information based on the time-series data acquired by the data acquisition unit 110 .

The physique feature information calculation unit 159 calculates physique feature information of the user by comparing the user's skeleton data generated by the posture estimation unit 151 and the standard skeleton data held by the standard skeleton data storage unit 231 . A specific example of the method for calculating physique characteristic information will be described later.

The conversion unit 163 is an example of a correction unit, and corrects the reference motion information acquired from the server 20 according to the user's physique feature information. In the following description, the reference motion information corrected according to the physique feature information of the user may be expressed as post-correction reference motion information.

The conversion unit 163 may also convert the skeleton features of each part of the user's skeleton data into the skeleton features of each part of the standard skeleton data. Note that the skeletal features in this specification include the length and thickness of bones.

Further, the conversion unit 163 may convert the skeletal feature of each part of the plurality of skeleton data included in the post-correction reference motion information to the skeletal feature of each part of the skeleton data of the user.

The motion comparison unit 167 compares the user motion information, which is time-series data of user skeleton data, with the reference motion information converted by the conversion unit 163, and outputs the comparison result to the conversion unit 163 or the display control unit 171. Output.

For example, the motion comparison unit 167 compares one frame of the user motion information with one frame of the reference motion information converted by the conversion unit 163, and outputs the comparison result to the conversion unit 163 or the display control unit 171. may

The display control unit 171 is an example of an output unit, and causes the operation display unit 120 to display guidance information that guides the user's actions based on the reference action information corrected by the conversion unit 163 . A specific example of the guidance information will be described later.

An example of the functional configuration of the information processing terminal 10 has been described above. Next, a functional configuration example of the server 20 will be described with reference to FIG.

<2-2. Server functional configuration example>
FIG. 3 is an explanatory diagram for explaining a functional configuration example of the server 20 according to the present disclosure. As shown in FIG. 3, the server 20 includes a communication section 210, a control section 220, and a storage section 230.

(Communication unit 210)
The communication unit 210 communicates various information with the information processing terminal 10 via the network 1 . For example, the communication unit 210 receives from the information processing terminal 10 a request signal requesting standard skeleton data or reference motion information. Further, under the control of the control unit 220 , the communication unit 210 transmits standard skeleton data and reference motion information to the information processing terminal 10 in response to a request signal received from the information processing terminal 10 .

(control unit 220)
The control unit 220 controls overall operations of the server 20 . For example, the control unit 220 causes the communication unit 210 to transmit various information held in the storage unit 230 in response to a request signal received from the information processing terminal 10 .

(storage unit 230)
Storage unit 230 holds software and various data. As shown in FIG. 3 , the storage section 230 includes a standard skeleton data storage section 231 and a reference motion storage section 235 .

The standard skeleton data storage unit 231 holds standard skeleton data having predetermined skeletal features for each part.

Although the standard skeleton data is an example of predetermined posture information, the predetermined posture information according to the present disclosure is not limited to such an example. For example, the predetermined posture information according to the present disclosure may be part information having predetermined skeletal features.

The reference motion storage unit 235 holds reference motion information as time-series data of skeleton data containing certain motion information. Details of the reference motion information will be described later.

The functional configuration example according to the present disclosure has been described above. Subsequently, details of the processing according to the present disclosure will be sequentially described with reference to FIGS. 4 to 13. FIG.

<<3. Details＞＞
<3-1. Posture Estimation>
FIG. 4 is an explanatory diagram for explaining an example of skeleton data including user posture information. Posture estimation section 131 acquires skeleton data US including position information and posture information of each part in the skeleton structure, for example, based on the time-series data acquired by data acquisition section 110 .

For example, the posture estimation unit 131 may generate skeleton data US of the user U using machine learning technology such as DNN (Deep Neural Network). More specifically, the posture estimating unit 131 uses an estimator obtained by a machine learning technique using, for example, a set of image data obtained by photographing a person and skeleton data as teacher data, to determine the position of the user U. to generate skeleton data US.

The skeleton data US may include bone information (position information, posture information, skeletal feature information, etc.) in addition to the information on the parts.

Although FIG. 4 shows the skeleton data of the whole body of the user U, the posture estimation unit 131 may generate the skeleton data of only parts as necessary.

<3-2. Object detection>
FIG. 5 is an explanatory diagram for explaining an example of object detection according to the present disclosure. The object detection unit 155 detects object information and object motion information of various objects o based on the time-series data acquired by the data acquisition unit 110 . Note that the object information includes the type of object (ball, racket, etc.).

For example, as shown in FIG. 5, the object detection unit 155 recognizes the wheelchair o1 on which the user U rides, the racket o2 and the ball o3 held by the user based on the time-series data acquired by the data acquisition unit 110, and further , to detect the motion direction and size of each object.

Also, the object detection unit 155 may detect the relationship between each of the objects o1 to o3 and the user U. For example, when the user U is in a wheelchair, the object detection unit 155 may recognize the wheelchair associated with the movement of the user as part of the user's body. In this case, the posture estimation unit 131 does not need to generate skeleton data of the lower limbs of the user U, for example.

Further, when the user uses the operation display unit 120 to input a sport event, the object detection unit 155 may detect an object corresponding to the input sport event. For example, when the user inputs "baseball" using the operation display unit 120, the object detection unit 155 may detect an object related to baseball (eg, a batting tool, a ball, etc.).

<3-3. Calculation of physique characteristic information>
FIG. 6 is an explanatory diagram for explaining an example of a method for calculating user's physique characteristic information. Physique feature information calculation section 159 compares the posture information of each part of the user estimated by posture estimation section 151 with predetermined posture information held by standard skeleton data storage section 231 to calculate the physique feature information of the user. calculate.

For example, the physique feature information calculation unit 159 calculates the difference between each of various skeletal feature information included in the user's skeleton data US and each of the various skeletal feature information included in the standard skeleton data RS as the user's physique feature information. .

More specifically, physique characteristic information calculation section 159 calculates the difference D between the length of one skeleton US1 of the right arm included in the skeleton data US of the user and the one skeleton RS1 of the right arm included in the standard skeleton data RS. It may be calculated as physique feature information.

Further, the physique feature information calculation unit 159 calculates a difference between each piece of skeletal feature information of a plurality of parts included in the skeleton data US of the user and each piece of skeletal feature information of a plurality of parts included in the standard skeleton data RS, The physique characteristic information of the user may be calculated based on each calculated difference.

Also, the physique feature information calculation unit 159 may correct the physique feature information of the user according to the user's physical information or health information that is not included in the skeleton information of the skeleton data. Physical information includes, for example, information about the user's body, such as height, weight, or muscle mass. Health information includes information about the user's health history, such as minor medical history, age, and physical disabilities. Information input through the operation display unit 120 may be used as the user's physical information and health information.

Also, the physique feature information calculation unit 159 may compare the center of gravity position of the skeleton data US of the user and the position of the center of gravity of the standard skeleton data RS, and calculate the physique feature information of the user based on the comparison result.

Although an example of a method for calculating the user's physique feature information has been described above, the physique feature information calculation unit 159 may calculate the user's physique feature information by combining a plurality of the above-described calculation methods. Next, an example of a correction method for correcting each of the plurality of reference skeleton data included in the reference motion information in accordance with the physique feature information of the user will be described.

<3-4. Correction of Reference Operation Information>
FIG. 7 is an explanatory diagram for explaining an example of correction of reference motion information. The conversion unit 163 converts the skeletal feature information (for example, bone length, thickness, etc.) of each part of the plurality of reference skeleton data CS1 included in the reference motion information into skeletal feature information of each part of the standard skeleton data. A conversion process C1 is executed to generate post-conversion skeleton data CS2.

Subsequently, conversion unit 163 executes correction processing C2 for correcting post-conversion skeleton data CS2 according to the physique feature information, physical information, and health information of the user calculated by physique feature information calculation unit 159. Generate skeleton data CS3.

The conversion unit 163 also executes a conversion process C1 for converting the skeleton feature information of each part of the user's skeleton data into the skeleton feature information of the standard skeleton data. This enables the motion comparison unit 167 to compare the motions of skeleton data having the same skeleton feature information.

Alternatively, the conversion process C1 may be a process of converting each piece of skeletal feature information of a plurality of pieces of skeleton data included in the reference motion information into skeletal feature information included in the user's skeleton data. Accordingly, the conversion unit 163 does not have to perform the conversion processing C1 on the skeleton data of the user.

Further, the conversion unit 163 executes a correction process C2 for correcting the converted skeleton data CS2 based on the weight parameter set for each event of the reference movement information in addition to the physique characteristic information of the user. CS3 may be generated. For example, when the user is practicing batting, the weight parameters of parts of the user that have a particularly large effect (for example, arms and hips) are set large, and the weight parameters of parts of the user that have a small effect (for example, the head) are set to be large. The parameter may be set small.

An example of correction of reference motion information according to the present disclosure has been described above. Next, a specific example of feedback based on the corrected reference motion information by the display control unit 171 will be described.

<3-5. Feedback example>
The information processing terminal 10 according to the present disclosure can provide various types of feedback to the user. Specific examples of feedback will be sequentially described below with reference to FIGS. 8 to 12. FIG.

(Feedback example 1)
FIG. 8 is an explanatory diagram for explaining Feedback Example 1 according to the present disclosure. As shown in FIG. 8, the display control unit 171 may cause the operation display unit 120 to display an image in which the skeleton data US of the user U and the reference skeleton data CS are superimposed on the user's display. More specifically, the reference skeleton data CS in the feedback example is obtained by converting the post-correction skeleton data CS3 shown in FIG. Later reference skeleton data.

Further, the operation comparison unit 167 compares the time-series data of the skeleton data US of the user U and the reference skeleton data CS, for example, compares the direction and distance of the difference between the time-series data of the skeleton data US and the reference skeleton data CS. The result is output to the display control unit 171 . Then, the display control unit 171 may cause the operation display unit 120 to display an arrow or the like indicating the direction and distance of the difference between the user's skeleton data US and the reference skeleton data CS. Display of such reference skeleton data CS and arrows is an example of guidance information that guides the user's posture.

(Feedback example 2)
FIG. 9 is an explanatory diagram for explaining Feedback Example 2 according to the present disclosure. The information processing terminal 10 may acquire the relationship between the reference motion information and the muscle load amount of each part from a database. For example, as shown in FIG. 9, when the user performs an exercise of lifting and lowering dumbbells, the conversion unit 163 calculates the muscle load amount of each part corresponding to the reference motion information acquired from the database according to the user's physique characteristic information. may be corrected. Then, the display control unit 171 causes the operation display unit 120 to display the muscle load amount of each part of the user as guidance information. As a result, the user can check the degree of achievement toward the muscle mass set in advance as a target, and can consider the type of training according to the degree of achievement.

(Feedback example 3)
FIG. 10 is an explanatory diagram for explaining Feedback Example 3 according to the present disclosure. The conversion unit 163 may correct the position, motion direction, or speed of the ball included in the heading motion as the reference motion information according to the user's physique feature information, object information, and object motion information.

Then, the motion comparison unit 167 compares the user's skeleton data with the ball position, motion direction, or speed included in each of the reference motion information, and compares various information such as the impact point, direction, or force of the ball. , to the display control unit 171 . Then, the display control unit 171 may cause the operation display unit 120 to display various types of information such as the impact point, direction, or force of the ball input from the motion comparison unit 167 as guidance information.

(Feedback example 4)
FIG. 11 is an explanatory diagram for explaining Feedback Example 4 according to the present disclosure. The solid-line skeleton data in FIG. 11 is the user's skeleton data, and the dashed-line skeleton data is the reference skeleton data. For example, when the user performs the same motion a plurality of times, the display control unit 171 displays an image in which the time-series data of the skeleton data of the user generated a plurality of times and the reference motion information are superimposed. You may make it display on the operation display part 120. FIG. This allows the user to check the user's habits and the like included in the user's actions.

In addition, the display control unit 171, for example, provides time-series data of one skeleton data obtained by averaging each skeleton data obtained by the user performing the same action a plurality of times, and the reference action information to the user. An image superimposed thereon may be displayed on the operation display unit 120 . It should be noted that such display of reference motion information is an example of guidance information.

(Feedback example 5)
FIG. 12 is an explanatory diagram for explaining Feedback Example 5 according to the present disclosure. The conversion unit 163 may correct the ball launching method of the skeleton data included in the reference motion information according to the user's physique feature information, the use of the wheelchair, the shape and type of the racket, and the like.

Then, the motion comparison unit 167 compares the skeleton data of the user and the reference motion information, and outputs the impact range of the racket and the movement direction of the wheelchair to the display control unit 171 as the comparison result. Then, the display control unit 171 may cause the operation display unit 120 to display the impact range of the racket and the movement direction of the wheelchair in addition to the skeleton data of the user and the range of the wheelchair on which the user is riding. The impact range of the racket and the motion direction of the wheelchair are examples of guidance information.

The details of the present disclosure have been described above. Next, an example of operation processing of the system according to the present disclosure will be described.

<<4. Operation example >>
<4-1. First Operation Example of Information Processing Terminal>
FIG. 13 is an explanatory diagram for explaining an operation processing example related to feedback of the information processing terminal 10 according to the present disclosure.

First, the data acquisition unit 110 photographs the user and acquires time-series data of the images (S101).

Subsequently, the posture estimation unit 151 generates user skeleton data from the time-series data of the acquired images (S105).

Also, the object detection unit 155 detects object information from the time-series data of the acquired images (S109).

Next, the communication unit 130 receives the standard skeleton data held in the standard skeleton data storage unit 231 (S113).

Then, the physique feature information calculation unit 159 compares the user's skeleton data with the standard skeleton data, and calculates the result of the comparison as the user's physique feature information (S117).

Next, the conversion unit 163 converts the skeleton information included in the user's skeleton data into skeleton information included in the standard skeleton data (S121).

Then, the communication unit 130 receives the reference motion information held in the reference motion storage unit 235 (S235).

Then, the conversion unit 163 corrects the time-series data of the skeleton data included in the reference motion information according to the user's physique feature information (S129).

Then, the motion comparison unit 167 compares the time-series data of the skeleton data of the user with the time-series data of the skeleton data included in the reference motion information (S133).

Then, the display control unit 171 causes the operation display unit 120 to display the result of the comparison (S137), and the information processing terminal 10 ends the processing.

The reference motion information may be stored in advance in the reference motion storage unit 235 as model data based on dynamics. good.

An example of acquiring another user's motion as reference motion information will be described below with reference to FIG.

(Second operation example of information processing terminal 10)
FIG. 14 is an explanatory diagram for explaining an example of operation processing of the information processing terminal 10 when acquiring another user's operation as reference operation information.

First, the data acquisition unit 110 photographs another user and acquires time-series data of the images (S201).

Subsequently, the posture estimation unit 151 generates skeleton data of other users from the time-series data of the acquired images (S205).

Also, the object detection unit 155 detects object information from the time-series data of the acquired images (S209).

Next, the communication unit 130 receives the standard skeleton data held in the standard skeleton data storage unit 231 (S213).

Then, the physique feature information calculation unit 159 compares the skeleton data of the other user with the standard skeleton data, and calculates the result of the comparison as the physique feature information of the other user (S217).

Next, the conversion unit 163 corrects the skeleton data of the other user based on the standard skeleton data and the physique feature information of the other user (S221).

Then, the conversion unit 163 corrects the corrected skeleton data of the other user according to the user's physique feature information (S225).

Then, the operation comparison unit 167 compares the time-series data of the skeleton data of the user with the corrected time-series data of the skeleton data of other users (S229).

Then, the display control unit 171 causes the operation display unit 120 to display the result of the comparison (S233), and the information processing terminal 10 ends the processing.

An example of the operation processing of the system according to the present disclosure has been described above. Next, examples of effects according to the present disclosure will be described.

<<5. Action and effect example>
According to the present disclosure described above, various effects can be obtained. For example, the information processing terminal 10 provides the user with guidance information based on reference motion information corrected according to the user's physique feature information. Therefore, the information processing terminal 10 can provide the user with guidance information that takes into consideration the range of motion of each part that may differ depending on the user.

Also, the reference motion information is corrected according to the user's physical information or health information. Thereby, for example, the information processing terminal 10 can provide the user with guidance information based on the reference motion information corrected within a range that the user can overcome, according to the user's age, medical history, and the like.

The information processing terminal 10 also compares the user's motion information with the reference motion information. Accordingly, the information processing terminal 10 can provide the user with guidance information that quantifies the difference between the user's motion information and the target reference motion information.

Also, the reference motion information may be time-series data of other users' posture information. As a result, the user can perform various types of training, for example, aiming at the motion information of a leader such as a teacher or an instructor.

Further, the information processing terminal 10 may detect object information and object motion information, and correct the reference motion information according to the detected various information. As a result, the information processing terminal 10 can provide the user with guidance information that considers possible effects of the user's training, for example, in a wheelchair, with an artificial arm, or an artificial leg.

In addition, the information processing terminal 10 may correct the reference motion information according to the weight parameter of each part owned by the user, which is set for each training event. Thereby, the information processing terminal 10 can provide the user with guidance information that reflects the degree of importance of each part, which may differ depending on the type of training.

<<6. Hardware configuration example >>
The embodiments of the present disclosure have been described above. Information processing such as generation of skeleton data and extraction of feature amounts described above is realized by cooperation between software and hardware of the information processing terminal 10 described below. Note that the hardware configuration described below can also be applied to the server 20 .

FIG. 15 is a block diagram showing the hardware configuration of the information processing terminal 10. As shown in FIG. The information processing terminal 10 includes a CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , a RAM (Random Access Memory) 1003 and a host bus 1004 . The information processing terminal 10 also includes a bridge 1005 , an external bus 1006 , an interface 1007 , an input device 1008 , an output device 1010 , a storage device (HDD) 1011 , a drive 1012 and a communication device 1015 .

The CPU 1001 functions as an arithmetic processing device and a control device, and controls general operations within the information processing terminal 10 according to various programs. Alternatively, the CPU 1001 may be a microprocessor. A ROM 1002 stores programs, calculation parameters, and the like used by the CPU 1001 . The RAM 1003 temporarily stores programs used in the execution of the CPU 1001, parameters that change as appropriate during the execution, and the like. These are interconnected by a host bus 1004 comprising a CPU bus or the like. Functions such as the physique characteristic information calculation unit 159 and the motion comparison unit 167 described with reference to FIG.

The host bus 1004 is connected via a bridge 1005 to an external bus 1006 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 1004, bridge 1005 and external bus 1006 do not necessarily have to be configured separately, and these functions may be implemented in one bus.

The input device 1008 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 1001 . etc. By operating the input device 1008, the user of the information processing terminal 10 can input various data to the information processing terminal 10 and instruct processing operations.

The output device 1010 includes display devices such as liquid crystal display devices, OLED devices, and lamps, for example. In addition, output device 1010 includes audio output devices such as speakers and headphones. The output device 1010 outputs reproduced content, for example. Specifically, the display device displays various information such as reproduced video data as text or images. On the other hand, the audio output device converts reproduced audio data and the like into audio and outputs the audio.

The storage device 1011 is a device for storing data. The storage device 1011 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 1011 is composed of, for example, an HDD (Hard Disk Drive). The storage device 1011 drives a hard disk and stores programs executed by the CPU 1001 and various data.

The drive 1012 is a reader/writer for storage media, and is built in or externally attached to the information processing terminal 10 . The drive 1012 reads out information recorded in the attached removable storage medium 30 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 1003 . Drive 1012 can also write information to removable storage medium 30 .

The communication device 1015 is, for example, a communication interface configured with a communication device or the like for connecting to the network 12 . The communication device 1015 may be a wireless LAN compatible communication device, an LTE (Long Term Evolution) compatible communication device, or a wired communication device that performs wired communication.

<<7. Supplement >>
Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field to which the present disclosure belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally belong to the technical scope of the present disclosure.

For example, the posture estimation unit 151 may generate a mesh model representing the surface shape of the user's body based on the time-series data acquired by the data acquisition unit 110 . Then, the physique feature information calculation unit 159 may calculate the difference between the user's mesh model and a predetermined mesh model as the user's physique feature information. Further, the display control unit 171 may cause the operation display unit 120 to display a screen in which the mesh model of the user and the mesh model of the reference motion information corrected according to the physique feature information are superimposed on the user. This allows the user to more clearly recognize the deviation amount of the reference motion information and the user's entire body.

In addition, the information processing terminal 10 may further include all or part of the functional configuration of the server 20 according to the present disclosure. When the information processing terminal 10 has all the functional configurations of the server 20 according to the present disclosure, the information processing terminal 10 can perform a series of processes according to the present disclosure without communicating via the network 1 .

Also, the server 20 may acquire the user's skeleton data generated by the information processing terminal 10 . Then, the server 20 may calculate the physique feature information of the user and correct the reference motion information according to the calculated user physique feature information. Then, the server 20 may transmit to the information processing terminal 10 guidance information that guides the action of the user.

In addition, when the difference between the time-series data of the user's skeleton data and the reference motion information is equal to or greater than a predetermined amount, the display control unit 171 divides the steps of the degree of achievement step by step and guides the user to the operation display unit 120 . information may be displayed. This allows the user to experience multiple achievements, which can improve the user's motivation.

Also, each step in the processing of the information processing terminal 10 and the server 20 in this specification does not necessarily have to be processed in chronological order according to the order described as the flowchart. For example, each step in the processing of the information processing terminal 10 and the server 20 may be processed in an order different from the order described as the flowchart.

It is also possible to create a computer program for causing the hardware such as the CPU, ROM, and RAM built into the information processing terminal 10 to exhibit functions equivalent to the components of the information processing terminal 10 described above. A storage medium storing the computer program is also provided.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the following configuration also belongs to the technical scope of the present disclosure.
(1)
a posture information acquisition unit that acquires posture information indicating a posture of a user;
a calculation unit that calculates physique characteristic information of the user by comparing the posture information of the user and predetermined posture information;
a correction unit that corrects the posture information prepared as reference motion information according to the user's physique feature information;
an output unit that outputs guidance information for guiding the user's motion based on the reference motion information corrected by the correction unit;
An information processing device.
(2)
The calculation unit
calculating a difference between each of the various types of skeleton information included in the posture information and each of the various types of skeleton information included in the predetermined posture information as physique feature information of the user;
The information processing device according to (1) above.
(3)
The correction unit is
correcting the posture information prepared as the reference motion information according to physical information or health information of the user that is not included in the posture information;
The information processing apparatus according to (1) or (2).
(4)
The output unit
Display information obtained by superimposing the reference motion information corrected by the correction unit and the posture information of the user is output as the guidance information.
The information processing apparatus according to any one of (1) to (3).
(5)
The information processing device is
a motion comparison unit that compares the user posture information with reference motion information corrected by the correction unit;
further comprising
The output unit
Outputting guide information for guiding the user's action based on the result of the comparison by the action comparison unit;
The information processing apparatus according to any one of (1) to (3).
(6)
a posture transformation unit that transforms the posture information of the user based on the predetermined posture information;
further comprising
The operation comparison unit
comparing the posture information of the user whose posture has been transformed by the posture transforming unit and the reference motion information corrected by the correcting unit;
The information processing device according to (5) above.
(7)
The output unit
outputting, as the guidance information, difference information between the user's posture information and the reference motion information corrected by the correction unit;
The information processing apparatus according to (5) or (6).
(8)
The reference motion information includes:
including pose information of other users,
The information processing apparatus according to any one of (1) and (7) above.
(9)
The information processing device is
an object information acquisition unit that acquires feature information of an object and motion information of the object;
further comprising
The correction unit is
correcting the reference motion information according to features of the object and motion information of the object;
The information processing apparatus according to any one of (1) to (8).
(10)
The correction unit is
correcting the reference motion information according to a weight parameter set for each part of the user, in addition to the physique characteristic information;
The information processing apparatus according to any one of (1) to (9).
(11)
the weight parameter is set for each item of the reference motion information;
The information processing device according to (10) above.
(12)
obtaining posture information indicating the posture of the user;
calculating physique feature information of the user by comparing the posture information of the user and predetermined posture information;
correcting the posture information prepared as reference motion information according to the physique characteristic information of the user;
outputting guidance information for guiding a motion of the user based on the corrected reference motion information;
A computer-implemented information processing method comprising:
(13)
to the computer,
a posture information acquisition function for acquiring posture information indicating the posture of the user;
a calculation function for calculating physique characteristic information of the user by comparing the posture information of the user and predetermined posture information;
a correction function for correcting posture information prepared as reference motion information according to the user's physique feature information;
an output function for outputting guidance information for guiding a motion of the user based on the reference motion information corrected by the correction unit;
A program that realizes

10 information processing terminal 20 server 110 data acquisition unit 120 operation display unit 130 communication unit 140 storage unit 150 control unit 151 posture estimation unit 155 object detection unit 159 physique feature information calculation unit 163 conversion unit 167 motion comparison unit 171 display control unit 210 communication Unit 220 Control unit 230 Storage unit 231 Standard skeleton data storage unit 235 Reference motion storage unit

Claims

a posture information acquisition unit that acquires posture information indicating a posture of a user;
a calculation unit that calculates physique characteristic information of the user by comparing the posture information of the user and predetermined posture information;
a correction unit that corrects the posture information prepared as reference motion information according to the user's physique feature information;
an output unit that outputs guidance information for guiding the user's motion based on the reference motion information corrected by the correction unit;
An information processing device.
The calculation unit
calculating a difference between each of the various types of skeleton information included in the posture information and each of the various types of skeleton information included in the predetermined posture information as physique feature information of the user;
The information processing device according to claim 1 .
The correction unit is
correcting the posture information prepared as the reference motion information according to physical information or health information of the user that is not included in the posture information;
The information processing apparatus according to claim 2.
The output unit
Display information obtained by superimposing the reference motion information corrected by the correction unit and the posture information of the user is output as the guidance information.
The information processing apparatus according to claim 3.
The information processing device is
a motion comparison unit that compares the user posture information with reference motion information corrected by the correction unit;
further comprising
The output unit
Outputting guide information for guiding the user's action based on the result of the comparison by the action comparison unit;
The information processing apparatus according to claim 3.
a posture transformation unit that transforms the posture information of the user based on the predetermined posture information;
further comprising
The operation comparison unit
comparing the posture information of the user whose posture has been transformed by the posture transforming unit and the reference motion information corrected by the correcting unit;
The information processing device according to claim 5 .
The output unit
outputting, as the guidance information, difference information between the user's posture information and the reference motion information corrected by the correction unit;
The information processing device according to claim 6 .
The reference motion information includes:
including pose information of other users,
The information processing apparatus according to claim 7.
The information processing device is
an object information acquisition unit that acquires feature information of an object and motion information of the object;
further comprising
The correction unit is
correcting the reference motion information according to features of the object and motion information of the object;
The information processing apparatus according to claim 8 .
The correction unit is
correcting the reference motion information according to a weight parameter set for each part of the user, in addition to the physique characteristic information;
The information processing apparatus according to claim 9 .
the weight parameter is set for each item of the reference motion information;
The information processing apparatus according to claim 10.
obtaining posture information indicating the posture of the user;
calculating physique feature information of the user by comparing the posture information of the user and predetermined posture information;
correcting the posture information prepared as reference motion information according to the physique characteristic information of the user;
outputting guidance information for guiding a motion of the user based on the corrected reference motion information;
A computer-implemented information processing method comprising:
to the computer,
a posture information acquisition function for acquiring posture information indicating the posture of the user;
a calculation function for calculating physique characteristic information of the user by comparing the posture information of the user and predetermined posture information;
a correction function for correcting posture information prepared as reference motion information according to the user's physique feature information;
an output function for outputting guidance information for guiding a motion of the user based on the reference motion information corrected by the correction function;
A program that realizes