JP2023168557A - Program, method, and information processing device - Google Patents

Abstract

To provide another technology for a user himself/herself to easily check his or her body motion.SOLUTION: There is provided a program for causing a computer to be operated. The program causes a processor of the computer to execute the steps of: sequentially sensing a body motion of a user; presenting to the user an avatar corresponding to the user in a mode to cause the shape of the body surface of the user to be perceived instead of an actually taken image when the user's external appearance is photographed, on the basis of a result of the sensing; specifying a position for drawing a graphic object as an indication of the body motion of the user in association with the avatar on the basis of the result of the sensing; and displaying the graphic object at the specified position.SELECTED DRAWING: Figure 37

Description

The present disclosure relates to a program, a method, and an information processing device.

2. Description of the Related Art A technique is known in which a user's movements are photographed using a camera or the like and the user's movements are evaluated based on the photographed moving images.

Japanese Unexamined Patent Publication No. 2016-81504 discloses a technology related to a device that uses a depth camera to recognize the type of competition based on the human skeleton, automatically evaluates the user's movement using this, and determines whether or not the user is promoted. Are listed.

Japanese Patent Application Publication No. 2016-81504

What is needed is another technology that allows users to easily see their own body movements.

According to one embodiment, a program for operating a computer is provided. The program includes the steps of having the computer processor sequentially sense the movement of the user's body, and based on the sensing results, perceive the shape of the user's body surface rather than a real image of the user's appearance. a step of presenting an avatar corresponding to the user to the user in a manner that causes the user to move; and a step of identifying, based on the sensing results, a position to draw a graphical object that is a guide for the user's body movement in association with the avatar. , displaying the graphical object at the specified position.

According to the present disclosure, while the user recognizes an avatar presented to the user based on the sensing results of the user's body movement, the user can also recognize a graphical object that is a guide to the user's body movement. This allows users to easily check their own body movements.

1 is a diagram showing the overall configuration of a measurement system 1. FIG. FIG. 2 is a block diagram of a terminal device 10 that constitutes the measurement system 1 of the first embodiment. 2 is a diagram showing a functional configuration of a server 20. FIG. 3 is a diagram showing data structures of a user information database 281, measurement item information 282, measurement history database 283, and friend list 284 stored in the server 20. FIG. A situation in which a user measures body movement using the measurement system 1 is shown. 3 is a diagram showing details of a mat 32. FIG. It is a flowchart which shows the process by which the terminal device 10 performs a ROM check and updates a measurement history. 2 is a flowchart illustrating a process in which the terminal device 10 evaluates joint range of motion by measuring the movement of the user's foot. 2 is a flowchart illustrating a process in which the terminal device 10 evaluates joint range of motion by measuring the user's upper body twisting motion. FIG. 4 is a diagram illustrating an example of notification content from the terminal device 10 in a situation where the user's body movement is measured. A situation in which a user's walking motion is measured by the measurement system 1 in the second embodiment is shown. Each part of the user's body identified by the terminal device 10 is shown based on the measurement results of the measuring device 31. It is a figure which shows the data structure of the measurement result database 287 and the evaluation result database 288 which the server 20 in 2nd Embodiment stores. It is a flowchart which shows the process which the terminal device 10 measures and evaluates a user's walking motion in 2nd Embodiment. FIG. 7 is a diagram illustrating an example of output when the terminal device 10 outputs a measurement result of measuring a user's walking motion and an evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. FIG. 7 is a diagram illustrating an example of output when the terminal device 10 outputs a measurement result of measuring a user's walking motion and an evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. 5 is a diagram illustrating an example in which the terminal device 10 outputs an evaluation result of the burden on the knee during the user's walking motion. FIG. This corresponds to step S1409 in FIG. 12 is a flowchart showing a process in which the terminal device 10 measures and evaluates the balance of the user's trunk in the third embodiment. FIG. 7 is a diagram showing an example in which the terminal device 10 outputs a measurement result of measuring the balance of the user's trunk and an evaluation result of the balance of the trunk in the third embodiment. It is a figure which shows the data structure of various data which the server 20 in 3rd Embodiment stores. This is an example of a screen in which the terminal device 10 presents possible troubles to the user's body based on measurement results and evaluation results of the user's body movements, and proposes an exercise menu corresponding to the troubles. The flow of processing of each device in the fourth embodiment is shown. It is a figure which shows the data structure of the various data which the server 20 stores in the modification of 4th Embodiment. It is a figure showing the flow of operation in a modification of a 4th embodiment. When the terminal device 10 presents possible troubles to the user's body based on the measurement results and evaluation results of the user's body movements, and proposes an exercise menu corresponding to the troubles, the user answers the questionnaire. This is an example screen of the situation. 5 is a diagram showing data structures of a trainee information database 2601, a trainer information database 2602, and a training history database 2603 stored in the server 20 in the fifth embodiment. FIG. It is a flowchart showing a series of processes of sensing the movement of the first user's body as point cloud data and transmitting information based on the first user's point cloud data obtained by sensing to the second user's terminal. 12 is a flowchart illustrating the flow of processing for receiving the trainer's voice input from the trainer's terminal and transmitting it to the trainee's terminal. Sensing the first user's body movements as point cloud data, and specifying the content of instruction for the first user based on information based on the first user's point cloud data obtained through sensing and the learned model. 3 is a flowchart showing a series of processing. When the first user's terminal device 10A senses the movement of the first user's body as point cloud data and transmits information based on the first user's point cloud data obtained by sensing to the second user's terminal. This is an example screen. A first user including the overall configuration when sensing the first user's body movement as point cloud data and transmitting information based on the first user's point cloud data obtained by sensing to a second user's terminal. and a second user's screen example. This is an example of a screen when sensing the body movements of a plurality of users as point cloud data and transmitting information based on the point cloud data of each user obtained through sensing to a defined virtual space. 10A is a diagram illustrating the concept of a process in which the terminal device 10A draws an auxiliary line based on the estimation result of the user's posture. FIG. An example of a screen presented to the user is shown. 10A is a diagram showing an example of a screen for adjusting the inclination of the terminal device 10A in front, back, right, and left directions while the user is in a standing position. FIG. FIG. 3 is a diagram illustrating a process of specifying a position at which a graphic object that is a guide to the movement of the user's body is to be drawn, based on a result of estimating the position of a specific body part of the user. 10A is a diagram showing an example of a screen on which the terminal device 10A causes the user to perform a lateral lifting motion. FIG. FIG. 3 is a diagram illustrating a process of specifying a position at which a graphic object that is a guide to the movement of the user's body is to be drawn, based on a result of estimating the position of a specific body part of the user. It is a diagram showing a process in which a user moves his or her body. FIG. 6 is a diagram showing a state in which the user fully raises both arms, which is the peak timing of the lateral lifting motion. FIG. 6 is a diagram illustrating an example in which the sensing results at the peak timing of the user's lateral lifting motion are saved and displayed as a comparable avatar. 10A is a diagram showing an example of a screen on which the terminal device 10A causes the user to perform a crouching motion. FIG. FIG. 3 is a diagram illustrating a process of specifying a position at which a graphic object that is a guide to the movement of the user's body is to be drawn, based on a result of estimating the position of a specific body part of the user. It is a diagram showing a process in which a user moves his or her body. FIG. 6 is a diagram showing a state in which the user has crouched down, which is the peak timing of the crouching motion. FIG. 6 is a diagram illustrating an example in which sensing results at the peak timing of a user's crouching motion are saved and displayed as a comparable avatar. 10A is a diagram illustrating an example of a screen on which the terminal device 10A causes the user to take a step and bend down. FIG. FIG. 6 is a diagram illustrating an example screen of a situation in which the user is shown that the user's initial posture is appropriate. FIG. 3 is a diagram illustrating a situation in which measurement of a user's motion is started. FIG. 6 is a diagram illustrating a situation in which it is determined whether or not the user's motion is appropriate, and the determined result is presented to the user. FIG. 6 is a diagram illustrating a situation in which it is determined whether or not the user's motion is appropriate, and the determined result is presented to the user.

Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

<First embodiment>
<Summary>
In the following embodiments, the flexibility of the user's body is evaluated by measuring the balance of the user's body and the movement of the user's body, and an exercise menu is provided to help the user acquire a beautiful posture (for example, a beautiful walking posture). The measurement system 1 for providing the following will be explained.

If the user's walking posture is not correct, the user will perform movements that are not necessarily necessary for walking, and the user's strength will be consumed unnecessarily, making it easy for the user to accumulate fatigue. By improving the user's walking posture, the user can move his or her body functionally, and as a result, the user can walk with less fatigue. Therefore, the measurement system 1 (1) measures the user's walking movements and evaluates the walking posture (evaluates whether the user is able to walk beautifully and with good balance), and (2) measures the user's movements while walking. Performs a ROM check that measures joint flexibility and evaluates joint range of motion (ROM: Range of motion); and (3) evaluates the user's muscle strength status and provides a training menu. .

If the user can walk in a well-balanced manner (the left and right movements of the body are not biased, the movement of muscles during walking is ideal, etc.), the way the user moves the body is functional. Here, the functional way of movement means that the user is able to move his muscles for walking without consuming unnecessary force. In other words, the user moves his/her muscles to obtain propulsive force for walking, and muscle movements lead to propulsive force with high efficiency. As a result, it is possible to give the impression that the user is walking beautifully. In order for users to learn how to walk in a well-balanced manner, it is effective to evaluate joint range of motion, and measurement system 1 not only evaluates joint range of motion but also uses the results of these joint range of motion evaluations. Based on this assumption, the user is also provided with muscle training necessary to make their walking posture look beautiful.

As described above, the measurement system 1 provides measurement (check) of the movement of the user's body and exercises for moving the user's body.

The measurement system 1 can be installed, for example, in a business office of a business company, in a commercial facility, in a store space, in a studio, in an individual's home, or the like. For example, a business company may introduce the measurement system 1 to further encourage employees to manage their health. Furthermore, by deploying the measurement system 1 at a store in a commercial facility, health management services may be provided to consumers.

<1 Configuration diagram of the entire system>
FIG. 1 is a diagram showing the overall configuration of a measurement system 1. As shown in FIG.

As shown in FIG. 1, the measurement system 1 includes a plurality of measurement units (in the illustrated example, measurement units 30A, 30B, and 30C are shown) configured by various sensors that measure the movement of the user's body. Each measurement unit may be referred to as a "measurement unit 30"), and a terminal device 10 (in the illustrated example, the terminal device 10 is shown) that controls the operation of the measurement unit. When describing a device, it may be written as "terminal device 10A", "terminal device 10B", etc. Also, each element (processor, etc.) that constitutes the terminal device may be written in the same way.) and a server 20 that accumulates the measurement results of the user's body movements.

The measurement unit 30 is connected to the terminal device 10 by wire or wirelessly. As shown in the figure, a plurality of measurement units 30 may be connected to one terminal device 10, or one terminal device 10 may be connected to each measurement unit 30 (that is, one measurement unit 30 may be connected to one terminal device 30). It is also possible to prepare two terminal devices 10 and measure users.

The measuring unit 30 includes measuring devices (in the illustrated example, measuring devices 31A, 31B, and 31C are shown. These may be collectively referred to as "measuring devices 31"), and a mat (in the illustrated example, measuring devices 31A, 31B, and 31C are shown. , mats 32A, 32B, and 32C (these may also be collectively referred to as "mats 32"), and poles that are three-dimensional objects (in the illustrated example, poles 33A, 33B, and 33C are shown). These are sometimes collectively referred to as "Paul 33").

The measuring device 31 is for measuring the movement of the user's body, and detects the distance and direction to the object to be measured using, for example, an RGB-D camera (depth sensor). The measurement device 31 may also include a microphone array or the like to detect the user's voice, the direction from which the voice is coming, and the like. For example, the measuring device 31 may be a device, such as a Kinect (registered trademark), that can accept gestures and voice recognition operations from the user. The measuring device 31 detects the user's position and the position of each joint of the user's body by detecting the user's body. Thereby, the measuring device 31 measures the movement of the user's body by setting a virtual bone that virtually connects the joints that make up the user. Furthermore, the measuring device 31 may be a terminal such as a smartphone, in addition to Kinect (registered trademark). For example, if a smartphone is equipped with a depth camera, the smartphone can acquire the user's body as point cloud data.

Furthermore, even if the measuring device 31 does not have a function to measure distance, it can identify the subject based on the captured image and use a trained model to estimate the distance between the subject and the camera. The distance to the subject (for example, the distance to each joint of the subject) may be acquired based on the following. Furthermore, even when the measuring device 31 cannot measure distance (such as an RGB camera), each part of the subject may be identified based on the photographed image and the skeleton (bone) may be estimated. Furthermore, the distance to the subject may be estimated by providing a plurality of cameras as the measuring device 31 and obtaining captured images with parallax.

The mat 32 is for displaying an approximate position of the user in order to measure the movement of the user's body. The mat 32 is placed at a position having a predetermined positional relationship with respect to the measuring device 31. That is, the mat 32 is placed within a range where the measuring device 31 can measure the movement of the user's body. The mat 32 includes a plurality of patterns for guiding the user's standing position, as will be described later. The user adjusts the position of the foot or takes a step while visually recognizing the pattern.

The pole 33 is used in conjunction with the mat 32 to measure the movement of the user's body, for example, the twisting movement of the user's body. The pole 33 is placed on the mat 32 at a position associated with a pattern that is a guideline for the standing position where the user's feet are placed. While keeping both feet on the mat 32 (that is, without moving the standing position of both feet), the user twists his body and moves to touch a pair of poles placed on both sides of the mat 32 with his hands. Therefore, the mat 32 is used as a target point for moving the user's arm in order to perform a body twisting motion. The mat 32 may be set at a height that corresponds to the user's height so that the user can grasp it with his/her hand when twisting the user's body.

As mentioned above, the combination of the measuring device 31, the mat 32, and the pole 33 that constitute the measuring unit 30 has been described. In the following description, it is assumed that one user uses one measurement unit 30. However, one measurement device 31 may be used to measure the body movements of a plurality of users. That is, a plurality of users may move their bodies within the measurement range of one measuring device 31.

The terminal device 10 and the server 20 are communicatively connected via a network 80. Furthermore, in this embodiment, each device (terminal device, server, etc.) can also be understood as an information processing device. That is, a collection of devices can be understood as one "information processing device", and the measurement system 1 may be formed as a collection of a plurality of devices. How to allocate the multiple functions required to realize the measurement system 1 according to this embodiment to one or more pieces of hardware depends on the processing capacity of each piece of hardware and/or the specifications required for the measurement system 1, etc. It can be determined as appropriate in view of the above.

The terminal device 10 is a device operated by each user. The terminal device 10 may be, for example, a stationary PC (Personal Computer) or a laptop PC. In addition, the terminal device 10 is realized by a mobile terminal such as a smartphone or a tablet that is compatible with a mobile communication system. As shown as a terminal device 10B in FIG. 1, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage section 16, and a processor 19. The server 20 includes a communication IF 22 , an input/output IF 23 , a memory 25 , a storage 26 , and a processor 29 .

The terminal device 10 is communicably connected to the server 20 via the network 80. The terminal device 10 is compatible with communication standards such as 5G and LTE (Long Term Evolution), and wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11, and is compatible with communication equipment. It is connected to the network 80 by communicating with.

The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 communicates with an external device. The input device 13 is an input device (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving input operations from a user. The output device 14 is an output device (display, speaker, etc.) for presenting information to the user. The memory 15 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 19 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

As illustrated, the terminal device 10 is connected to the measurement unit 30 by wire or wirelessly. By measuring the movement of the user's body by the measurement unit 30, it is possible to evaluate the flexibility of the user's body and the extent of joint range of motion.

The server 20 manages information about each user. The server 20 manages, as user information, parameters (for example, height) that are referred to when measuring the body movements of each user.

Specifically, the server 20 performs user matching, etc. in order to support multiple users in moving their bodies for health management and measuring their body movements as a process for promoting interaction between users. I do. Additionally, the server 20 transmits and receives messages between users.

The server 20 may manage virtual points held by each user. For example, suppose that a business company has an in-house point system that each employee can use like virtual currency. The points can be used like virtual currency for purchasing within the company (for example, they can be used for payments at in-house vending machines, etc.), or can be given to other employees. For example, when a user takes a measurement using the measurement system 1, the server 20 may give the points to the user, or when the user takes a measurement together with other users using the measurement system 1 (for example, the same Measurements may be taken at different locations, measurements taken at the same location at the same time, etc.), and the points may be given to the user, or the points may be given when the joint range of motion evaluation result exceeds a predetermined value. Good too.

The communication IF 22 is an interface for inputting and outputting signals so that the server 20 communicates with external devices. The input/output IF 23 functions as an interface with an input device for accepting input operations from a user and an output device for presenting information to the user. The memory 25 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 29 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

<1.1 Configuration of terminal device 10>
FIG. 2 is a block diagram of the terminal device 10 that constitutes the measurement system 1 of the first embodiment. As shown in FIG. 2, the terminal device 10 includes a plurality of antennas (an antenna 111, an antenna 112), a wireless communication section (a first wireless communication section 121, a second wireless communication section 122) corresponding to each antenna, and an operation A reception unit 130 (includes a touch sensitive device 131; may also include various pointing devices such as a mouse and a touch pad, or may include a keyboard and a gesture detection device), and a display 132 , an audio processing section 140 , a microphone 141 , a speaker 142 , a position information sensor 150 , a camera 160 , a storage section 180 , and a control section 190 . The terminal device 10 also has functions and configurations that are not particularly illustrated in FIG. 2 (for example, a battery for holding power, a power supply circuit that controls the supply of power from the battery to each circuit, etc.). As shown in FIG. 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

The antenna 111 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 111 receives radio waves from space and provides a received signal to the first wireless communication unit 121.

The antenna 112 radiates a signal emitted by the terminal device 10 as a radio wave. Further, the antenna 112 receives radio waves from space and provides a received signal to the second wireless communication unit 122.

The first wireless communication unit 121 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 111 so that the terminal device 10 communicates with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 112 so that the terminal device 10 communicates with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation, demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10 and provide received signals to the control unit 190.

The operation reception unit 130 has a mechanism for accepting user input operations. Specifically, the operation reception unit 130 includes a touch sensitive device 131 and the like as a pointing device. Touch sensitive device 131 receives input operations from the user of terminal device 10 . The touch sensitive device 131 uses, for example, a capacitive touch panel to detect the position of the user's touch on the touch panel. The touch sensitive device 131 outputs a signal indicating the user's touch position detected by the touch panel to the control unit 190 as an input operation. Note that the terminal device 10 may communicate with the measuring device 31 and receive the measurement results of the measuring device 31, thereby receiving input operations from the user. For example, when the measurement device 31 acquires point cloud data, it detects the user's body and each joint that makes up the user's body based on the point cloud data, and uses the user's gestures as input based on the detection results. accept. Gestures that can be input by the user include body movements such as hand movements and head movements. For example, the operation reception unit 130 may detect that the user takes a predetermined pose (raise both hands horizontally and take a "T"-shaped pose), or the operation reception unit 130 may detect that the user takes a predetermined pose with both hands raised horizontally, or that the user holds the hands in a predetermined shape. It is also possible to detect that the

The display 132 displays data such as images, videos, and text under the control of the control unit 190. The display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The terminal device 10 and the output device (display 132, speaker 142) may be realized as separate devices.

The audio processing unit 140 modulates and demodulates the audio signal. The audio processing unit 140 modulates the signal provided from the microphone 141 and provides the modulated signal to the control unit 190. The audio processing unit 140 also provides an audio signal to the speaker 142. The audio processing unit 140 is realized, for example, by a processor for audio processing. The microphone 141 receives a voice input and provides a voice signal corresponding to the voice input to the voice processing unit 140 . The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10 .

The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. The satellite positioning system receives signals from at least three or four satellites, and detects the current position of the terminal device 10 equipped with a GPS module based on the received signals. For example, in the measurement system 1, when recording the location where the user performed the measurement, the detection result of the location information sensor 150 (information on latitude and longitude, information on a facility specified by latitude and longitude and map information, information on a facility identified by the latitude and longitude and map information, (e.g., information on business establishments that can be used) may be stored in the server 20 in association with the measurement results. In other words, the terminal device 10 may determine the location where the user performed the measurement based on the detection result of the position information sensor 150 without the user inputting the location.

The camera 160 is a device that receives light with a light receiving element and outputs it as a photographed image. The camera 160 is, for example, a depth camera that can detect the distance from the camera 160 to the object to be photographed. In this embodiment, the measuring device 31 may be realized by the camera 160 of the terminal device 10. That is, in addition to realizing the terminal device 10 and the measuring device 31 as separate devices, the terminal device 10 may be equipped with the functions of the measuring device 31.

The storage unit 180 is configured with, for example, a flash memory, and stores data and programs used by the terminal device 10. In one aspect, the storage unit 180 stores user information 181, measurement item information 182, measurement history information 183, friend list 184, and training item information 185. The details of these will be described later, but the outline is as follows. The terminal device 10 may acquire this information from various databases of the server 20, which will be described later.

User information 181 is information about users in measurement system 1. The user information includes information that identifies the user, the user's name, the user's affiliation, the user's physical information (height, weight, age, etc.), and the like.

The measurement item information 182 is information on each measurement item when performing a ROM check. The measurement item information 182 includes information on an audio guide output to the user for each measurement item.

The measurement history information 183 indicates the history of measurement results obtained by measuring the user's body movements.

The friend list 184 shows information about users who are registered as friends. It is also possible to allow messages to be sent and received between users registered as friends.

Training item information 185 includes information on each training item when providing a muscle training menu to the user. For example, it includes the name of the training and video and audio information that serves as a guide for the user to move his or her body. The terminal device 10 uses the measuring device 31 to measure the movement of the user's body using video and audio as a guide, and sequentially displays the measurement results on the display 132. The terminal device 10 compares the ideal way of moving the body with the measurement results (movements of each joint) when the user moves the body on the display 132, and determines the ideal way of moving the body. It is also possible to prompt the user to do so.

The control unit 190 controls the operation of the terminal device 10 by reading a program stored in the storage unit 180 and executing instructions included in the program. Control unit 190 is, for example, an application processor. The control unit 190 functions as an input operation reception unit 191, a transmission/reception unit 192, a data processing unit 193, and a notification control unit 194 by operating according to a program.

The input operation reception unit 191 performs a process of accepting a user's input operation on an input device such as the touch sensitive device 131. The input operation reception unit 191 determines whether the user's operation is a flick operation, a tap operation, or a drag (swipe) operation based on the information of the coordinates where the user touches the touch sensitive device 131 with a finger or the like. Determine the type of operation, such as whether it is an operation.

The transmitting/receiving unit 192 performs processing for the terminal device 10 to transmit and receive data with external devices such as the server 20 and the measurement unit 30 according to a communication protocol.

The data processing unit 193 performs calculations on the data input by the terminal device 10 according to a program, and outputs the calculation results to a memory or the like.

The notification control unit 194 performs processes such as displaying a display image on the display 132 and outputting audio to the speaker 142. For example, when checking the ROM, the notification control unit 194 outputs an audio guide, and when providing a training menu for exercise, the notification control unit 194 outputs an audio guide and a video showing a model body movement to the user. present.

<1.2 Functional configuration of server 20>
FIG. 3 is a diagram showing the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication section 201, a storage section 202, and a control section 203.

The communication unit 201 performs processing for the server 20 to communicate with an external device such as the terminal device 10.

The storage unit 202 stores data and programs used by the server 20. The storage unit 202 stores a user information database 281, measurement item information 282, a measurement history database 283, a friend list 284, and the like.

The user information database 281 shows information on each user who performs measurements using the measurement system 1. Details will be described later.

Measurement item information 282 indicates information on each of a plurality of measurement items. Details will be described later.

The measurement history database 283 shows the history of measurements performed by each user. Details will be described later.

The friend list 284 is a database for holding information on users registered as friends by each user.

The training item information 285 includes information on each training item when providing a training menu for exercise to the user. The training menu includes information on a plurality of training items depending on the load on the user's body, the body part to be trained, and the like.

The control unit 203 performs functions shown as various modules by the processor of the server 20 performing processing according to a program.

The operation content acquisition module 2041 acquires the user's operation content. The operation content acquisition module 2041 includes, for example, an operation for the user to acquire past measurement results by the measurement unit 30 and evaluation results of the user's body movements, an operation for sending and receiving messages with other users, and a measurement by the measurement unit 30. We accept operations to apply to participate in measurements when there is an opportunity to do so.

The reception control module 2042 controls the process by which the server 20 receives signals from an external device according to a communication protocol.

The transmission control module 2043 controls the process by which the server 20 transmits a signal to an external device according to a communication protocol.

Matching module 2044 matches users. For example, the matching module 2044 may detect that a plurality of users have taken measurements using the measurement system 1 at the same measurement venue, match the plurality of users, and prompt them to register as friends. Further, the matching module 2044 may extract and match users who have the same history of measurement by the measurement system 1, or may extract and match users whose ROM check results by the measurement unit 30 are similar. It is also possible to do so. By matching multiple users in this way and encouraging them to register as friends, it is possible to encourage multiple users to train together to achieve a beautiful posture, so measurement can be continued using the measurement system 1. can support.

The measurement progress module 2045 advances the process of measuring the movement of the user's body by the measurement unit 30. The measurement progress module 2045 reads out each measurement item shown in the measurement item information 282, and sequentially outputs to the user audio guidance for measurement according to the measurement item. For example, the server 20 may transmit each measurement item to the terminal device 10 and may sequentially measure the movement of the user's body according to each measurement item.

The training progress module 2046 advances the process of providing a training menu to the user through video and audio. The training progress module 2046 reads each training item shown in the training item information 285 and sequentially outputs to the user an audio guide, a video guide, etc. for training according to the training item.

<2 Data structure>
FIG. 4 is a diagram showing the data structure of the user information database 281, measurement item information 282, measurement history database 283, and friend list 284 stored in the server 20.

As shown in FIG. 4, each record of the user information database 281 includes the item "user identification information (user ID)", the item "user name", the item "gender", the item "height", and the item " ``Weight'', item ``affiliation'', item ``evaluation value'', item ``company points'', etc.

The item "user identification information (user ID)" is information that identifies each user.

The item "user name" is a name set by the user.

The item "gender" indicates the gender of the user. For example, when a user registers user information in order to start using the measurement system 1, the gender may be set.

The item "height" indicates the height of the user as information to be referred to when measuring the movement of the user's body.

The item "weight" indicates information about the user's weight. In cases where weight information is not referred to when measuring the user's body movements, data regarding the item "weight" may not be retained.

The item "Affiliation" indicates information about the user's affiliation, such as when the user belongs to an organization. For example, the item "Affiliation" includes information such as company name, organization name, office name, etc. The item "affiliation" may include information about the location of the office where the user works.

The item "evaluation value" includes information on an evaluation value evaluated based on a measurement result obtained by a user performing a ROM check in the measurement system 1, and an evaluation value based on a measurement result obtained by measuring the user's walking posture. The evaluation value includes, for example, information on the classification results when the terminal device 10 classifies the results of the user's ROM check based on the measurement results by the measurement unit 30.

The item "Internal points" indicates the amount of virtual points that the user has, if there are virtual points used in the business company to which the user belongs.

Each record of the measurement item information 282 includes the item "Measurement item number (measurement item No.)", the item "Measurement site", the item "Guide voice", the item "Foot position on the mat", and the item " body part to be measured.

The item "measurement item number (measurement item No.)" is information for identifying each measurement item.

The item "measurement site" indicates the body site to be measured. For example, the following areas can be evaluated for joint range of motion:

(i) Measurement of upper limbs - Shoulder blade: Directions of movement include flexion, extension, fist-up, and pull-down.
- Shoulder: Directions of movement include flexion, extension, abduction, adduction, external rotation, and internal rotation.
・Elbow: The directions of movement include flexion and extension.
・Forearm: There are two directions of movement: pronation and supination.
- Hand: Directions of movement include flexion, extension, flexion, and ulnar flexion.

(ii) Measurement of fingers: Thumb: Directions of movement include flexural abduction, ulnar adduction, palmar abduction, palmar adduction, flexion, and extension.
・Fingers: Directions of movement include flexion, extension, abduction, and adduction.

(iii) Measurement of lower limbs - Crotch: Directions of movement include flexion, extension, abduction, adduction, external rotation, and internal rotation.
・Knee: The direction of movement is flexion and extension.
・Legs: Directions of movement include flexion and extension.
・Foot: Directions of movement include eversion, inversion, abduction, and adduction.
- Thumb: Directions of movement include flexion and extension.
・Toes: Directions of movement include flexion and extension.

(iv) Measurement of the trunk - Neck: Directions of movement include flexion, extension, rotation (left rotation, right rotation), and lateral flexion (left flexion, right flexion).
- Thoracolumbar region: Directions of movement include flexion, extension, rotation (left rotation, right rotation), and lateral flexion (left flexion, right flexion).

The item "guide audio" indicates the content of the audio that is notified to the user when performing the measurement indicated in the measurement item. The audio guide includes a voice when the user does not place their feet in the specified position on the mat 32 (for example, "Please place both feet on the specified position on the mat"), and a voice when the user does not place their feet on the mat 32 for measurement. When the terminal device 10 detects that the user has placed their feet in a prescribed position on the mat, a voice prompting the user to move their body (for example, "Please twist your body while placing both feet in the prescribed position on the mat") and a voice (for example, "measured successfully") when measurement data of the user's body movement is obtained based on the measurement results of the measurement device 31.

The item "position of feet on mat" includes information on where on the mat 32 the user's feet are placed (position where the user places his feet on the mat 32) when performing measurement. For example, the terminal device 10 detects whether the user's feet are placed at a prescribed position on the mat, based on the information of the item “position of feet on the mat”. Upon this detection, the terminal device 10 outputs to the user a voice indicating the content of the body movement for measurement.

The item "body part to be measured" indicates a part of the user's body to be measured in the measurement item.

Each record in the measurement history database 283 includes an item "measurement number (measurement No.)", an item "date and time", an item "user identification information (ID)", an item "measurement location", and an item "measurement data". and the item “evaluation results”. The server 20 receives the measurement results of the user's body movements from the terminal device 10 . The server 20 updates the records in the measurement history database 283 based on the information received from the terminal device 10.

The item "measurement number (measurement No.)" indicates each measurement result when the user measures the movement of the body using the measurement unit 30.

The item “date and time” indicates the timing at which the user performed the measurement using the measurement unit 30. .

The item “user identification information (ID)” indicates information for identifying the user whose body movement was measured by the measurement unit 30.

The item “measurement location” indicates the location where the user performed measurement using the measurement unit 30. When the server 20 receives a measurement result from the terminal device 10, the server 20 refers to the user information associated with the measurement result and determines whether the user has performed the measurement based on the information associated with the user (user information database 281). You may also specify the location. For example, based on the information on the user's affiliation shown in the user information database 281, the information on the location where the affiliation is located may be the location where measurement was performed by the measurement unit 30. Alternatively, the terminal device 10 may obtain location information using the location information sensor 150, so that the location of the terminal device 10 may be determined as the location where the measurement was performed by the measurement unit 30.

The item “measurement data” indicates the measurement result of the user's body movement measured by the measurement unit 30. For example, when the user's body movement is measured as time-series point cloud data, the time-series point cloud data may be the measurement result in the item "measurement data." Further, regarding the user's body measured by the measuring device 31, the terminal device 10 identifies the positions of the user's joints based on the measurement results, and the movement of each joint is stored in the server 20 as the measurement result of the user's body movement. It may also be retained. For example, the terminal device 10 can measure the chronological movement of the user's joints by detecting specific body parts such as shoulders and elbows based on captured images of the user's body.

The item "evaluation result" indicates the evaluation result of the level of the user's body movement based on the measurement results shown in the item "measurement data." For example, when holding time-series data of the user's joint movements in the item "measurement data," the terminal device 10 evaluates the user's body movements by comparing it with reference body movement data. do. For example, the terminal device 10 may store multiple levels of data as reference data depending on the degree of the user's body movement, so that the terminal device 10 can select a movement that is most similar to the measurement result of the user's body movement. Reference data can be specified.

Each of the records in the friend list 284 includes the item "User 1", the item "User 2", the item "Friend registration date", the item "Measurement history at the same timing", the item "Message sending/receiving history", etc. include.

The item "User 1" indicates one of the users who are mutually registered as friends.

The item "User 2" indicates the other user among the users who are mutually registered as friends.

The item "friend registration date" indicates the date and time when the users were registered as friends.

The item "Measurement history at the same timing" indicates a history in which each user registered as a friend performed measurements at the same location or at the same timing. For example, when the measurement unit 30 is placed in an office space of a business, multiple users may perform measurements at the same timing by performing measurements within a certain time period. . The item "Measurement history at the same timing" may include information on locations where multiple users have performed measurements. For example, by referring to the friend list 284, the server 20 can identify the frequency with which a plurality of users are both measuring body movements using the measurement unit 30 and training using the training menu.

The item "Message transmission/reception history" indicates the history of messages sent/received between users who are registered as friends.

<3 Configuration of measurement unit 30>
FIG. 5 shows a situation in which a user measures body movements using the measurement system 1. FIG. 5 shows an example in which a plurality of measurement units 30 are installed and a plurality of users perform measurements. Further, an example is shown in which one terminal device 10 is connected to a plurality of measurement units 30.

The measuring device 31A measures the movement of the body of the first user User_A in a sensing range 38A. In the detection range 38A, a mat 32A is installed at a position that has a predetermined positional relationship with the measuring device 31A. Here, the position of the mat 32A may be, for example, a certain distance (for example, about several meters) from the front of the measuring device 31A. The mat 32A may be installed with respect to the measuring device 31A so that when the first user User_A rides on the mat 32A, the measuring device 31A can measure the movement of the user's whole body. Pole 33A is installed on the right side and left side of mat 32A, respectively. For example, if there is a pattern on the mat 32A that indicates a predetermined position (which may also be called a home position) for the user to place both feet, a set of poles 33A is placed on the outside of the mat 32A based on the pattern. It may also be arranged. The pole 33A is installed as a guide for the user to grab when the user twists his or her upper limbs on the mat 32A with both feet in a prescribed position. As a result, when the user makes a twisting motion of the upper limb, the user is prompted to move his/her hand toward the pole 33A, so that the form of the upper limb movement can be stabilized to some extent even when measurement opportunities are increased. In other words, variations in time-series measurement results when measuring the user's body movements can be suppressed.

The measuring device 31B measures the movement of the second user User_B's body in a sensing range 38B. In the detection range 38B, a mat 32B is installed at a position that has a predetermined positional relationship with the measuring device 31B. Here, the position of the mat 32B may be, for example, a certain distance (about several meters) from the front of the measuring device 31B. The mat 32B may be installed relative to the measuring device 31B so that when the second user User_B rides on the mat 32B, the measuring device 31B can measure the movement of the user's whole body. Pole 33B is installed on the right side and left side of mat 32B, respectively.

The first user User_A moves his body by checking the position of his feet using the pattern on the mat 32A as a guide while visually checking the video and audio displayed on the display 132A. Similarly, the second user User_B moves his body according to the video and audio guide displayed on the display 132B. Thereby, the body movements of the first user User_A and the second user User_B can be measured by the measuring devices 31A and 31B, respectively.

In the illustrated example, the first user User_A is about to start a bending and stretching exercise after placing both feet at the home position on the mat 32A based on the output from the display 132A. The second user User_B is making a step with his right foot according to the output from the display 132B.

FIG. 6 is a diagram showing details of the mat 32. As shown in the figure, patterns that serve as a guide for placing the user's right foot and left foot are arranged at multiple locations.

For example, positions for placing the right foot such as a first home position 41R, a second home position 42R, and a third home position 43R are used as a guide for placing both feet of the user across multiple measurement items. are shown multiple times.

For the left foot, a plurality of positions for placing the left foot are shown, such as a first home position 41L, a second home position 42L, and a third home position 43L.

Note that in the following description, the home positions 41R and 41L may be collectively referred to as the home position 41, etc. Similarly, home positions 42R and 42L may be collectively referred to as home position 42, and home positions 43R and 43L may be collectively referred to as home position 43.

For each measurement item (measurement item information 282), the terminal device 10 guides the user to perform a predetermined body movement after specifying the interval at which the user spreads the right and left legs. For example, information on the home position to be used may be provided in the audio guide accompanying the measurement of the measurement item.

When measuring the measurement item, the terminal device 10 determines whether the user is using a predetermined home position based on the measurement result of the measurement device 31. For example, when the terminal device 10 identifies each of the home position patterns shown on the mat 32 based on the sensing results of the measuring device 31, based on the detection results of the user's ankles, etc. It may be possible to identify the pattern. If a user places their feet in a position different from the expected home position when measuring a measurement item, prompt the user to place their feet in the appropriate position using audio or video guidance. This may also be done by giving a notification.

In addition, the terminal device 10 includes, as a measurement item, the movement of each joint when the user makes a step-stepping motion, which the terminal device 10 uses the measurement unit 30 to measure. Using the measurement items, the terminal device 10 can evaluate, for example, the level of balance when the user moves his or her body by taking a step.

The mat 32 includes a pattern group 45R that is a target for stepping with the right foot when there is a stepping motion as a measurement item. Similarly, it includes a pattern group 45L that is a target for stepping with the left foot.

These pattern groups 45R and 45L may be collectively referred to as "pattern group 45."

The pattern group 45 includes patterns that serve as a guideline for a plurality of stepping positions depending on the user's height. In the illustrated example, patterns are arranged in 10 cm increments in a range of user heights from "140 cm" to "200 cm." By making the pattern imitate the toe portion of a foot, the user can easily recognize that this is the target position for stepping forward. For example, the user places both feet at the initial positions of the home positions 41R and 41L or the positions of the patterns 44R and 44L (these may be collectively referred to as "patterns 44"), and moves one foot at a time according to the user's height. Make a step motion to reach the position of the pattern. The terminal device 10 uses the measurement unit 30 to measure the movement of the user's body (movement of the joints) when performing the stepping motion. When measuring the stepping action of the user, the terminal device 10 specifies the home positions 41 to 43 or the pattern 44 as the initial position of the foot, depending on the distance the user wants the user to step, and prompts the user to do so. It can also be used as a guide.

In the illustrated example, the position of the pole 33 is also shown. For example, the poles 33 (on the right and left sides) may be arranged so as to be lined up in a straight line with respect to the home positions 41, 42, and 43. For example, the user attempts to grasp the pole 33 by twisting the upper limbs while placing both feet at one of the home positions 41, 42, and 43.

Note that, as shown in the figure, arrow patterns may be arranged in the pattern groups 45R and 45L. In the illustrated example, the arrow pattern is arranged to overlap the patterns corresponding to the heights of the pattern groups 45R and 45L. Thereby, the user can be made aware of the action of stepping forward, and when the user looks at the mat 32, it is possible to reduce the possibility that the user is confused about how to move the body during measurement.

In addition, as shown in the figure, regarding the size of the pattern, since the pattern serves as a guide for the user to place their foot, the length in the vertical direction (direction in which the foot is stepped) is the size of a typical foot. It may be about the same size (length about 22 cm to 28 cm). The shape of the pattern may be a footprint or a shape similar to a footprint so that the user can easily understand that the user is placing his or her foot on the pattern.

<4 Operation>
Hereinafter, a measurement process in which the terminal device 10 measures the movement of the user's body will be described.

FIG. 7 is a flowchart showing a process in which the terminal device 10 performs a ROM check and updates the measurement history.

In step S700, the control unit 190 of the terminal device 10 displays a screen on the display 132 for logging in by a user who performs measurements, and receives an operation for logging in from the user. The terminal device 10 authenticates the user by, for example, reading information from a recording medium such as an IC card owned by each user, or by reading a two-dimensional code displayed on a communication terminal owned by the user. Get information. For example, by communicating with the server 20, the terminal device 10 registers information such as the user's height (user information database 281), past measurement history (measurement history database 283), and the user as a friend. information on other users (friend list 284). Additionally, the terminal device 10 acquires measurement item information (measurement item information 282) from the server 20 and stores it as measurement item information 182.

In step S701, the terminal device 10 detects that the mat 32 and the pole 33 are installed in the measurement range of the measurement device 31 based on the measurement results of the measurement device 31. For example, the terminal device 10 detects the mat 32 and the pole 33 by performing image recognition using the measurement result of the measuring device 31 as a two-dimensional image. The terminal device 10 detects that the mat 32 and the pole 33 are at predetermined positions with respect to the measuring device 31. That is, the terminal device 10 detects that the mat 32 and the pole 33 are in a position suitable for measurement and in a position where the user can easily check the audio and video of the guide.

In step S703, the terminal device 10 refers to the measurement item information 182 and performs a process of sequentially measuring each measurement item. The terminal device 10 reads out the first measurement item and outputs information (such as the position where the user places the foot on the mat 32) that serves as a guide for the measurement using audio and video. The terminal device 10 detects the user's joints based on the measurement results of the measuring device 31, and detects that the user's feet are placed at predetermined positions on the mat (positions defined by the measurement items).

In step S705, the terminal device 10 outputs a voice urging the user to make a predetermined movement with his or her body. The terminal device 10 measures the movement of the user's body by detecting the movement of the user's body using the measuring device 31 .

In step S707, the terminal device 10 sequentially reads the measurement items from the measurement item information 182, outputs audio guidance, detects that the user's feet are in the specified position on the mat 32, and detects the movement of the user's body. Measure.

In step S709, when the measurement of each measurement item is completed, the terminal device 10 evaluates the range of motion of the user's joints based on the measurement results of the user's body movements. Specifically, the terminal device 10 identifies the position of each joint of the user based on the measurement results of the measuring device 31, and performs the measurement based on time-series data of the joint position (movement of the joint position that changes over time). Assess the range of motion of the target body part. The terminal device 10 outputs the evaluation result of the joint range of motion on the display 132 or the like, and transmits it to the server 20.

In step S753, the server 20 receives the user's ROM check result from the terminal device 10, and updates the database (measurement history database 283, user information database 281) based on the user's measurement result.

Next, a description will be given of the processing of each device when measuring the movement of a user stepping forward as a measurement item.

FIG. 8 is a flowchart showing a process in which the terminal device 10 evaluates joint range of motion by measuring the movement of the user's foot.

In step S801, the terminal device 10 determines that the user's feet are placed at a prescribed position on the mat 32 (item "Foot position on mat" of measurement item information 282) based on the measurement results of the measuring device 31. Detect. The terminal device 10 may guide the user to place the user's feet in the prescribed position if the user's feet are not placed in the prescribed position.

In step S803, the terminal device 10 outputs audio and video that guide the user to step the user's right foot from a prescribed position to a stepping position corresponding to the user's height.

In step S805, the terminal device 10 detects that the user's right foot has stepped from a prescribed position to another position on the mat based on the measurement result by the measuring device 31, and measures the movement of the user's right foot. Record the results. If the user's right foot steps to a position different from the target position, the terminal device 10 guides the user to step the right foot to the target position again in order to re-measure. It may also be a thing.

In step S807, the terminal device 10 outputs a voice that guides the user to step the left foot from a prescribed position to a step position corresponding to the user's height (step length).

In step S809, the terminal device 10 detects that the user's left foot has stepped from a prescribed position to another position on the mat based on the measurement result of the measuring device 31, and measures the movement of the user's left foot. Record the results.

In step S<b>811 , the terminal device 10 evaluates the range of motion of the joints of the lower limbs based on the measurement results of the movement of the user stepping on the mat 32 , outputs the evaluation results to the display 132 , etc., and transmits them to the server 20 .

Next, the processing of each device when measuring a user's upper body twisting motion will be described.

FIG. 9 is a flowchart showing a process in which the terminal device 10 evaluates joint range of motion by measuring the user's twisting motion of the upper body.

In step S901, the terminal device 10 detects that the user's feet are placed at prescribed positions on the mat based on the measurement results of the measuring device 31.

In step S903, the terminal device 10 determines whether the height of the pole 33 is appropriate based on information on the user's height. For example, the terminal device 10 associates and stores information about the height of the pole 33 that is appropriate depending on the height of the user, so that the height of the pole 33 detected by the measurement result of the measuring device 31 is appropriate. Determine whether it exists or not. For example, the appropriate height of the pole 33 may be determined based on the position of the user's hand when the user stands upright. Here, the position of the hand when the user stands upright may vary depending on the height of the user. In other words, it is assumed that the taller the person is, the higher the position of their hands will be when standing upright. Further, the terminal device 10 detects the position of the user's hand (height from the ground) when the user is in an upright posture based on the measurement result of the measuring device 31, and thereby determines the appropriate height of the pole 33. The determination may be made based on the position of the hand. That is, the height of the pole 33 may be adjusted according to the measurement result of the user's body by the measuring device 31.

Note that the height of the pole 33 is determined by, for example, determining the first position where the pole 33 contacts the ground and the second position where the top of the pole 33 is located based on the measurement results of the measuring device 31. The determination may be made based on information on the distances from the position of the device 31 to the first and second positions of the poles 33. In addition, if the pole 33 has a structure that allows it to be stretched, the pattern of the pole 33 includes a pattern indicating the height of the pole 33, and the terminal device 10 indicates the height of the pole 33. It is also possible to detect a pattern.

The terminal device 10 may determine whether the position where the pole 33 is installed is appropriate. For example, the terminal device 10 detects a pattern indicating the home position etc. on the pole 33, and determines whether the position of the pole 33 is appropriate based on the positional relationship between the position of the pattern and the position where the pole 33 is in contact with the ground. It may also be determined whether or not there is.

In step S905, the terminal device 10 twists the user's body to the right side while keeping both feet on the mat at the specified positions, puts the right pole with the left hand (pole 33 placed on the right side), and the left pole with the right hand (pole 33 placed on the left side). It outputs audio and video that guide the user to grab the pole 33).

In step S907, the terminal device 10 records the measurement result of the user's twisting of the body to the right based on the measurement result of the measurement device 31.

In step S909, the terminal device 10 outputs a voice that guides the user to twist his or her body to the left while keeping both feet on the mat at a prescribed position, and to grasp the left pole with the right hand and the right pole with the left hand.

In step S911, the terminal device 10 records the measurement result of the user's twisting of the body to the left based on the measurement result.

In step S913, the terminal device 10 evaluates the range of motion of each joint of the body based on the measurement result of the user's body twisting motion, and outputs the evaluation result. In evaluating the range of motion of a joint, whether or not the user was able to grasp the pole 33 by twisting his or her upper body may also be used as an evaluation item.

As described above, by setting the height of the pole 33 to be approximately the same as the height of the user's hand when standing upright, when the user twists his/her upper body, the pole 33 can be seen while looking back. I am trying to move my hands and arms with a goal of 33. Thereby, it is possible to reduce the amount of vibration caused by the user's twisting of the upper body, and when the terminal device 10 accumulates measurement results, it becomes easy to compare the time-series data of the measurement results.

<5 Screen example>
FIG. 10 is a diagram illustrating an example of notification content from the terminal device 10 in a situation where the user's body movement is measured.

The example screen in FIG. 10A shows a situation in which the terminal device 10 receives an operation from the user to log in to perform measurement. This corresponds to the process of step S700 in FIG. For example, the measurement unit 30 is installed in a business office, a store space, etc. of a business company, and when a plurality of users use the measurement unit 30, the users are first authenticated.

As shown in FIG. 10(A), the terminal device 10 displays a notification section 132L on the display 132. The terminal device 10 displays operation details for the user to log in on the notification section 132L. In the illustrated example, the measurement unit 30 is installed in a business office, and the notification section 132L indicates that login is possible by reading the user's employee ID card. The terminal device 10 displays on the notification unit 132C that another user is attempting to perform measurement at the same timing. As in the example of FIG. 5, when a plurality of measurement units 30 are installed and a plurality of users take measurements together, the terminal device 10 displays the progress of another user making the measurements on the notification unit 132C. . In the example of FIG. 10(A), the terminal device 10 displays on the notification unit 132C that another user who performs measurement at the same timing has already completed the login process.

The example screen in FIG. 10(B) shows a situation in which the terminal device 10 measures the movement of the user stepping forward. This corresponds to the processing in steps S801 to S811 in FIG.

As shown in FIG. 10(B), the terminal device 10 displays a notification section 132D, a notification section 132E, a guide output section 132F, and a detection result display section 132G on the display 132.

The terminal device 10 displays information on the logged-in user (that is, the user performing the measurement) on the notification unit 132D. This allows the user to easily check whether he or she is mistakenly performing measurements as another user.

The terminal device 10 displays a history of measurements performed by the user on the notification unit 132E. In the illustrated example, the number of times the user has performed measurements using the measurement unit 30 is displayed, but in addition to this, evaluation results of past measurements, etc. may also be displayed. Thereby, the user can easily confirm that the measurement is being continued, and can be motivated to continue the measurement.

In the guide output unit 132F, the terminal device 10 notifies the user of information on the target pattern for stepping according to the user's height using video and audio. Thereby, the user can easily imagine stepping toward the target pattern, and can easily perform the measurement operation.

The terminal device 10 displays the detection result by the measuring device 31 on the detection result display section 132G. For example, the terminal device 10 displays information such as that both feet of the user are in a prescribed position for measurement (both feet are placed at the home position) on the detection result display section 132G. As a result, the user can easily recognize that the measurement is being performed in accordance with the correct procedure, and can perform the measurement with a sense of security that no measurement errors will occur.

The screen example in FIG. 10C shows a situation in which the user's upper body twisting motion is measured on the terminal device 10.

In the screen example of FIG. 10(C), the terminal device 10 detects that the height of the pole 33 is not appropriate, and instructs the user to extend the pole 33 to a predetermined height using the guide output unit 132F. There is a display encouraging you to do so. This corresponds to the process of step S903 in FIG.

The example screen in FIG. 10(D) shows a situation in which the terminal device 10 notifies the user that the measurement has been completed, the evaluation result based on the measurement result, and a request to register as a friend with another user.

In the screen example shown in FIG. 10(D), the terminal device 10 displays, on the guide output unit 132F, the fact that measurement has been completed for all of the plurality of measurement items, and the results of evaluating the user's body movements for the measurement items. The terminal device 10 refers to the user's past measurement history and displays the comparison results with the past measurement results (such as a change in joint range of motion compared to the previous time) on the guide output unit 132F.

The terminal device 10 displays, on the notification unit 132M, a message prompting the user to register the user as a friend with another user, and the reason for suggesting the other user as a friend candidate (measurements were taken at the same location, etc.).

<Summary>
(1) As described above, according to the present embodiment, when checking the ROM, there is no need to apply a physical measuring instrument such as an angle meter to the user's body. You can easily measure it yourself. Further, by proceeding with the measurement according to the measurement items, it is possible to evaluate the range of motion of each joint of the user's lower body and upper body.

(2) When measuring the movement of the user's body, the mat 32 and the pole 33 can be used to make the user recognize the target point where the user moves the body. This stabilizes the form in which the user moves his or her body, making it easier for the user to accumulate and compare measurement results and check changes in body movement. For example, by using the mat 32, it is possible to (i) specify the stride length and range of movement when the user takes a step, and to instruct the user to perform the specified movement in the measurement; can be encouraged. When the user's body motion is continuously measured, by defining the motion that the user performs during the measurement, it is possible to easily compare the measurement results over time. In other words, by defining the user's body movements, it is possible to more easily evaluate how the user's body movements have changed. Furthermore, by using the mat 32, (ii) the user's standing position with respect to the measuring device 31 can be defined. The terminal device 10 can determine whether the mat 32 is placed at a predetermined position with respect to the measuring device 31 based on the measurement result of the measuring device 31. Therefore, when measuring the movement of the user's body using the mat 32, the user's standing position is defined by each pattern of the mat 32. Thereby, the movement of the user's body can easily fall within the measurement range of the measuring device 31. For example, even if the user makes a motion of sticking out their arm directly above or to the side, by placing the mat 32 in a predetermined position relative to the measuring device 31 in advance, the movement of the user's body will be controlled by the measuring device 31. It is possible to prevent the sensor from going out of the measurement range. Furthermore, by using the mat 32, (iii) when having the user take a specific action, it is possible to make the user take an appropriate posture before taking measurements. For example, the home position can define the user's standing position, and when the user performs a predetermined action (squatting, raising one leg, etc.), the user is in a position suitable for measurement (e.g., with a predetermined foot width). measurement can be started.

With these, the flexibility of each joint of the user's whole body can be easily evaluated.

(3) Furthermore, in the first embodiment, the terminal device 10 may suggest an exercise suitable for the user according to the measurement results of the user's body movements (such as the results of a ROM check). . For example, assume that the terminal device 10 holds exercise menus (training item information 185, 285) in association with classification results of ROM check evaluation results (measurement history information 183, measurement history database 283). That is, it is assumed that information on a proposed training menu is held in advance in association with measurement results and evaluation results of the user's body movements. The terminal device 10 refers to the measurement history database 283, identifies an exercise menu to be proposed to the user based on the measurement results and evaluation results of the user's body movements such as ROM check, and displays the identified exercise menu on the display 132, etc. It is also possible to present the information to the user by displaying the information on the screen or the like. For example, the range of motion of each joint of the user can be evaluated as a result of the ROM check. The server 20 may predetermine in the training item information 285 the range of motion of a joint necessary for the user to perform an exercise. For example, as a result of the ROM check, if the range of motion of the user's hip joint is smaller than a certain level (if the hip joint is stiff), the predetermined training item is not provided to the user as an exercise (the terminal device 10 Processing such as displaying images may not be executed). Further, based on the results of the ROM check, the user may be provided with an exercise menu according to the evaluation results of each joint of the user's body.

In addition, the terminal device 10 measures the movement of the user's body while performing the training shown in the training item, and determines whether the user is able to move the body as specified in the training. Determine. As the user continues to exercise, it is expected that the muscle strength of each part of the body will gradually improve to improve the walking posture, and the user will be able to perform the movements required for exercise. Thereby, the terminal device 10 may determine that the user has completed the exercise menu, and may present another exercise menu to the user.

Furthermore, an exercise menu may be proposed to the user based on the evaluation results of the measurement results of the user's body movements and the history of exercise menus executed by other users of the same degree.

(4) In the above description of the first embodiment, an example has been described in which the movement of the user's body is measured and evaluated by installing the measuring device 31, the mat 32, and the pole 33 in the real world.

In addition, an object corresponding to the mat 32 and an object corresponding to the pole 33 may be placed in the virtual space, and the movement of the user's body may be measured. For example, assume that the user views the display 132 or wears a head-mounted display (HMD). The terminal device 10 displays a field-of-view image according to the settings of the virtual camera in the virtual space on the display 132 or the HMD. The terminal device 10 tracks the movement of the user's body by attaching motion sensors to each part of the user's body (head, shoulders, elbows, hands, hips, knees, feet, etc.). The user moves his or her body while visually recognizing each object installed in the virtual space. Thereby, the movement of the user's body can be measured and evaluated without installing the mat 32 and the pole 33 in the real world.

In addition to installing the mat 32 and pole 33 in the real world, it is also possible to irradiate an image corresponding to the mat 32 onto the floor, etc., and to place an image (such as a stereoscopic image) corresponding to the pole 33 in the real world. It may also be displayed in . For example, in the real world, the virtual mat 32 and pole 33 may be displayed so as to be visible to the user by irradiating light from a laser, a projector, or the like.

<Second embodiment>
A measurement system 1 in a second embodiment will be explained. In the second embodiment, the user's walking motion is measured and the walking posture is evaluated, and thereby the user is supported to acquire a walking style that is beautiful and allows the user to easily exert the functions of the body. I will explain about it.

FIG. 11 shows a situation in which a user's walking motion is measured by the measurement system 1 in the second embodiment.

<1 Overview>
The measuring device 31A measures the motion of the first user User_A walking from a predetermined position to the measuring device 31A in the detection range 38A. The terminal device 10A notifies the first user User_A on the display 132A that the walking state will be measured and how to move the body when walking. As shown in the figure, the terminal device 10A notifies the user to urge the user to walk so as to put his or her weight on the front foot during the walking motion.

In the example of FIG. 11, the coordinate axes are defined as follows. That is, with the position of the first user User_A as a reference, the direction of walking is defined as "front", the direction opposite to the direction of walking is defined as "back", and the axis determined by the direction of walking is defined as "front-back axis". On the other hand, with the position of the first user User_A as a reference, the direction perpendicular to the walking direction on the surface where the walking feet touch the ground is defined as the "left-right axis", and the direction of the user's right half of the body is defined as the "right direction". The direction of the left side of the body is defined as the "left direction." The axis in the torso direction from the feet of the first user User_A to the top of the head is defined as a "vertical axis."

The terminal device 10A notifies the first user User_A via the display 132A to walk toward the measuring device 31A starting from a position a predetermined distance away from the measuring device 31A. Here, the starting point from which the first user User_A starts the walking movement is, for example, a position several meters away from the measuring device 31A, and the walking cycle of the first user User_A (the movement of stepping with the right foot and the movement of stepping with the left foot) The starting point may be set at a position where it is possible to repeat (one cycle) a plurality of times. The measuring device 31A can measure the walking cycle of the first user User_A multiple times. When the terminal device 10A measures the user's walking motion using the measuring device 31A, if the first user User_A is not located at the starting point, the terminal device 10A sends a notification urging the first user User_A to move to the starting point. may be displayed on the display 132A or the like. In response to the presence of the first user User_A at the starting point, the terminal device 10A notifies the first user to start measuring the walking motion (or a notification prompting the first user to start the walking motion). It's okay.

FIG. 12 shows the parts of the user's body identified by the terminal device 10 based on the measurement results of the measuring device 31. The terminal device 10 identifies joints of extremities, head, shoulders, trunk, pelvis, feet, heels, toes, ribs, anterior superior iliac spine, etc. as characteristics of the user's body. do. The terminal device 10 identifies the thoracic vertebrae (the center of the trunk), the center of the pelvis, and the like. For example, the terminal device 10 prompts the user to take a predetermined pose (a pose in which both arms are stuck out to the side to form a T-shape, a pose in which both arms are attached to the torso, etc.), and the measurement device in these predetermined poses The position of each joint of the user in the measurement results of the measurement device 31 may be specified based on the measurement results of the measurement device 31.

<2 Data structure>
FIG. 13 is a diagram showing the data structure of the measurement result database 287 and the evaluation result database 288 stored in the server 20 in the second embodiment.

The measurement result database 287 is a database that holds the measurement results of each user's body movements (for example, walking movement, trunk balancing movement, etc.) measured by the measuring device 31.

Each of the records of the measurement result database 287 includes the item "measurement data ID", the item "date and time", the item "user ID", the item "elapsed time from the start of measurement", and the item "measurement data".

The item “measurement data ID” indicates information for identifying each piece of measurement data measured by the measurement device 31. In the second embodiment, the terminal device 10 starts and ends the measurement of the user's body movement (for example, the measurement of the walking cycle), and thereby associates information identifying each piece of measurement data with the measurement data. and save it on the server 20.

The item “date and time” indicates the date and time when the user's body movement was measured by the measuring device 31.

The item “user ID” is information that identifies each user whose body movement is measured by the measurement device 31.

The item “time elapsed since the start of measurement” indicates the time elapsed since the measurement device 31 started measuring the movement of the user's body.

The item "measurement data" indicates the position of each part of the user's body (head, shoulders, . . . ) as described in FIG. 12 at the timing indicated in the item "elapsed time from start of measurement." For example, in the item "measurement data", measurement results obtained by measuring distances from the position of the measuring device 31 to each part of the user's body in a coordinate system based on the position of the measuring device 31 may be stored.

The evaluation result database 288 is a database that holds evaluation results obtained by evaluating the user's body movement (for example, walking movement) based on the measurement results of the measuring device 31 (measurement result database 287).

Each record of the evaluation result database 288 includes the item "evaluation result ID", the item "measurement data ID", the item "measurement object", and the item "evaluation result".

The item “evaluation result ID” is information that identifies each evaluation result obtained by evaluating the measurement results of the user's body motion by the measuring device 31.

The item "measurement data ID" is information that identifies each measurement result (measurement result database 287) of the measurement device 31 that is the target of evaluation.

The item "measurement target" indicates the content to be measured as a user's body movement. Examples of objects to be measured include the user's posture during walking motion, the balance of the user's trunk, and the like.

The item “evaluation result” indicates the evaluation result of the user's body movement based on the measurement result of the measuring device 31. For example, when evaluating a user's posture during walking, the evaluation targets include ``the user's step,'' ``whether the user is able to shift their weight to the front foot,'' and ``whether the user is able to shift their weight to the front foot.'' Are you ready?” and so on. Details will be described later. Further, when evaluating the balance of the user's trunk, evaluation targets include "are the positions of the shoulders, hips, and feet misaligned?" Details will be described later.

<3 Operation>
Hereinafter, a process in which the terminal device 10 measures the user's body movement and evaluates it based on the measurement results in the second embodiment will be described.

FIG. 14 is a flowchart showing a process in which the terminal device 10 measures and evaluates a user's walking motion in the second embodiment.

In step S1403, the terminal device 10 detects that the user is at a predetermined position within the measurement range of the measurement device 31 based on the measurement result of the measurement device 31.

In step S1405, the terminal device 10 outputs a guide (audio, video) to the user to make a walking motion. The terminal device 10 detects each joint of the user based on the measurement results of the measuring device 31, and records the position of each joint during the user's walking motion.

In step S1407, the terminal device 10 classifies the cycle of the user's walking motion into a plurality of phases based on the measurement results of the measurement device 31. The terminal device 10 determines an evaluation value for evaluating the walking motion according to evaluation items for each aspect of the user's walking motion.

In step S1409, the terminal device 10 outputs the measurement results of the measuring device 31 and the evaluation results of the walking motion, together with the ideal walking posture model, to the display 132 or the like. The terminal device 10 transmits the measurement results and evaluation results to the server 20.

In step S1453, the server 20 updates the measurement result database 287 and the evaluation result database 288 based on the measurement results and evaluation results of the user's walking motion.

<4 Output examples of measurement results and evaluation results>
FIG. 15 is a diagram illustrating an output example when the terminal device 10 outputs measurement results of measuring the user's walking motion and evaluation results of evaluating the user's walking motion based on the measurement results in the second embodiment. It is. In the illustrated example, the terminal device 10 displays these measurement results and evaluation results on the display 132. It is also possible to output the results.

FIG. 15A shows a situation in which the terminal device 10 prompts the measuring device 31 to walk a certain number of steps (for example, about several meters) in order to measure the user's walking motion. This corresponds to the processing in steps S1403 and S1405 in FIG.

On the display 132, the terminal device 10 displays on the display unit 132P1 a "notification urging the user to walk so as to put his/her weight on the front foot when stepping forward" when performing a walking motion. The terminal device 10 displays, on the display units 132P2 and 132P3, an image showing the identified position of each part of the user's body (each part of the body illustrated in FIG. 12) as the measurement result of the measuring device 31. The terminal device 10 connects each part of the user's body in a linear manner to display the measurement results of the user's body in a way that is easy for the user to visually recognize. In the example of FIG. 15, parts of the right side of the user's body are connected by dotted lines, and parts of the left side of the user's body are connected by solid lines. Thereby, the user can easily recognize whether the measurement result of the measuring device 31 is a movement of the right side of the user's body or a movement of the left side of the user's body.

The terminal device 10 displays, on the display unit 132P2, the results of specifying the position of each part of the user when viewed from the front of the user (anterior-posterior axis shown in FIG. 11) at any time based on the measurement results of the measuring device 31. .

The terminal device 10 displays, on the display unit 132P3, the results of specifying the position of each part of the user when viewed from the user's side (left-right axis shown in FIG. 11) at any time based on the measurement results of the measuring device 31. . That is, the terminal device 10 displays the measurement results by the measurement device 31 on the display units 132P2 and 132P3 in real time. This allows the user to obtain feedback on the measurement results of the user's own body movements in real time.

The terminal device 10 notifies the user to urge the user to perform a walking motion to approach the measuring device 31. Here, the terminal device 10 may display the measurement results as if the movement of the user's body was measured at a position a certain distance from the user's body on the display units 132P2 and 132P3. For example, the terminal device 10 generates a three-dimensional virtual section, and based on the measurement results of the measuring device 31 (that is, the distance from the position of the measuring device 31 to the position of each part of the first user User_A), the terminal device 10 generates a three-dimensional virtual section. Objects corresponding to the positions of each part of the user's body are placed in the virtual space. For example, the terminal device 10 specifies the coordinates corresponding to the measuring device 31 in the three-dimensional virtual space, and determines the position of each part based on the measurement results of the measuring device 31 with respect to the coordinates corresponding to the measuring device 31. Place objects corresponding to each part at the corresponding coordinates. The terminal device 10 holds information on the position of the object corresponding to each part in the three-dimensional virtual space in association with the elapsed time accompanying the measurement by the measuring device 31. That is, by reproducing the locus of movement of the object corresponding to the position of each part of the user in the three-dimensional virtual space, the user's walking motion can be reproduced as the locus of the position of each part of the user. The terminal device 10 determines coordinates that have a certain positional relationship (for example, a position a certain distance from the "shoulder center" on the front-back axis, or a chest The virtual camera may be set at a position a certain distance from the waist on the left-right axis) and moved in conjunction with the movement of objects corresponding to the positions of each body part.

The terminal device 10 notifies information for appropriately performing measurement by the measuring device 31 on the display section 132P4. This corresponds to step S1403 in FIG. For example, when measuring the user's walking motion using the measuring device 31, the terminal device 10 determines whether the user starts walking at a predetermined position and prompts the user to start walking from the predetermined position. Give notice. As shown in the figure, if the position at which the user starts walking is too close to the measuring device 31, the terminal device 10 issues a notification urging the user to start walking from further back.

FIG. 15(B) shows a situation in which the terminal device 10 outputs measurement results and evaluation results regarding a certain aspect of the user's walking motion. This corresponds to the process of step S1409 in FIG.

In FIGS. 15(B) and 15(C), the measurement results of the user's body are displayed with the vertical axis as a reference. Thereby, it is possible to present to the user how the user's body is able to rotate with respect to the vertical axis.

The terminal device 10 outputs, on the display 132, the evaluation results of the balance of the rotation of the shoulders and the balance of the rotation of the hips during the walking motion, as the evaluation results of the beauty of the user's walking posture. The terminal device 10 divides the walking motion into a plurality of phases, and outputs evaluation results for each phase, such as the balance of shoulder rotation and the balance of hip rotation. In this way, by presenting the user with measurement results and evaluation results regarding multiple aspects of walking motion, it is possible to provide suggestions for making the user's walking motion more beautiful.

The terminal device 10 displays, on the display unit 132P5, the inclination of the rotation of the shoulder in each of the left rotation and the right rotation in a walking motion cycle (one cycle is an action of stepping with the right foot and then stepping with the left foot).

The terminal device 10 displays, on the display unit 132P6, a "line connecting both shoulders" determined by the positions of the right shoulder and left shoulder, a "line connecting both hip joints" determined by the positions of both hip joints, and the center of the positions of the right shoulder and left shoulder. The "center position of the shoulder" and the "center position of the waist" determined by the position of the hip joints (for example, the positions of the right pelvis and left pelvis) are displayed. In one cycle of walking motion, the terminal device 10 determines the degree of left rotation of the shoulder (with respect to the vertical axis) based on the position of the left shoulder (or both shoulders) when the user takes a step with the left foot and the position of the hip joint. Assess how far the shoulders are rotating. For example, the terminal device 10 draws a "line connecting both shoulders" based on the amount by which the shoulders rotate due to counterclockwise rotation in one cycle with the position of the waist as a reference. The terminal device 10 displays the score (score "4.8 points") as the evaluation result on the display unit 132P6.

The terminal device 10 displays a "line connecting both shoulders", "a line connecting both hip joints", "center position of shoulder", and "center position of waist" on display unit 132P7. In one cycle of walking motion, the terminal device 10 calculates the degree of left rotation of the shoulder with respect to the vertical axis based on the position of the right shoulder (or both shoulders) when the user takes a step with the right foot and the position of the hip joint. Evaluate. The terminal device 10 draws a "line connecting both shoulders" based on the amount by which the shoulders rotate due to clockwise rotation in one cycle with the position of the waist as a reference. The terminal device 10 displays a score as an evaluation result (score "4.5 points") on the display unit 132P7, which is a lower score than the left rotation.

The terminal device 10 displays, on the display units 132P6 and 132P7, the "center position of the shoulder" and the "center position of the waist" in the left foot stepping motion and the right foot stepping motion in one cycle of walking motion. This allows the user to recognize how the upper body is tilted with respect to the position of the waist during walking. For example, the user can be made aware of the degree to which the user's body is moving in the left-right axis and the degree to which it is moving in the front-back axis when the user is walking. In the illustrated example, in the display part 132P6, the degree of overlap between the "center position of the shoulder" and the "center position of the waist" during left rotation is large, and the upper body is not shaken, whereas in the display part 132P7, When rotating to the right, the overlap between the "center position of the shoulders" and the "center position of the waist" is relatively small, and the upper body is swayed laterally more than when rotating to the left. Therefore, the score that is the evaluation result is lower when rotating to the right than when rotating to the left.

FIG. 15C shows a phase in which the terminal device 10 outputs measurement results and evaluation results when the user's front foot touches the ground during the walking motion, as a phase of the user's walking motion. This corresponds to step S1409 in FIG.

The terminal device 10 displays, on the display unit 132P8, the measurement results and evaluation results of the inclination of the hip rotation when the user puts his or her front feet on the ground during walking motion.

The terminal device 10 displays, on the display unit 132P9, a "line connecting both hip joints" when the left foot is placed on the ground as the front foot. As shown in the figure, since the "line connecting both hip joints" is tilted, it is shown that the user is able to rotate his or her hips when the user puts his left foot on the ground.

The terminal device 10 displays, on the display unit 132P10, a "line connecting both hip joints" when the right foot is placed on the ground as the front foot. As shown in the figure, the "line connecting both hip joints" is slanted, which indicates that the user is able to rotate his or her hips when the user puts his right foot on the ground.

FIG. 16 is a diagram illustrating an output example when the terminal device 10 outputs measurement results of measuring the user's walking motion and evaluation results of evaluating the user's walking motion based on the measurement results in the second embodiment. It is. This corresponds to the processing in steps S1407 and S1409 shown in FIG.

In the example of FIG. 16, the terminal device 10 displays the measurement results and evaluation results of one cycle of the user's walking motion in a manner that can be compared with a reference example (model example, role model) for making the walking posture look beautiful. The output example shown in FIG. 16 may also be output by the terminal device 10 by printing it on a paper medium using a printer or the like.

On the display 132, the terminal device 10 divides one cycle of walking motion into a plurality of steps, and in each step displays a model for making the walking posture look beautiful, a measurement result of the user's walking motion, and an evaluation result for the measurement result. and is displayed.

In the example of FIG. 16, one cycle of walking motion is divided into the first step "stepping forward", the second step "trying to put your weight on the front foot that you have stepped forward (trying to get on the front foot)", and the second step "stepping your foot forward". It is divided into step 3, ``Put all your weight on the front foot that you took the step with (get over the front foot)''.

Here, the items evaluated by the terminal device 10 in the first step, second step, and third step include items that are common in each of these steps and items that are not common in each step.
(1) Evaluation items for the first step - Tilt of the upper body during walking: Whether the parts of the upper body (head, shoulders, etc.) are tilted forward or backward relative to the position of the waist. For example, is the angle of inclination of the line determined by the position of the pelvis, the position of the thoracolumbar region, and the position of the center of the shoulder within a certain range with respect to the vertical axis?
・The distance between the front and back of the feet when trying to step forward: The extent to which the front and back of both feet are opened. For example, is the distance between the ankle of the front foot and the ankle of the rear foot greater than a certain distance?
・The position of your hips when you try to step forward: Are your hips left behind when you try to take a step forward? For example, is the position of the pelvis anterior or posterior relative to the shoulders and thorax?
(2) Evaluation items for the second step・Tilting of the upper body when the front feet touch the ground: Are the parts of the upper body (head, shoulders, etc.) leaning forward or backward relative to the position of the waist?・Bend of the knee when the front foot touches the ground: Are each part of the front foot (ankle, knee, pelvis) bent significantly or not? For example, is the angle formed by the line connecting the ankle and knee and the line connecting the knee and pelvis within a certain angle? (3) Third step evaluation item The inclination of the upper body at the time of placing the weight on the front foot: Is the upper body part (head, shoulders, etc.) tilted forward or backward relative to the position of the waist? - The extension of the pivot leg when the weight is placed on the front foot: Stepping When you place your weight on the front foot that you landed on, do the parts of the pivot foot (ankle, knee, pelvis) extend or bend? For example, whether the angle formed by the line connecting the ankle and the knee and the line connecting the knee and the pelvis is within a certain angle, specifically, the terminal device 10 displays the following on the display unit 132P11. Display a model for the first step. The terminal device 10 displays the measurement results of the user's body in the first step on the display unit 132P21. The terminal device 10 displays the evaluation result of the user's body movement in the first step on the display unit 132P31.

The terminal device 10 displays, on the display units 132P11, 132P12, and 132P13, the position of each body part (joint) when each of these evaluation items is favorable for a beautiful walking posture. For example, in the display section 132P11, if the upper body is not tilted, the front and hind legs are spread apart to a certain extent (the stride length is not too short), and the hip position is appropriate (not too far back). The location of each part of the body is shown.

In the illustrated example, the display section 132P31 displays, as evaluation results, the evaluation result RF_ic of the right foot when stepping with the right foot and the evaluation result LF_ic of the left foot when stepping with the left foot for the evaluation item "Inclination of the upper body". It is displayed superimposed on a predetermined bar. The predetermined bar is drawn according to the possible range of the evaluation result. The terminal device 10 distinguishes and displays areas with good evaluation results and areas with poor evaluation results in the predetermined bar, and in the illustrated example, emphasizes and displays areas with good evaluation results. are doing. The terminal device 10 sets the evaluation result RF_ic of the right leg and the evaluation result LF_ic of the left leg in the evaluation item "Tilting of the upper body", in which the evaluation results are in a good range, with neither leaning too much forward nor leaning too far back. are displayed in association.

The terminal device 10 also associates the evaluation results of the right foot and the left foot with a predetermined bar for the evaluation item "front and back spread of the legs" and the evaluation item "hip position" as evaluation results on the display unit 132P31. is displayed.

The terminal device 10 displays a model for the second step on the display unit 132P12. The terminal device 10 displays the measurement results of the user's body in the second step on the display unit 132P22. The terminal device 10 displays the evaluation result of the user's body movement in the second step on the display unit 132P32.

The terminal device 10 displays a model for the third step on the display section 132P13. The terminal device 10 displays the measurement results of the user's body in the third step on the display unit 132P23. The terminal device 10 displays the evaluation result of the user's body movement in the third step on the display unit 132P33.

FIG. 17 is a diagram illustrating an example in which the terminal device 10 outputs an evaluation result of the burden on the knee during the user's walking motion. This corresponds to step S1409 in FIG.

FIG. 17(A) shows an example in which measurement results and evaluation results of hip joint adduction and abduction in the user's walking motion are output. The terminal device 10 displays the measurement results of the user's walking motion on the display 132 based on the front-back axis (as viewed from the front of the user), so that the user can easily check the bending of the knee during the walking motion. do it like this.

The terminal device 10 displays the measurement results during walking motion on the display section 132P41. The terminal device 10 displays, on the display unit 132P42, the measurement results when walking with the right foot as the pivot foot. The terminal device 10 displays, on the display unit 132P43, the measurement results when walking with the left foot as the pivot foot.

On the display unit 132P44, the terminal device 10 displays a pattern of preferable measurement results ("Both pivot and swing legs are straight") and a pattern of measurement results that may hurt the knee ("Knee ``Possibility of hurting the outside of the knee'' and ``Possibility of hurting the inside of the knee.'' In other words, if the hip joint is internally rotated too much or externally rotated too much during walking, there is a risk of knee pain during walking.

As described above, by referring to the output result of the terminal device 10, the user can be prompted to walk without hurting his or her knees.

FIG. 17(B) is a diagram showing an example of outputting an evaluation result as to whether the knee is too far ahead of the ankle in the user's walking motion. The terminal device 10 displays the measurement results of the user's walking motion on the display 132 based on the left-right axis (as viewed from the side of the user), and checks whether the user is moving his or her knees too much in advance during the walking motion. Allow users to easily check the information.

The terminal device 10 displays the measurement results during walking motion on the display unit 132P51. The terminal device 10 displays, on the display unit 132P52, the measurement results when walking with the right foot as the pivot foot. The terminal device 10 displays, on the display unit 132P53, the measurement results when walking with the right foot as the pivot foot. The terminal device 10 displays, on the display unit 132P51, "the front load on the knee is small" as the evaluation result of the user's walking posture.

On the display unit 132P54, the terminal device 10 displays a pattern of preferred measurement results ("good") and a pattern of measurement results that may hurt the knee ("knees are too far forward") for the user's reference. ) and display. In other words, if the user tries to take a step with the front foot and the knee is too far ahead of the ankle, the user may injure the knee. The terminal device 10 determines whether the angle at which the line connecting the ankle position and the knee position and the line connecting the knee position and the pelvis position intersect is greater than or equal to a predetermined angle (if the angle from the knee to the ankle is forward). When the user tries to take a step forward, it is evaluated whether the knee is moving too far in front of the user.

As described above, the walking motion of the user is evaluated regarding multiple aspects of the user's walking cycle, such as when the user tries to take a step with the pivot foot, when the pivot foot lands, and when the user tries to put weight on the pivot foot. Thereby, the user is prompted to make his walking motion more beautiful by referring to the evaluation results of each aspect. Furthermore, by allowing the user to refer to the model body movement and the measurement results of the user's body movement, the user is encouraged to approach a beautiful walking posture.

<Third embodiment>
A measurement system 1 in a third embodiment will be explained. In the third embodiment, a process for displaying the measurement results of the balance of the user's trunk and enabling the user to obtain a state in which the trunk is balanced will be described.

<Operation>
Hereinafter, in the third embodiment, a process in which the terminal device 10 measures the movement of the user's body and evaluates it based on the measurement results will be described.

FIG. 18 is a flowchart showing a process in which the terminal device 10 measures and evaluates the balance of the user's trunk in the third embodiment.

In step S1803, the terminal device 10 outputs a guide (audio, video) to measure the balance of the user's trunk (shoulders, hips, legs) based on the measurement results of the measuring device 31. The terminal device 10 displays the measurement results of the positions of the user's shoulders, hips, and feet by displaying them in circles based on the vertical axis. The terminal device 10 outputs a guide that prompts the user to balance his/her trunk, depending on how the positions of the user's shoulders, hips, and feet overlap.

In step S1805, the terminal device 10 evaluates the balance of the user's trunk based on the measurement results of the measurement device 31 and outputs an evaluation value. The terminal device 10 transmits the measurement results of the measurement device 31 and the evaluation results to the server 20.

In step S1853, the server 20 updates the measurement result database 287 and the evaluation result database 288 based on the measurement results and evaluation results of the user's trunk balance.

<Output example>
FIG. 19 is a diagram illustrating an example in which the terminal device 10 outputs a measurement result of measuring the balance of the user's trunk and an evaluation result of the balance of the trunk in the third embodiment.

FIG. 19(A) shows an example of a screen in which the terminal device 10 outputs measurement results in real time in order to measure the balance of the user's trunk using the measuring device 31. This corresponds to the process of step S1803 in FIG. In the example of FIG. 19A, the display unit 132P61 displays circular figures (such as a perfect circle or an ellipse) corresponding to each of the shoulders, hips, and legs of the user's body.

The terminal device 10 displays the measurement results of each part of the user's body on the display unit 132P62 based on the anteroposterior axis.

The terminal device 10 displays, on the display unit 132P63, the positions of the shoulders, hips, and feet of the user's body based on the measurement results of the measuring device 31 based on the vertical axis. The terminal device 10 displays the measurement results of the measurement device 31 on the display section 132P63 at any time. Therefore, when the user's body shakes, the circular figures corresponding to the shoulders, hips, and legs do not overlap, causing a shift. The screen example shown in FIG. 19A shows that each part of the user's body is not aligned and shifted along the vertical axis. The user is prompted to move his or her body to balance the trunk while looking at the display on the display section 132P63 of the display 132.

The terminal device 10 provides a notification on the display unit 132P64 to urge the user to maintain a balanced state of the trunk. Based on the measurement results of the measuring device 31, the terminal device 10 specifies the movement of the user's body necessary to balance each part of the user's body. In the illustrated example, the head and feet are located on the back side of the user's waist. In other words, the position of the user's waist is in a state of protruding forward. The terminal device 10 displays, on the display unit 132P64, a prompt to adjust the inclination of the body in the front-back direction. This allows the user to easily recognize the body movements necessary to bring the trunk into a balanced state.

FIG. 19(B) shows a display example of measurement results in a state where the user's trunk is well balanced. In the display section 132P63, the user's shoulders, hips, and legs are lined up along the vertical axis, and the body trunk is balanced. The terminal device 10 notifies the user on the display unit 132P64 that the trunk is in a balanced state.

FIG. 19C shows a display example when the measurement of trunk balance is finished and the evaluation result is output. This corresponds to the process of step S1805 in FIG.

The terminal device 10 displays on the display unit 132P65 that the measurement of the trunk balance has been completed, and outputs an evaluation result based on the measurement result.

The terminal device 10 determines a training menu for adjusting the balance of the trunk as a training menu to propose to the user based on the evaluation result of the trunk balance. In the illustrated example, the terminal device 10 displays, on the display unit 132P65, the evaluation result of the user's trunk balance and training menu candidates to be proposed to the user according to the evaluation result. The terminal device 10 receives, on the display section 132P66, an operation for starting a training menu proposed to the user. The terminal device 10 reads the corresponding training menu in response to the user's input operation on the display unit 132P66.

The terminal device 10 receives a training menu designation from the user on the display unit 132P67. That is, the display unit 132P67 accepts input operations for starting other training, regardless of the training menu that the terminal device 10 proposes to the user based on the evaluation result of trunk balance.

The items described in each of the above embodiments may be combined.

<Fourth embodiment>
In the description of the fourth embodiment, the measurement results of the user's body movements include a ROM check (first embodiment), a walking posture check (second embodiment), and a measurement of trunk balance. (Third Embodiment) A technique for proposing a training menu suitable for a user will be described. By measuring and evaluating the user's body movements as described above, the terminal device 10 can specify which parts of the user's body should be trained, and perform training according to the identified results. , suggest a suitable training menu.

With this configuration, when adding new training menu content, it is possible to determine which part of the body the training menu content is suitable for, and whether it is appropriate for any concerns that may arise in the user's body. By associating the content with the user, content can be provided to the user according to the measurement results of the user's body movement, and content can be added even more easily.

(1) As described above, the terminal device 10 may propose an exercise menu to the user based on the measurement results and evaluation results at each step of the user's walking cycle. For example, the terminal device 10 compares a model body movement with the user's body movement (example in FIG. 16), and identifies muscle parts that the user would like to improve. For example, the server 20 may select an exercise menu that is expected to allow the user to take a large step if the front and rear opening of the foot is small in the evaluation results of the first step (stepping the foot forward) as the user's walking motion. This is specified by referring to the training item information 285. That is, the server 20 may store the proposed exercise menu in advance in association with the evaluation results of the evaluation items of each step of walking motion and the evaluation item of trunk balance.

(2) The terminal device 10 may propose a training menu to the user so that the left-right difference in the user's body movements can be resolved. As described above, the terminal device 10 evaluates the left-right balance of the user's body movements by checking the ROM and checking the walking posture. For example, the terminal device 10 evaluates the left-right balance of the rotation movement with respect to the vertical axis based on the evaluation results of the shoulder rotation and hip rotation during the user's walking movement. The terminal device 10 identifies the muscle region that is the cause of the asymmetrical movement of the user's body. For example, the terminal device 10 holds in advance information about muscle parts that are the cause of the left-right difference in the amount of shoulder rotation. As described above, the terminal device 10 can eliminate the balance difference between the left and right by associating and storing in advance the target of evaluation during walking motion and the information on the muscle parts that are the cause of the left-right difference. It is possible to identify the muscle parts that should be trained for this purpose.

(3) Based on the evaluation result of the user's body movement, the terminal device 10 predicts the state that may occur in the user's body in the future, and proposes a training menu to the user based on the prediction result. Good too.

For example, the terminal device 10 identifies (i) "stiff neck/shoulders", "sagging upper arms", "rolled shoulders, forward head posture", "lower back pain", " Items such as ``leg swelling'' and ``arm bulge'' are associated with (ii) information on the user's body parts and stored as predetermined information. Further, the terminal device 10 stores information on a training menu to be proposed as training in association with each of the above-mentioned problems that may occur in the user's body.

The terminal device 10 performs a ROM check and a walking movement check to identify muscle parts that are desired to be improved based on these measurement results and evaluation results. For example, in the ROM check, if the body is not well twisted, the terminal device 10 stores in advance which muscle parts are associated. For example, in the ROM check, the terminal device 10 stores muscle information in association with evaluation items for range of motion of shoulders, hips, and the like.

The terminal device 10 identifies troubles that may occur in the future by referring to the predetermined information, such as by identifying the part of the muscle that moves stiffly. In accordance with this identification result, the terminal device 10 determines a training menu to be proposed to the user in order to cope with future problems that may arise in the user's body.

As described above, information on each part of the muscle is associated with the evaluation items of balance during walking movement and torsion of the body, and information on each part of the muscle is associated with each item of trouble that may occur in the user's body. information is associated with it. Thereby, the terminal device 10 can propose a training menu customized to each user based on the evaluation result of the user's body movement.

<Data structure in the fourth embodiment>
FIG. 20 is a diagram showing the data structure of various data stored in the server 20 in the fourth embodiment.

In the measurement item information 282, the server 20 maintains, for each measurement item, the item "Possible future worries" and the item "Related muscle parts."

The item "Possible future troubles" includes information that identifies troubles that may occur in the user's body in the future (for example, tags such as "stiff shoulders" and "stiff neck") depending on the evaluation results of the measurement results for each measurement item. ).

The item "related muscle part" is information about a body part (muscle part) that is desirable to be trained depending on the evaluation result of the measurement result for each measurement item. In the item "Related muscle parts", information on body parts may be held in stages depending on the degree of the evaluation result.

In the training item information 285, the server 20 holds, for each training item, the item "content", the item "corresponding worries", and the item "related muscle part".

The item "content" indicates content (video, motion data that serves as a reference for the user's body movements, audio data, etc.) that is presented to the user when performing training.

The item "corresponding trouble" is information (for example, tags such as "stiff shoulders" and "stiff neck") that identifies troubles that may occur in the user's body in the future depending on the evaluation result for each training item.

The item "related muscle part" is information about a body part (muscle part) that is desirable to train depending on the evaluation result for each training item. In the item "Related muscle parts", information on body parts may be held in stages depending on the degree of the evaluation result.

The server 20 maintains the item "possible future troubles" in the evaluation result database 288. The item "Possible future troubles" is information (for example, tags such as "stiff shoulders" and "stiff neck") that specifies, for each evaluation result, troubles that may occur in the user's body in the future depending on the evaluation results.

As described above, the server 20 updates the evaluation result database 288 according to the evaluation result of the user's body movement. For example, in accordance with the measurement results of the ROM check (measurement item information 282), information about troubles that may occur in the user's body in the future can be specified. The server 20 refers to the evaluation result database 288 to identify information about possible future troubles about the user, and refers to the training item information 285 to extract training items to be proposed to the user. When extracting a plurality of training items to be proposed to the user, the server 20 sets priorities based on predetermined rules. For example, the server 20 may prioritize training items that can resolve problems corresponding to unfavorable evaluation results of the user's body movements, depending on the degree to which the evaluation results of the user's body movements are unfavorable.

<Screen example in the fourth embodiment>
FIG. 21 shows an example of a screen in which the terminal device 10 presents possible troubles to the user's body based on the measurement results and evaluation results of the user's body movements, and proposes an exercise menu corresponding to the troubles. be.

In FIG. 21A, the terminal device 10 displays the results of identifying troubles that may occur in the user's body based on the measurement results and evaluation results of the user's body movements.

The terminal device 10 displays, on the display section 132P71, a display prompting the user to take measurements every day via the terminal device 10, and displays, on the display section 132P72, conditions that may occur in the user's body in the future. In the illustrated example, the operation reception unit 132P73 displays the possibility of suffering from “stiff neck and shoulders” and, in response to the user's input operation to the operation reception unit 132P73, displays the message “Stiff neck and shoulders”. Display the training menu that corresponds to the . In addition, the terminal device 10 displays, in the operation reception unit 132P74, that the problem of “sagging upper arms” may occur, and in response to the user's input operation to the operation reception unit 132P74, the terminal device 10 displays “sagging upper arms”. Display a training menu that addresses your concerns. In addition, the terminal device 10 responds to the user's input operation to the operation reception unit 132P75 while displaying that troubles such as “rolled shoulders and forward head posture” may occur in the operation reception unit 132P75. to display a training menu that addresses concerns about ``rolled-in shoulders and forward-headed posture.''

The terminal device 10 displays, on the display section 132P76, a message for starting training menus that are smaller in number than those displayed on the display section 132P76 among the training menu candidates proposed to the user for users who do not have time. The operation reception section 132P77 is displayed. Based on predetermined rules, the terminal device 10 sets priorities and the like for training menu candidates to be proposed to the user, and displays them on the display unit 132P76.

The terminal device 10 receives, on the display unit 132P78, an operation from the user to edit a training menu desired to be executed, such as "favorites".

The terminal device 10 presents a questionnaire to the user by accepting an input operation to the operation receiving unit 132P91, and receives responses to the questionnaire. The terminal device 10 may identify the user's concerns based on the input results of the questionnaire, and propose a training menu corresponding to the identified concerns.

FIG. 21(B) shows a situation in which a recommended training menu is proposed to the user. The terminal device 10 displays, on the display unit 132P81, information regarding body parts (such as "Are you suffering from stiff neck and shoulders?") regarding the troubles identified based on the measurement results and evaluation results to the user. . The terminal device 10 displays, on the display unit 132P81, the cause of the user's body-related worries (such as "the position of the shoulders tends to protrude forward") based on the measurement results and evaluation results. The terminal device 10 displays, on the display unit 132P81, guidelines for resolving the user's physical problems ("It is expected to improve the blood flow in the upper body", "Stretching the back of the thighs") based on the measurement results and evaluation results. ” etc.) and suggest exercises corresponding to the guidelines.

For example, in response to an input operation to the operation reception section 132P73, the terminal device 10 displays operation reception sections 132P83 to 132P87 for starting training based on a recommended training menu, as shown on the display section 132P82. The terminal device 10 may play back a video for starting training in response to input operations to these operation receiving units 132P83 to 132P87.

In response to the input operation to the operation reception unit 132P77, the terminal device 10 displays the operation reception unit with the number of training menus to be proposed narrowed down for users who do not have time, as shown on the display unit 132P88. 132P83 to 132P85 are displayed.

Thereby, the user can be encouraged to continue training depending on the user's current state.

<Operation example>
The operation of each device in the above fourth embodiment will be explained.

FIG. 22 shows the processing flow of each device in the fourth embodiment.

In step S2203, the terminal device 10 evaluates at least one of the user's joint range of motion, walking posture, and trunk balance based on the measurement results of the user's body movements, and outputs the evaluation result. The terminal device 10 transmits the measurement results and evaluation results to the server 20.

In step S2253, the server 20 updates the database (such as the evaluation result database 288) based on the user's various measurement results such as the measurement results and evaluation results received from the terminal device 10.

In step S2255, the server 20 identifies physical problems that may occur to the user in the future based on the evaluation result database 288 and the like. Based on the training item information 285, the server 20 selects a training menu related to the body part corresponding to the identified problem from among the plurality of training menus, and transmits the selected training menu to the user's terminal device 10.

In step S2205, the terminal device 10 displays a training menu corresponding to physical problems that may occur in the user's future.

<Modified example>
In the fourth embodiment, a question may be presented to the user, and the priority of the training menu to be proposed to the user may be set according to the answer (questionnaire result) received from the user.

This not only proposes a training menu based on the measurement results of the user's body, but also proposes a training menu based on the results of presenting questions about the user's body and receiving answers to the questions from the user. be able to. That is, the server 20 updates the priority of the training menu proposed to the user based on the result of the answer to the question. The server 20 presents the user with a training menu that has a high priority for the user, based not only on the body evaluation results but also on the user's answers to questions. Thereby, the server 20 can propose a training menu that is more suitable for the user's physical condition.

For example, by accepting a questionnaire about the body from the user, the terminal device 10 can determine which muscles of the user's body have not yet been measured by the measurement unit 30 in the measurement system 1 (which have not been evaluated based on the measurement results). Information about the body part can be received from the user. Based on the questionnaire results input by the user, the terminal device 10 can also identify problems that may occur in the future regarding muscle parts that have not yet been evaluated in the measurement system 1. The terminal device 10 makes suggestions to the user to appropriately deal with future worries that may arise in the user's body, based on the worries identified based on the measurement results of the body movement and the answers to the questionnaire from the user. Decide on a training menu.

<Data structure in modified example>
FIG. 23 is a diagram showing the data structure of various data stored in the server 20 in a modification of the fourth embodiment.

In the user questionnaire information 289, the server 20 stores, for each questionnaire result, the item "answer options," the item "current worries," the item "possible future worries," and the item "related muscle parts." Hold.

The item “answer options” includes information on options for the user to respond to the question content (questionnaire) presented to the user of the terminal device 10. For example, if the question is such that the user feels physically unwell, the user is presented with options for parts of the body that are likely to be unwell. In this case, it is possible to identify the user's concerns regarding body parts according to the results specified by the user for each option.

The item "Current worries" contains information that specifies the physical troubles that the user is currently having (for example, "lower back pain", "knee pain"), depending on the evaluation results of the questionnaire responses. tags such as "pain").

The item "Possible future troubles" contains information that identifies troubles that may occur in the user's body in the future (for example, "stiff shoulders", "stiff neck", etc.) depending on the evaluation results of the questionnaire responses. tag).

The item "related muscle part" is information about a body part (muscle part) that is desirable to be trained depending on the evaluation result of the questionnaire response contents for each questionnaire result. In the item "Related muscle parts", information on body parts may be held in stages depending on the level of the answer.

The server 20 maintains the item "current worries" in the evaluation result database 288. For each evaluation result, the item "Current worries" contains information that specifies the physical troubles the user may be currently having (for example, "lower back pain", "knee pain", etc.), depending on the evaluation results of the questionnaire responses. tags such as "pain").

<Operation in modified example>
As described above, the server 20 updates the evaluation result database 288 according to the evaluation results of the user's body movements and the user questionnaire information 289. For example, in accordance with the measurement results of the ROM check (measurement item information 282), information about troubles that may occur in the user's body in the future can be specified. For example, in accordance with the results of a questionnaire about the user's body (user questionnaire information 289), it is possible to identify physical concerns that the user is currently having or is likely to have in the near future. The server 20 refers to the evaluation result database 290 to identify information on possible future problems for the user, and refers to the training item information 285 to extract training items to be proposed to the user.

When extracting a plurality of training items to be proposed to the user, the server 20 sets priorities based on a predetermined rule, depending on the degree to which the evaluation results of the user's body movements are unfavorable. For example, the server 20 may prioritize training items that can resolve problems corresponding to unfavorable evaluation results of the user's body movements, depending on the degree to which the evaluation results of the user's body movements are unfavorable. The server 20 modifies and sets the priorities of training items to be proposed to the user based on the questionnaire results. For example, the server 20 selects training items that can resolve the user's physical concerns identified from the questionnaire results, depending on the degree to which the physical evaluation results related to the contents of the questionnaire responses are not better than the evaluation results of the ROM check. may be given priority.

FIG. 24 is a diagram showing the flow of operations in a modification of the fourth embodiment.

As shown in FIG. 24, the terminal device 10 presents a question (questionnaire) to the user of the terminal device 10 at an arbitrary timing such as step S2202 or step S2204.

The process of step S2202 will be explained. The terminal device 10 displays the question on the display 132, for example, in response to the user completing the process of logging into the measurement system 1 in step S700. The terminal device 10 receives input of answers to questions from the user. The terminal device 10 receives the input and transmits the response result to the server 20 .

In step S2252, the server 20 updates the user questionnaire information 289 based on the information received from the terminal device 10.

In step S2253, the server 20 stores various databases such as the measurement result database 287 in response to receiving the measurement result by the measurement unit 30 etc. in step S2203 and the evaluation result based on the measurement result from the terminal device 10. Update.

The process of step S2204 will be explained. For example, the terminal device 10 presents a question to the user (eg, displays it on the display 132) in response to completing the measurement of the user's body movement in step S2203. The terminal device 10 receives the input and transmits the response result to the server 20 .

In step S2254, the server 20 updates the user questionnaire information 289 based on the information received from the terminal device 10.

In step S2256, the server 20 identifies the user's current concerns and physical concerns that may occur to the user in the future, based on the user questionnaire information 289, the evaluation result database 288, and the like. Based on the training item information 285, the server 20 selects a training menu corresponding to the identified problem from among the plurality of training menus, and transmits the selected training menu to the user's terminal device. For example, a user's concerns are identified based on the user's answers to a question (user questionnaire information 289), and a training menu to be proposed to the user is specified according to the identified concerns (training item information 285). . Further, the server 20 specifies a training menu to propose to the user based on the evaluation result database 288 as described above. The server 20 responds to the terminal device 10 regarding the one or more training menus identified in this manner. At this time, the server 20 may set priorities for the plurality of specified training menus based on a predetermined rule, and may respond to the terminal device 10 with the training menus according to the priorities.

<Screen example 1 in the fourth embodiment>
FIG. 25 shows a situation in which the terminal device 10 presents possible troubles to the user's body based on the measurement results and evaluation results of the user's body movements, and proposes an exercise menu corresponding to the troubles. This is an example of a screen when responding to a questionnaire. For example, the terminal device 10 displays the screen (display unit 132P71) shown in FIG. 21(A) by accessing the server 20 or the like.

The terminal device 10 determines the timing at which the screen is displayed (the timing at which information for displaying the screen is received from the server 20 and displayed on a browser, etc.), or when a predetermined period of time has elapsed since the screen was displayed. In response to meeting a predetermined condition such as timing and the user performing a predetermined operation on the screen, the display unit 132PA2 shown in FIG. 25(A) is displayed to prompt the user to respond to the questionnaire.

FIG. 25(A) shows a screen on which the terminal device 10 displays the results of identifying troubles that may occur in the user's body based on the measurement results and evaluation results of the user's body movements (FIG. 21(A) A display section 132PA2 for the user to respond to the questionnaire is displayed in an overlay manner on the display section 132P71) described above.

The terminal device 10 displays, on the display section 132PA2, a question to the user ("What kind of problems do you have with your body?") on the display section 132PA3.
The server 20 determines the content of the question text to be presented to the user on the display unit 132PA3. For example, the server 20 holds a plurality of question sentences in advance, and holds information about the user's body parts and information about candidate answers to the question sentences in association with each question sentence. The server 20 refers to the measurement result database 287 and the evaluation result database 288, determines a question to be presented to the user based on the history of measurements performed by the user, and displays the determined question on the terminal device 10. It may also be a thing. For example, the server 20 determines whether a certain period of time has passed since the user last measured, based on the user's measurement history using the measurement system 1 (such as the time interval between measurements). The user may be presented with a question (such as "What kind of trouble do you have with your body?") asking whether or not the user is aware of any discomfort in each part.

Further, for example, the server 20 may determine the body parts to be included in the question text presented to the user based on the measurement results of the user's body parts. For example, if there is a part of the user's body that has not been measured for at least a certain period of time (for example, there is no history of measuring the waist, or if a certain period of time has passed since the last measurement) ), the relevant body part may also be included in the question (e.g., "How's your back feeling?").

In the example of FIG. 25(A), the terminal device 10 displays answer candidates for the question text on the display unit 132PA4, and accepts a selection from the user. In the illustrated example, the display unit 132PA4 displays "lower back troubles" for specifying lower back complaints, "knee troubles" for specifying knee complaints, and other body responses as answer candidates for the question text. "＊＊＊" is displayed to display possible answers regarding body parts. The terminal device 10 accepts the designation of "back problems" from the user, and highlights the candidate so that the user can easily recognize that the user has designated it.

The terminal device 10 displays, on the display unit 132PA5, a question regarding the degree of worry (the degree of discomfort in the body part).

The terminal device 10 displays, on the display section 132PA6, possible answers to the question text displayed on the display section 132PA5, and accepts selections from the user. In the illustrated example, the terminal device 10 displays the following options on the display unit 132PA6: "Mild" for a light level of worry, "Moderate" for a medium level of worry, and "Severe" for a heavy level of worry. ing. The terminal device 10 receives the designation of the option "moderate" from the user, and highlights the designated option.

The terminal device 10 receives, on the display unit 132PA1, an operation from the user indicating that the answer to the question text has been completed. Thereby, the terminal device 10 transmits the user's answer result to the server 20. The server 20 updates the user questionnaire information 289 by receiving the user's response results. For example, when the user specifies information related to a body part such as "lower back trouble" as the response result, the server 20 updates the user questionnaire information 289 based on the user's "lower back trouble".

FIG. 25(B) is an example of a screen in which a training menu is proposed to the user based on the results of receiving answers to questions from the user, the measurement results of each part of the user's body, and the evaluation results. .

Based on the answers received from the user to the question text (user questionnaire information 289), the server 20 provides information regarding concerns about the user's body parts (such as "It seems like the burden on your knees is moderate", etc.). information including the body part and the degree of worry) is displayed on the display section 132P81. The terminal device 10 displays, on the display unit 132P81, the cause of the user's physical concerns (such as "Are you in an uncomfortable posture for a long time?"). The terminal device 10 displays, on the display unit 132P81, a guideline for resolving the user's physical problems (such as "relax the muscles around the waist") and a proposal for an exercise corresponding to the guideline.

<Other variations>
(1) Asking the user questions about their exercise experience In the above embodiment, the server 20 presents a question to the user and proposes a training menu based on the answer to the question. explained. The server 20 may ask the user about the user's exercise experience as the question to be presented to the user.

For example, the server 20 may present a question to the user regarding his past experience playing sports and his exercise habits. The server 20 receives answers to the question from the user (such as sports the user participated in in the past, whether or not the user had a habit of exercising, etc.). Based on the user's answers, if the user has relatively little exercise experience (e.g., the user has played sports for less than a certain period of time, the habit of exercising is less than a predetermined frequency, etc.), the server 20 On the other hand, among the training items included in the training item information 285, training items that place a relatively small load on the body or training items that are relatively difficult may be suggested. For example, for each training item in the training item information 285, the difficulty level (training that can be performed without problems by a person who is not used to exercise, the lower the difficulty level, etc.), the load on the body (for example, the specific part (upper arm, etc.) ) holds information on the extent of the different loads placed on the body for each training method. Thereby, a training menu can be presented based on the user's exercise experience.

According to the above example, by proposing a training menu that is relatively easy to accomplish for a user who is not accustomed to exercise, the user can easily feel a sense of accomplishment by completing the training menu. Thereby, it is possible to provide motivation for the user to continue training.

(2) Evaluate whether the user is accustomed to exercise based on the measurement results of the user's body movements In addition to this, the server 20 evaluates the user's exercise experience based on the measurement results of the user's body movements by the measurement unit 30. The training menu to be presented to the user may be determined based on the evaluation results.

For example, in association with the measurement results of the user's body movements such as joint range of motion, walking posture, trunk balance, etc., the server 20 determines the pattern of measurement results when the user is not accustomed to exercise and the pattern when the user is accustomed to exercise. The pattern of measurement results is stored in advance. The server 20 evaluates the degree to which the user is accustomed to exercise based on the measurement results of the user's body movements. For example, the server 20 compares the above pattern with the measurement results of the user's body movements and evaluates the degree of agreement, thereby outputting an evaluation value indicating whether the user is accustomed to exercise. For example, the server 20 presents the user with a model body movement and evaluates that the user is accustomed to exercise by being able to move the user's body to the same extent as the body movement. At this time, if there is a discrepancy between the model body movement and the measurement result of the user's body movement, a voice may be output to the user to correct the discrepancy. For example, if the hand position differs from the model movement, a voice output prompting the user to make the hand position follow the model movement (such as "move the hand position in a predetermined direction") is displayed. Good too. In addition, if the user's body movement and the model movement match (that is, the user can perform a movement equivalent to the model movement), the user's body movement is appropriate. Output the information in audio, etc.

For example, when measuring the user's body movements, the server 20 determines the evaluation value of whether the user is accustomed to exercise based on whether or not the user is able to move the body according to a video or audio guide presented to the user. It may also be output. For example, the server 20 may evaluate that the user is accustomed to exercise when the user is able to move his or her body in accordance with the voice output.

The server 20 determines a training menu to propose to the user based on the evaluation value.

As described above, the server 20 maintains the history of measurement results by the measurement unit 30 in the measurement history database 283, and informs the user of the timing and evaluation results of whether or not the user is accustomed to exercise. A history of training menus proposed to the user may be displayed. Thereby, the user can confirm that he/she is gradually getting used to exercise and that the training menu has changed, and can feel a sense of self-efficacy.

(3) Asking the user a question about continuing the training In addition to this, the server 20 may ask the user a question about continuing the training. For example, the user may be asked questions regarding the user's personality, such as whether or not he or she has been able to continue training in the past, and how long it will take to stop training.

The server 20 determines a training menu to present to the user based on the results of the answers to the questions. For example, as described in the fourth embodiment and the like, assume that a problem that may occur in the user's body is identified, and a training menu related to the body part associated with the problem is presented to the user. If the user gives a negative answer as to whether or not he or she can continue the training (e.g., he or she has a tendency to not be able to continue the training), the server 20 changes the training menu proposed to the user according to the problem. It may also be a thing. For example, if there are multiple training menus related to body parts associated with the task, another training menu is suggested to the user. This is expected to make it easier for the user to engage in training without getting bored, allowing the user to continue training.

Based on the measurement history database 283 and the like, the server 20 identifies the interval between the timings at which the user performed training. That is, the frequency with which the user performs training is specified. The server 20 may present a question to the user regarding continuing training based on the length of the interval. For example, when the frequency of training is decreasing, by accepting questions from the user regarding continuing training, it is possible to propose another training menu so that the user can continue without getting bored. Furthermore, the server 20 may change the training menu proposed to the user based on the length of the interval. This allows the user to perform training without getting bored, and allows the user to continue training.

<Fifth embodiment>
<Summary>
In the fifth embodiment, a technique will be described in which a user can move his or her body and more easily check whether or not the user is performing ideal movements.

(1) By visually reducing the amount of information about body movements when the user moves, the body movements can be easily confirmed.
Specifically, when a user performs a specific exercise (for example, an exercise to evaluate trunk muscle movement or joint range of motion such as a lateral lift, or an exercise for strength training), Body movements are sensed as point cloud data, and based on the point cloud data, an avatar corresponding to the user and auxiliary lines corresponding to the user's body movements are drawn and presented to the user. This allows the user to more easily confirm whether each part of the body is moving as intended.

(2) Make it even easier for trainers to instruct trainees online using video conferencing tools, etc.
In the fifth embodiment, a trainee (first user) who receives instruction on how to move the body based on the above technology, and a trainer (second user) who instructs the trainee how to move the body, Explain the aspects of teaching online.

Specifically, by allowing the trainer to check an avatar that expresses the trainee's body movements as point cloud data on the trainer's terminal, the trainer can view not only the image taken of the trainee's body but also the relevant avatar. It is possible to check the movement of an avatar with a reduced amount of visual information compared to an image. This makes it even easier for trainers to give advice on how to move their bodies to trainees online.

In other words, in the fifth embodiment, (1) the first processor of the terminal of the trainee (first user) senses the movement of the trainee's (first user) body as point cloud data using the measurement unit 30 or the like; , (2) The first processor transmits information based on point cloud data of the trainee (first user) (for example, an image of the trainee's avatar drawn based on the point cloud data, or the point cloud data itself) to the trainer (first user). A series of techniques for executing various processes such as sending data to the terminals of two users will be described.

Here, the body movements of the trainee (first user) are sensed as point cloud data, and the first processor sends information based on the trainee's (first user) point cloud data to the trainer's (second user) terminal. However, examples of devices that execute various processes include terminal devices owned by users, such as computers, smartphones, and tablet terminals.

<Data structure in the fifth embodiment>
FIG. 26 is a diagram showing the data structure of a trainee information database 2601, a trainer information database 2602, and a training history database 2603 stored in the server 20 in the fifth embodiment.
Although not shown in the figure, the server 20 also performs various types of body movements performed by the user (movements for evaluating joint range of motion such as lateral lifts, movements for muscle strength training such as push-ups, (predetermined body movements such as yoga, other movement types), information on the type of auxiliary line when displaying the user's body movement as an avatar of a collection of point cloud data, and the auxiliary line. It manages a database that associates information on the drawing position (position associated with the user's body part represented by the avatar).

The trainee information database 2601 is a database for managing information on users who become trainees. The trainer information database 2602 is a database for managing information on users who become trainers. The training history database 2603 is a database for managing the history of training of trainees under the guidance of a trainer.

As shown in FIG. 26, the trainee information database 2601 includes the item "trainee ID", the item "name", the item "age", the item "gender", the item "weight", and the item "body fat percentage". , the item "purpose", the item "ideal figure", the item "screen display", the item "physical fitness level", the item "remarks", etc.

The item "trainee ID" is information that identifies each user receiving training.

The item "name" is information indicating the name of each user receiving training.

The item "age" is information indicating the age of each user receiving training.

The item "gender" is information indicating the gender of each user receiving training.

The item "weight" is information indicating the weight of each user undergoing training.

The item "body fat percentage" is information indicating the body fat percentage of each user undergoing training.

The item "purpose" is information indicating the purpose of each user receiving training. Specifically, the user identified by the name "Mori" with the user ID "U001" aims to "shape up", and the user identified by the name "Sato" with the user ID "U002" The purpose of training is to diet. Here, in a certain aspect, a user (trainer) who is in a position to instruct a user (trainee) undergoing training may present a training menu to the user (trainee) depending on the purpose. Also, in some aspects, the terminal device 10 or the server 20 may specify a training menu to be presented to the user based on a trained model, which will be described later.

The item "ideal figure" is information indicating the ideal figure of each user undergoing training. Specifically, the user identified by the name "Mori" with the user ID "U001" has an ideal figure of "not specified", and the user identified by the name "Sato" with the user ID "U002" "〇〇" is specified as the ideal figure. Here, the ideal figure may be an actual celebrity, or a fictional entity (for example, if the ideal body shape is that of a manga character, the user can specify that character). You can also use it as Furthermore, in some aspects, the server 20 may specify and present a training menu based on the "ideal figure" and the contents of the "purpose".

The item "screen display" is information indicating the display mode when the user is photographed with a camera or the like. As a display mode, when a user is photographed with an RGB camera, etc., the photographed image generated by the photographing is displayed as is, and a sensor capable of measuring distance (depth camera, LiDAR (light detection and ranging), etc.) is used. ) may measure the user's body movements as point cloud data and display them as an avatar based on the point cloud data. Specifically, the user identified by the name "Mori" with the user ID "U001" displays the screen as an "avatar", and when a video image of the user taken with a camera is displayed on the screen, This indicates that the user's appearance is not displayed as a photographed moving image as it is, but as some kind of avatar (for example, an avatar using a character) based on the photographed moving image.

This makes it possible to accommodate the requests of users undergoing training, such as not wanting others to see their face and body as it is their first time training, and wanting to protect their privacy, allowing users to receive training with peace of mind. I can do it.

The item "physical fitness level" is information indicating the physical fitness level of each user who undergoes training, which is a graded evaluation of the physical exercise ability of each user. Specifically, when sensing the body movements of a user undergoing training, the physical fitness level is used as a criterion for determining how much exercise the user is likely to be able to do, based on information about the user's body movements. has been identified. The user identified by the name "Mori" with the user ID "U001" has a physical fitness level of "5", and the user identified by the name "Sato" with the user ID "U002" has a physical fitness level of "2". be. At this time, as a method for identifying the physical fitness level, when the terminal device 10 or the server 20 senses the movement of the user's body, the physical fitness level of the user may be identified based on a learned model, or A user (trainer) who is in a position to instruct a user (trainee) receiving the training may specify the physical fitness level of the trainee based on the sensing results.

As a result, the user can receive a training menu that matches his or her physical fitness level, so that the user can continue training without difficulty. Furthermore, in the case of group training, the server 20 can identify users with the same physical fitness level and match users (trainers, trainees) so that the identified users can train together. For example, when training is carried out between users (trainees) who have different physical fitness levels, there will be differences in the content of the trainer's instruction and the training content of the trainees, which may reduce motivation. So that the physical fitness level can be prevented.

The item "remarks" is information indicating circumstances that should be specially considered for each user receiving training. Circumstances that should be taken into special consideration include matters that should be considered based on the safety of training (safety based on a medical perspective, safety of the training content itself, etc.). For example, if the user has been prevented from excessive training by a doctor due to high blood pressure, write this in the notes. The server 20 may specify the optimal training menu for the user undergoing training based on the content described in the item "remarks." In addition, the trainer's terminal device 10B acquires the information in the item "Remarks" from the server 20 and presents it to the trainee, thereby providing information that can be used as a reference when the trainer decides on a training menu for the trainee. Good too.

As a result, even if the user has a chronic illness, the server 20 etc. can specify the training menu to be presented to the user based on the relevant item "Notes", and the trainer can select a training menu based on the relevant item "Notes". This allows the user to continue training with peace of mind.

The trainer information database 2602 includes the item "Trainer ID", the item "Name", the item "Gender", the item "Years of experience", the item "Specialty area", the item "Recommended physical fitness level", and the item "Guidance". Includes "possible types" and items such as "remarks".

The item "Trainer ID" is information that identifies each user who is in a position to instruct users receiving training.

The item "Name" is information indicating the name of each user who is in a position to instruct the users undergoing training.

The item "gender" is information indicating the gender of each user who is in a position to instruct users receiving training.

The item "years of experience" is information indicating the number of years of training guidance experience of each user who is in a position to instruct users receiving training. Specifically, the user identified by the name "Tanaka" with the trainer ID "T001" has 5 years of experience, and the user identified by the name "Kato" with the trainer ID "T003". This indicates that the user identified by has "10 years" of experience.

As a result, for example, in a certain situation, when a user undergoing training (trainee) specifies a user (trainer) who is in a position to instruct the user, the user who is in the position to instruct the user (trainer) can be specified by referring to the number of years of experience. You can receive training with peace of mind. A user in a position of teaching (trainer) can also improve his or her popularity by being designated as a user in a position of teaching based on the information.

The item "specialty area" is information indicating the specialty area in training guidance of each user (trainer) who is in a position to instruct a user (trainee) receiving training. Specifically, the user identified by the name "Tanaka" with the trainer ID "T001" specializes in "yoga" and "exercise," and the user identified by the name "Saito" with the trainer ID "T002" indicates that his/her specialty is "muscle training" and "exercise."

As a result, the server 20 or the like can be used as a user (trainer) who is in a position to instruct the user, depending on the information such as the item "purpose" and the item "ideal figure" in the above-mentioned trainee information database 2601 of the user undergoing training. It may also be possible to specify. Thereby, the trainee can be matched with a trainer that matches the purpose of receiving training. For example, if a trainee's goal is to exercise, the server 20 or the like may assign a trainer who is good at training that matches the goal as an instructor.

The item "recommended physical fitness level" indicates the recommended physical fitness level of the trainee when receiving training from the user (trainer), which is defined for each user (trainer) who is in a position to instruct the user (trainee) receiving training. . Specifically, for instruction from a user (trainer) identified by the name "Tanaka" with trainer ID "T001", the recommended physical fitness level is set to "3 or higher", and the above-mentioned user ID "U001" Since the user (trainee) identified by the name "Mori" in ``Mori'' has a physical fitness level of ``5'', it can be seen that the user has reached the level of physical fitness to receive instruction from the user, and the user ID is ``U002''. Since the user (trainee) identified by the name "Sato" has a physical fitness level of "2", it can be seen that the physical fitness level has not reached the level required to receive instruction from the user.

Thereby, the user undergoing training can receive training that suits his or her physical fitness level. In addition, users who are in a position to instruct users undergoing training can also instruct training for users who are above a specified level, which can reduce the occurrence of variations in training menus and progress status among users. , it can also be expected to maintain the motivation of users undergoing training. The server 20 or the like may match the trainer and the trainee based on the trainer's item "recommended physical fitness level" and the trainee's item "physical fitness level".

The item "teachable type" indicates the type of training that can be taught by a user (trainer) who is in a position to instruct a user undergoing training. Specifically, the user identified by the name "Tanaka" with the trainer ID "T001" has the coachable types "individual" and "group", and is identified by the name "Saito" with the trainer ID "T002". The user indicates that the only instructable type is "individual".

This allows the server 20, etc., to determine whether the user (trainer) who is in the position of instructing the user undergoing training is suitable for individual instruction or group instruction, so the server 20 etc. In the matching process, training can be divided into one-on-one individual instruction training and one-on-multiple group instruction training. Furthermore, the trainer's "instructable type" may be presented to the trainee on the trainee's terminal device 10.

The item "remarks" is information such as other special notes for each user (trainer) who is in a position to instruct users undergoing training. Special notes may include information that objectively evaluates the training performance of the trainer. Specifically, it can be seen that the user identified by the trainer ID "T001" and the name "Tanaka" is "**Grand Prix Winner."

As a result, in a certain situation, when a user receiving training (trainee) specifies a user (trainer) who is in a position to instruct the user receiving training, information in the relevant item "Remarks" is sent to the trainee on the terminal device 10 of the trainee. By presenting the information, the trainee can designate a trainer while taking this information into consideration. Users (trainers) who are in a position to instruct users undergoing training can also expect to increase their chances of being designated as an instructor by recording this information.

The training history database 2603 includes the item "Training ID", the item "Date and time", the item "Meeting ID", the item "Trainee ID", the item "Trainer ID", the item "Sensing result", and the item "Guidance". content,” the item “trainer evaluation,” and the item “matching ID.”

The item "Training ID" is information that identifies each training by a trainer and a trainee.

The item "date and time" indicates the date and time when the training was performed.

The item "meeting ID" is information that identifies each communication session when online training guidance is provided by communication between the trainer's terminal device 10B and the trainee's terminal device 10A. For example, the information identifying the communication session includes information such as a URL for an online meeting.

The item "Trainee ID" is information that identifies the trainer who performed the training.

The item "Trainer ID" is information that identifies a trainee who has received training guidance from a trainer.

The item "sensing result" is the result (point cloud data) of the trainer's body movement measured by the measurement unit 30 by the terminal device 10A during training, or the point cloud data is placed in a virtual space and used to set the virtual camera. A video rendered based on the video (in the video, an avatar that expresses the trainee's body movements as a collection of point cloud data, and the way the trainee's body moves (the type of body movement presented to the user, such as lateral lift) (including auxiliary lines drawn accordingly).

The item "Instruction content" indicates the instruction content given by the trainer to the trainee regarding how to move the trainee's body. For example, when a trainer instructs a trainee how to move the body (including advice on how to move the body to guide the user to a desired posture) by voice, (advice from a trainer urging the user to take a desired posture when the arms, legs, etc.) are bent), audio containing the instruction content, or analysis results of the audio are stored in the item "Instruction content".

The item "trainer evaluation" indicates the evaluation that the trainer received from the trainee. For example, after training is conducted online, a screen for inputting the trainer's evaluation is presented to the trainee on the trainee's terminal device 10A, thereby accepting the trainer's evaluation from the trainee.

The item "matching ID" is information that identifies each matching when the server 20 matches a trainer and a trainee. Training between a trainer and a trainee may start when a match is made, or training may not be performed even though a match is made.

<Operation example in the fifth embodiment>
Hereinafter, in the fifth embodiment, the first processor of the terminal of the first user (trainee) in the system 1 senses the movement of the body of the first user as point cloud data, and the first processor senses the movement of the body of the first user as point cloud data. A series of processes when transmitting information based on the user's point cloud data to the terminal of the second user (trainer) and performing various processes will be described.

(Sensing the movement of the first user and transmitting it to the second user's terminal)
FIG. 27 is a flowchart showing a series of processes for sensing the movement of the first user's body as point cloud data and transmitting information based on the first user's point cloud data obtained through sensing to the second user's terminal. It is.

In the following description, regarding the terminal device 10 shown in FIG. 1, the first user's terminal device 10 will be referred to as a "terminal device 10A," and the configuration of the terminal device 10A will be described as a "processor 19A." Similarly, the second user's terminal device 10 will be described as a "terminal device 10B", and the configuration of the terminal device 10B will be described as a "processor 19B" or the like.

In addition, in the example of FIG. 27, by establishing communication between the first user's terminal device 10A and the second user's terminal device 10B, the video captured by the camera 160 of each terminal and the sound collected by the microphone 141 are transmitted. Send and receive audio to and from each other.

For example, the terminal device 10A and the terminal device 10B are running an application for holding an online video meeting, and identification information for participating in the meeting (e.g., meeting ID issued by the application, URL etc.) to establish communication.

The terminal device 10A acquires the movement of the first user's body as point cloud data using the measurement unit 30 or the like. The terminal device 10A arranges the point cloud data in the virtual space, and moves the virtual camera in the virtual space to the position of the front of the first user's body (anterior-posterior axis in the explanation of FIG. 11) and the side of the first user's body. (the left and right axis in the explanation of FIG. 11), rendering is performed and the rendering results are displayed. The rendering result includes an avatar as a set of point cloud data viewed from the front of the first user, and an avatar as a set of point cloud data viewed from the side of the first user. The terminal device 10A draws an auxiliary line corresponding to the user's body movement in association with the display area of the above rendering result (avatar viewed from the front, avatar viewed from the side).

The terminal device 10A displays an image in which the avatar and auxiliary lines are drawn on the screen, and shares the image with the terminal device 10B using an application for conducting an online video meeting. As a result, the video captured by the camera 160A of the first user, the audio captured by the microphone 141, and the image in which the avatar and auxiliary lines are drawn are transmitted from the terminal device 10A to the terminal device 10B. The second user of the terminal device 10B confirms the body movement of the first user by visually confirming the image captured by the camera 160A and the screen-shared image (the image in which the avatar and auxiliary lines are drawn). Then, the advice to the first user is notified to the first user by voice or the like.

In step S2701, the processor 19A of the first user's terminal device 10A uses the measurement unit 30 or the like to sense the movement of the first user's body as point cloud data.

Specifically, the processor 19A displays a menu of body movements performed by the first user to the first user, and allows the first user to make a specific movement (for example, side movement) for sensing the body movement. Encourage them to do movements such as lifting their legs, movements that are training items (squats, etc.). The processor 19A of the first user's terminal device 10A senses the movement of the first user's body as point cloud data by photographing the specific movement using the measurement unit 30 or the like.

In step S2703, the processor 19A generates an avatar representing the first user (trainee) and body movements performed by the first user as information based on point cloud data of the first user obtained through sensing by the measurement unit 30 or the like. (the type of exercise menu to be presented to the first user).

In step S2705, the processor 19A displays the video generated in step S2703 on the display 132A of the terminal device 10A. The processor 19A separates the displayed video (video including the avatar and auxiliary lines) and the photographed image generated by the camera 160A that photographs the first user (not a collection of point cloud data, but a photographed image taken with an RGB camera). (photographed image) is transmitted to the second user's terminal device 10B.

In step S2751, the processor 19B of the second user's terminal device 10B receives the video (avatar and auxiliary line) generated by the terminal device 10A from the first user's terminal device 10A as information based on the first user's point cloud data. ). The processor 19B receives the first user's photographed image (the photographed image that is not a collection of point cloud data) generated in step S2705.

In step S2753, the processor 19B generates a first avatar corresponding to the body movement of the first user as a set of point cloud data based on the video (including the avatar and auxiliary lines) received in step S2751 and the photographed image. A video including the above is displayed on the display 132B or the like. The processor 19B displays the first user's captured image (not a set of point cloud data) on the display 132B or the like. Thereby, the first avatar and the captured image of the first user are presented to the second user as information about the first user's body movement.

Here, in step S2703, the processor 19A specifies various information regarding the first user's exercise (which parts of the body are easy to move, which parts are difficult to move, etc.), physical strength level, etc. based on the trainee information database 2601 of the server 20, etc. You can also do it. At this time, the processor 19A may store the identified information in the storage unit 180 of the first user's terminal device 10A, or may transmit the information to the server 20 and store it in the trainee information database of the storage unit 202 of the server 20. 2601 may be stored.

Furthermore, the processor 19A refers to the trainer information database 2602 based on the results of sensing the body movement of the first user using the measurement unit 30, etc., and identifies the second user (trainer) to whom the information is to be transmitted. You may.

Thereby, a user who is a trainee can receive training from an optimal instructor based on information about body movements at the time of starting training.

Here, in step S2751, data sensed by the measurement unit 30 or the like is received from the first user's terminal device 10A, and based on the received information, the terminal device 10B performs rendering based on the point cloud data. You can also use it as That is, the first avatar representing the movement of the first user's body may be rendered in the terminal device 10B instead of the terminal device 10A based on the point cloud data measured by the terminal device 10A. The terminal device 10B generates a video including the first avatar, and displays the generated video on the display 132B or the like. At this time, the movement of the first user's body may be drawn from a plurality of viewpoints, and a video in which the first avatars drawn for each viewpoint are arranged may be presented to the second user.

In one aspect, the terminal device 10B draws an auxiliary line at a position based on a body part that evaluates whether the body movement performed by the first user is ideal, and displays the video with the auxiliary line drawn on the second user. It may be presented to the user. For example, when a trainer evaluates whether a user's body movements are ideal based on whether the user's back muscles are straight or whether the knees are at a predetermined angle to the ground, , the terminal device 10B displays an auxiliary line corresponding to the shape of the avatar's back muscles in the vicinity of the avatar's back muscles. When evaluating whether the knees are at a predetermined angle with respect to the ground, the terminal device 10B displays information near the avatar's knee portion (or in an area where the knee portion is displayed) at the predetermined angle. Display an auxiliary line near the avatar's knees.

In addition, the display mode of body surface positions other than joint positions was changed based on the body part used to evaluate whether or not the body movement performed by the first user was ideal, and the display mode of the body surface was changed. The video may be presented to the second user.

The surface of the body that is not a joint position is, for example, a surface that can be recognized as the external appearance of the body, such as the surface of the abdomen. For example, in the terminal device 10B, the displayed color may be changed depending on whether the first user's body movement is too large or small compared to the ideal movement. For example, if the abdomen is moving too far forward than the ideal movement, the abdomen is displayed in a warm color, and if the movement of the abdomen is smaller than the ideal movement, the abdomen is displayed in a cool color. , change the displayed color. Thereby, the first user or the second user can easily confirm how to move the user's body to achieve the ideal movement. Furthermore, this allows the user to train while understanding how far his or her body movement deviates from the ideal movement. Furthermore, by changing the display mode of the body surface, it is possible to grasp the movement of muscles on the body surface other than joints.

Here, in the above example, regarding the video including the first user's first avatar displayed on the display 132B etc., the first user's photographed image taken when generating the first avatar and the first avatar's photographed image are used. The explanation has been made assuming that both the video and the video are presented to the second user. In addition, the terminal device 10B on the trainer side may refer to the trainee information database 2601 and display a screen on the display 132 or the like in accordance with the specification of the item "screen display".

For example, in the terminal device 10B, while displaying the body movement of the first user as a first avatar (that is, a collection of point cloud data), the captured image of the first user (not a collection of point cloud data) is displayed. may be hidden. In addition, when multiple trainees receive training from a trainer, the captured images of the trainees (not a collection of point cloud data) can be displayed on the trainer's terminal device 10, while the images can be displayed simultaneously to the trainer. It may also be hidden from the terminal devices 10 of other trainees undergoing training. As a result, both the point cloud data and captured images of the trainee's sensing results are disclosed to the trainer, and the trainees share an avatar (a collection of point cloud data) but not the captured images. You can also do it.

This allows users to train while protecting their privacy.

Further, in a certain aspect, at least one of the processor 19A, the processor 19B, or the processor 29 of the server 20 defines a virtual space and displays a video including a first avatar associated with the first user in the virtual space. A first object for displaying a video image including a second avatar associated with a third user who trains together with the first user is arranged, and the processor 19A or 19B A video including the first object and the second object may be displayed on the display 132A, the display 132B, or the like.

This allows users to experience training in the same virtual space, such as when training is conducted with multiple people, and provides a sense of unity and accomplishment during the training, which can be expected to maintain and improve motivation.

Next, with reference to FIG. 28, the flow of processing when a trainer instructs a trainee online will be described.

In the following explanation, the trainer instructs the trainee by voice and controls an application running on the trainee's terminal device 10 using the trainer's voice, thereby making it easier for the trainer to obtain information suitable for the training. An example of how to do this will be explained below. Specifically, on the trainee's terminal, the trainer can control operations such as drawing an avatar based on point cloud data, drawing auxiliary lines, and determining the type of body movement to be imposed on the trainee using voice input. do it like this.

FIG. 28 is a flowchart showing the flow of processing for receiving the trainer's voice input from the trainer's terminal and transmitting it to the trainee's terminal.

In step S2851, the processor 19B of the second user's terminal device 10B receives the voice input by the second user through the microphone 141B or the like, and transfers the second user's voice from the second user's terminal device 10B to the first user's voice. It is transmitted to the terminal device 10A.

In step S2801, the processor 19A of the first user's terminal device 10A outputs the second user's voice received from the terminal device 10B to the first user through the speaker 142A or the like.

In step S2803, the processor 19A analyzes the second user's voice received from the terminal device 10B using voice recognition. The processor 19A identifies the content of the second user's voice based on the voice recognition analysis result. Based on the content of the identified second user's voice, the processor 19A identifies the content of the instruction to the application that draws the avatar based on the first user's point cloud data. The processor 19A controls the application based on the specified contents of the instruction to the application.

Specifically, the content of the second user's voice identified by the processor 19A is as follows:
(i) There is content for making the first user's body movement ideal. For example, the processor 19A uses voice recognition to recognize words related to the user's body parts (knees, elbows, arms, shoulders, etc.) from the second user's voice. In addition, the processor 19A uses words in the second user's voice that indicate how to move the first user's body parts to make an ideal movement (for example, "bring your knees close to your body", "legs"). (e.g., ``to keep the body level'', ``to straighten one's back'', etc.).

(ii) The audio content specified by the processor 19A includes content indicating the type of exercise (training menu, etc.) that the first user is to engage in for training the first user.

When the content of the second user's voice includes a type of exercise, the processor 19A sets the type of exercise to be imposed on the first user as the exercise specified by the second user in the voice, and also assigns the type of exercise to the first user. The auxiliary line drawn in association with the corresponding avatar is made to correspond to the specified exercise type. This makes it even easier for the second user to evaluate the first user's body movements.

(iii) The content of the audio specified by the processor 19A includes content that causes the application running on the terminal device 10A that draws the first user's avatar and auxiliary lines to perform a specific process. For example, in the application, commands indicated by the second user's voice include the following.
- Drawing mode of the avatar (including the position where the virtual camera is placed with respect to the point cloud data representing the body movements of the first user; in other words, whether the avatar corresponding to the first user is drawn from the front or from the side) (including whether to draw from other directions)
- How to draw auxiliary lines (as auxiliary lines, draw auxiliary lines parallel to the movement of the user's body, draw auxiliary lines along the shape of the user's body, whether to draw auxiliary lines or not) etc.)
- How to draw an avatar when there is a difference in movement compared to the ideal movement (if the movement of a specific body part is too large or small compared to the ideal movement, the avatar is drawn with a different color) (including drawing parts of the body)
- Instructions to temporarily stop the application, other contents

When the processor 19A recognizes the speaker as a trainer through voice recognition (for example, when the speaker is a trainer registered in the trainer information database 2602), the processor 19A recognizes the first user based on the voice of the speaker. It may also be possible to control an application that draws an avatar. This can prevent the application from being unintentionally controlled by users other than the trainer. Furthermore, when communicating with another terminal device 10 such as the terminal device 10B using a web conference tool, the processor 19A determines whether the first user It may also be possible to control an application that draws an avatar. For example, in web conferencing tools, etc., information (such as icons) that identifies each user participating in the meeting is displayed, and information that identifies each user who is speaking is displayed separately. There is. The processor 19A may assume that the speaker is the trainer and perform the above processing when the icon corresponding to the trainer is displayed separately from the icons of other users in the web conference tool etc. good.

For example, when multiple trainees receive guidance from a trainer, depending on the trainee's voice, the application may be required to perform training-related processing (processing to change the type of exercise that the trainees are asked to perform, auxiliary lines to be drawn with the avatar to accompany the change). It is possible to have the trainer perform the processing without having the trainer perform the processing (change processing, etc.). Thereby, even if each user speaks aloud during training, the training can be performed as intended by the trainer.

As described above, as a result of the second user giving voice guidance to the first user, the display mode of the video including the first avatar is changed, and the second user displays the video based on the video including the changed avatar. It is possible to proceed with instruction to one user.

(Sensing the first user's movements and identifying instructional content based on the learned model)
FIG. 29 shows a diagram in which the movement of the first user's body is sensed as point cloud data, and information based on the point cloud data of the first user obtained through sensing and the learned model are used to send information to the first user. It is a flowchart which shows a series of processes which specify the content of instruction. This allows the trainee to check how to move his or her own body using the trained model, even if a trainer is not present.

In step S2901, the processor 19A of the first user's terminal device 10A senses the movement of the first user's body as point cloud data using the measurement unit 30 or the like.

In step S2902, the processor 19A specifies the content of instruction for the first user based on information based on the first user's point cloud data obtained through sensing and the learned model.

Here, a method for creating a trained model will be exemplified.
(i) For example, after acquiring information regarding point cloud data of one or more users, various information (e.g., age, weight, purpose, etc.) and information on the user (trainer) who is in a position to instruct the users undergoing training and the content of the training that the trainee actually received, and information such as the history of changes in various trainee information due to training. Create a trained model.

In other words, the trained model can be said to output how much the trainee changes by training the trainee by learning information about the trainee and the contents of instruction by the trainer as time-series data.

In addition, the trained model is associated with the information of the user who acquired the information based on the point cloud data that is the learning data of the trained model (that is, the information of the trainee), or the information of the trainer who provided guidance to the trainee. It may be. Thereby, when a trainee or a trainer specifies a trained model to be used by the trainee, the trainee or trainer can refer to the trainee information or the trainer information associated with the trained model. Alternatively, the processor 19A or the like may specify a trained model to propose to a user who is a trainee based on user information associated with the trained model.

(ii) Alternatively, the trained model can be used by a trainer to tag (mark body parts) time-series data of the results of measuring how the trainee's body moves (point cloud data, captured images of the trainee). Generate the relevant time-series data as learning data by making comments on how to move in the ideal way, saying that you should breathe, that you should take a break because your heart rate is too high, etc. It may also be a thing.

For example, in one aspect, based on the measurement results of the trainee's body movements and the learned model, a training menu tailored to the constitution of the trainee user may be presented, the timing for breathing during training, the heart rate etc. It is possible to specify and present the timing to the user, such as when to get up, when to drink water, and when to take a break.

For example, in one situation, a trainee wears a wearable device (wristwatch type, glasses type, ring type, etc.) and uses the trained model described above while measuring the heart rate, breathing timing, etc. of the trainee user. Accordingly, the learned model may output the content to be taught to the trainee.

In addition, in other aspects, the user's exercise intensity (intensity of exercise), including calorie consumption, is identified from information about point cloud data acquired from the user, and advice is provided to the user, such as taking a break and hydrating. be able to.
(iii) As a method for generating a trained model, as explained in the first embodiment, a trained model is generated using measurement data (measurement history database 283) accumulated as a result of ROM check as training data. You can also use it as For example, a learned model may be generated based on the measurement data, a training menu performed by the user, and the results of responses to a questionnaire or the like input by the user.
Thereby, the type of exercise to be proposed to the user, the content of advice to the user, etc. may be output to the trained model generated based on the measured data of the ROM check.

In addition, as a method for generating a trained model, as described in the second embodiment, measured data (measurement result database 287) accumulated as a result of evaluating a user's walking posture is used as training data to create a trained model. It is also possible to generate . For example, a learned model may be generated based on the measurement data, a training menu performed by the user, and the results of responses to a questionnaire or the like input by the user.
Thereby, the type of exercise to be proposed to the user, the content of advice to the user, etc. may be output using a trained model generated based on measurement data for evaluating walking posture.

For example, as a method for generating a trained model, the trained model may be generated based on measurement data accumulated as a result of estimating the user's posture (such as joint positions). Specifically, as a method for generating a trained model on the server 20 or the like, a trained model is generated based on the results of estimating the joint positions of the user, the training menu performed by the user, and the content of advice from the trainer. It is also possible to generate the trained model and store the generated trained model.

As described above, the server 20 may specify the instruction content for the user based on the information based on the estimation result of the user's posture and the learned model, and may present the specified instruction content to the user.

Further, the server 20 or the like may store the trained model in association with information about the user who has acquired learning data (results of estimating the user's posture) of the trained model.

Further, the server 20 or the like may specify the user's exercise intensity in the same manner as described above based on the information based on the estimation result of the user's posture and the learned model. The server 20 may specify the instruction content for the user based on the result of specifying the user's exercise intensity, and may present the specified instruction content to the user.

In step S2903, the processor 19A presents the instruction content specified by the trained model to the first user by displaying it on the display 132A or the like.

<Screen example in the fifth embodiment>
30 to 32 are diagrams showing examples of screens displayed on the terminal device 10.

FIG. 30 shows how the first user's terminal device 10A senses the movement of the first user's body as point cloud data, and transmits information based on the first user's point cloud data obtained through sensing to the second user's terminal. This is an example screen when sending to.

In FIG. 30, a camera 2901 of the measurement unit 30 represents a device for sensing the movement of the first user's body.

Specifically, the camera 2901 photographs an image of the first user through a photographing unit installed in the camera 2901. At this time, the camera 2901 is configured to be able to measure distance and direction, and is configured using, for example, an RGB-D camera or LIDAR (Laser Imaging Detection and Ranging) technology.

The previous screen 2902 is a screen that displays, when the first user performs training, an image that includes the first user's avatar (that is, an image drawn based on point cloud data) when the first user previously performed the same type of training. be. The terminal device 10A, for example, refers to the training history database 2603 of the server 20 to obtain point cloud data indicating the movement of the first user's body when the first user underwent training in the past.

In the illustrated example, the first user's avatar is shown as a collection of shapes such as circles on the previous screen 2902, but it may also be displayed as a collection of point cloud data measured by the measurement unit 30. .

By displaying an avatar based on point cloud data from the first user's past training, the first user and the second user can easily confirm how much the first user's body movements have changed due to training. I can do it. For example, when the first user holds his knees and performs a movement of curling his body, the previous screen 2902, which is a previous training screen, is displayed on the terminal device 10A. The first user or the second user can compare the display contents of the previous screen 2902 and the current screen 2903, which will be described later. Thereby, the first user can confirm the difference in the movement of his or her own body.

Further, the terminal device 10A may always display the previous screen 2902, or may display or turn off the display according to an instruction from the first user or the second user. At this time, the terminal device 10A may display, as an avatar representing the first user, an image of the first user drawn from a plurality of viewpoints, such as the front, side, and top. Furthermore, an auxiliary line may be drawn at a position based on a body part that is used to evaluate whether or not the body movement performed by the first user is ideal. In the illustrated example, an auxiliary line is drawn on the past screen 2902 in association with the area where the first user's avatar is displayed.

The current screen 2903 is a screen that displays an image of the first user's avatar based on the sensing results of the measurement unit 30 when the first user is training.

In the illustrated example, a first user 2094 is shown, and the first user is actually training on a mat or the like. The terminal device 10A photographs the movement of the first user with the camera 2901, and acquires information on the first user's point cloud data.

The first user image 2905 is a captured image of the first user captured by the camera 160 or the like. In the first user image 2905, the first user is not drawn as a collection of point cloud data, but an image taken with a 2D camera or the like is displayed. That is, in the first user image 2905, a photographed image equivalent to what a human being can visually recognize with the naked eye is displayed.

Thereby, the terminal device 10A allows the first user to compare an actual image of the first user himself with an image based on the information of the point cloud data. In addition, the terminal device 10 does not display the photographed image of the first user itself in the first user image 2905 according to the first user's specification, but allows the second user etc. to identify that the first user is the first user. It is also possible to display icons, character strings, etc.

The second user image 2906 shows an image of a second user who is in a position to instruct users undergoing training. The terminal device 10A displays the second user's photographed image in the second user image 2906 by receiving the second user's photographed image from the second user's terminal device 10B. Thereby, the first user and the second user can perform training while communicating and viewing each other's images through a video call between terminals via the network 80 or the like.

FIG. 31 shows the overall configuration when sensing the first user's body movement as point cloud data and transmitting information based on the first user's point cloud data obtained by sensing to the second user's terminal. It is an example of a screen of a 1st user and a 2nd user including.

In FIG. 31, a camera 3001, like the camera 2901 in FIG. 30, is a device for sensing the movement of the first user's body as point cloud data.

Similar to the current screen 2903 in FIG. 30, the current images 3002 and 3006 are screens that display images of the first user as a collection of point cloud data when the first user is training. At this time, the current image 3002 is displayed on the display 132A of the first user's terminal device 10A, and the current image 3006 is displayed on the display 132B of the second user's terminal device 10B.

The first user 3003 is a diagram showing a state in which the first user is actually training on a mat or the like, similar to the first user 2904 in FIG. 30.

Like the first user image 2905 in FIG. 30, the first user images 3004 and 3007 show actual captured images, which are not videos including an avatar, when the first user is captured by the camera 2901. At this time, the first user image 3004 is displayed on the display 132A of the first user's terminal device 10A, and the first user image 3007 is displayed on the display 132B of the second user's terminal device 10B. .

Second user images 3005 and 3008, like second user image 2906 in FIG. 30, show images of a second user who is in a position to instruct users undergoing training. At this time, the second user image 3005 is displayed on the display 132A of the first user's terminal device 10A, and the second user image 3008 is displayed on the display 132B of the second user's terminal device 10B. . In a certain situation, as shown in the screen example, when the second user inputs a voice, the second user's voice (such as "Let's bend down more!") is transmitted through the speaker 142A of the terminal device 10A. ) may be output.

The second user 3009 visually recognizes the screen displayed on the display 132B or the like by the terminal device 10B, and provides guidance based on the first user's image. At this time, the second user can give instructions to the first user by voice. The second user checks an image based on the point cloud data and/or an image equivalent to the one seen with the naked eye as an image of the first user, and responds to the movement of the first user's body. can give advice. By reducing the amount of information for the second user as a collection of point cloud data compared to the first user's image shown by the actually captured image, the second user can more easily discern the first user's body movements. This makes it easier to evaluate and give advice.

FIG. 32 is an example of a screen when sensing the body movements of multiple users as point cloud data and transmitting information based on the point cloud data of each user obtained through sensing to a defined virtual space. .

In FIG. 32, a virtual space 3101 indicates a space in which one or more users undergoing training place objects for displaying images including avatars associated with themselves.

A trainer image 3102 shows an image of a second user who is in a position to instruct a user undergoing training. The second user's terminal device 10B photographs the second user and transmits the photographed image to the computer that manages the virtual space at any time.

The user image 3103 is a screen that displays images of one or more users when they are training. At this time, the displayed image may be only a video including the avatar as a collection of point cloud data associated with the user, or may be a combination of images taken by each user that do not include the avatar. Good too.

This allows each user to place in the virtual space an image that includes an avatar that corresponds to their own body movements, and allows each user to view the image of each user's avatar placed in the virtual space. This allows users to feel as if multiple users are training at the same location. Thereby, it is possible to obtain a sense of unity and accomplishment in training, and it is possible to provide each user with motivation to continue training. In response to matching each trainee receiving training from the trainer, the server 20 enables communication between each trainee's terminal and the trainer's terminal, and constructs a virtual space on either the server 20 or each user's terminal. You can also do it. That is, the server 20 may specify a computer that serves as a host when each user communicates.

<6 Variations>
A modification of this embodiment will be described. That is, the following aspects may be adopted.

(1) In the examples shown in FIGS. 31 and 32 above, an avatar based on the sensing results of the user who received training in the past and an avatar based on the sensing results of the user currently receiving training from the trainer are used. An example of displaying them side by side was explained.

In addition, avatars corresponding to trainees may be displayed side by side based on time series data of point cloud data of trainees sensed in time series during training with a trainer.

In other words, the avatar is based on the point cloud data sensed at the timing of starting training or within a certain period of time after training starts, and the result of real-time sensing based on the trainer's guidance or a certain period of time has passed since the trainer's guidance started. The avatar based on the sensing result at the timing may be displayed side by side on the terminal device 10A.

In addition, during the course of one training session, parts of the trainee's body movement that are particularly close to the ideal body movement will be highlighted and reflected in the avatar compared to other parts. You can also do it. For example, the processor 19A may evaluate how close the trainee's body movement is to the above-mentioned auxiliary line drawn with the avatar, and draw the screen according to the evaluation result. For example, when extending the knee in a specific direction is an ideal movement, the processor 19A draws an auxiliary line along the specific direction. The processor 19A changes the drawing mode of the avatar's knee part or the auxiliary line associated with the knee part as the movement of the knee approaches the direction along the auxiliary line in the way the trainee moves the body. Good too.

This allows trainees to easily check how much they have changed before and after a single training session, or during the training process, so they can check how much they themselves have changed (such as whether they are getting closer to their goals). It becomes easier. This can further improve the trainee's motivation to continue training.

When training online, the amount of visual information is reduced compared to when training is done face-to-face rather than online (visual information is visible to the naked eye in face-to-face training, whereas online training uses images on a two-dimensional display). There is a risk that the quality of the communication experience will deteriorate. Additionally, when training online, it is extremely difficult for the trainer to touch the trainee's body. Therefore, by providing training result feedback to the trainee as described above, it is possible to provide the trainee with motivation to continue training online.

As described above, when the way the user moves his or her body changes to the ideal way after one training session, a learned model may be generated based on the sensing results.

(2) In the above embodiments, an example has been described in which an avatar is displayed based on the result of sensing the user's body movement, and an avatar corresponding to the trainer is displayed to the trainee in online training between the trainer and the trainee.

In addition, the processor 19A may train the user while checking how the user moves his/her own body as the above-mentioned avatar, without relying on communication between the trainer and the trainee. For example, the user sets the way to move the body on the terminal device 10A, draws auxiliary lines, etc. according to the set way to move the body, and notifies the user of information about the way to move the body ( guidance on how to move the user's body), and sensing how the user moves the body using the measurement unit 30 or the like.

(2-1) Sensing method of user's body movement The terminal device 10A presents the user with the result of sensing how the user's body moves as an avatar or the like. Furthermore, as described in Embodiments 1 and 2, the terminal device 10A can identify the position of each joint of the user's body and estimate the skeleton (bone) based on the sensing results of the user's body. (Figure 12).

The terminal device 10A may estimate the posture based on the position of each joint of the user and the estimation result of the skeleton in the following manner.
- The terminal device 10A estimates the posture of the user shown in the video based on the photographed result (two-dimensional video) of the user as the subject, without using point cloud data.
That is, the terminal device 10A photographs the user using a camera that does not have a configuration for optically measuring the distance to the subject, such as LiDAR, and based on the photographic result, determines the posture of the user in the photographed image. It may also be possible to estimate joint positions, skeletons, etc.
For example, by preparing a trained model that receives a photographed image as input and outputs the positions of the joints, the posture of the skeleton, etc. of the person who is the subject in the photographed image, the terminal device 10A can receive the user's photographic results, The user's posture can be estimated based on the learned model.
Such a trained model is generated by preparing learning data in advance that associates information such as joint positions and bone structure in the photographed image about the person who is the subject of the photographed image, and having the computer perform learning. It is possible.
- As explained in other embodiments, the terminal device 10A senses the user's body movement as point cloud data based on the camera or measurement unit 30 etc. installed in the terminal device 10A, etc., and uses the sensing results as point cloud data. Based on this, joint positions, skeleton, etc. are estimated.

(2-2) Drawing auxiliary lines based on joint position and skeleton estimation results The terminal device 10A draws auxiliary lines for the user to use as a guide when moving the body, based on the estimation results of the user's posture. You can also do it. The terminal device 10A may draw the auxiliary line based on the estimation result of the user's posture as follows.
- The type of body movement that the terminal device 10A etc. presents to the user (for example, exercises for evaluating trunk muscle movements and joint range of motion such as lateral lifts, or exercises for muscle strength training) The position at which the auxiliary line is drawn is determined based on the position of the joint (motion, etc.), the position of the joint, and the estimation result of the skeleton.
For example, depending on the type of body movement, the user may move the body a certain amount (for example, an action of stepping forward with a front foot to evaluate joint range of motion). The terminal device 10A displays an avatar to the user and draws an auxiliary line as a guideline for the amount of body movement based on the estimation result of the position of the user's specific joint.
For example, when the user steps out with his or her front foot, the terminal device 10A first prompts the user to take a predetermined posture (for example, stand upright before stepping out) using audio or video. The terminal device 10A estimates the amount of the user's stride based on the position of the joints (for example, shoulder joint, hip joint, knee joint) in the posture (for example, the user's height and the length of the user's lower legs). (the amount of stride length to be taken can be specified), and based on the estimated amount of stride length, the position at which the auxiliary line should be drawn is determined based on the estimation result of the joint position.
The terminal device 10A associates in advance the position of a joint to be referred to for drawing an auxiliary line and the positional relationship for drawing an auxiliary line with respect to the position of the joint, depending on the type of body movement performed by the user. It is also possible to memorize it.

FIG. 33 is a diagram illustrating the concept of a process in which the terminal device 10A draws an auxiliary line based on the estimation result of the user's posture.

In the example (A) of FIG. 33, it is assumed that the terminal device 10A senses the movement of the user's body as point cloud data. The terminal device 10A stores three-dimensional object data of the avatar AVT_A corresponding to the user based on the sensing result. Based on the three-dimensional object data of the avatar AVT_A, the terminal device 10A estimates the positions of the user's joints (for example, each joint shown in FIG. 12) of the three-dimensional object data. In example (A), when the three-dimensional object data of the avatar AVT_A is placed in the virtual space, the terminal device 10A specifies the joint position JNT_01A, which is the coordinate of the estimated position of the user's hip joint in the avatar.
In the example (A) of FIG. 33, the terminal device 10A moves the three-dimensional object data of the auxiliary line LN_01A to a position a certain distance DIST_01A away from the joint position JNT_01A, depending on the type of the user's body movement. Decide to place it. The terminal device 10A may set the certain distance based on the user's physique (obtained by measuring the user's height, sitting height, shoulder width, etc. based on the sensing results of the user's body movement). . Depending on the user's physique, a guideline amount by which the user moves his or her body can be determined. In example (A), the terminal device 10A arranges the three-dimensional object data of the auxiliary line LN_01A so that it is parallel to the ground when the user moves his or her body.

In the example (B) of FIG. 33, the terminal device 10A sets the three-dimensional object data of the auxiliary line LN_02A based on the joint position JNT_01A so that it overlaps with the joint position JNT_01A, and the ground surface on which the user moves his/her body. It is placed perpendicular to.

FIG. 34 shows an example of a screen presented to the user.
As shown in FIG. 34, the terminal device 10A displays a plurality of avatars corresponding to the user on the screen 3403, such as a left side 3403A, a front 3403B, and a right side 3403C, as a result of sensing the user's body movements. Place the results drawn from the viewpoint on the same screen.

For example, the terminal device 10A arranges three-dimensional object data (that can be recognized by the user as an avatar) corresponding to the sensing results of the user's body movement (for example, point cloud data) in the virtual space. In the virtual space, the terminal device 10A uses a plurality of virtual cameras (for example, a virtual camera corresponding to the right side, a virtual camera corresponding to the front, and a virtual camera corresponding to the left side) to three-dimensional object data corresponding to the sensing result of the user. corresponding virtual camera). Thereby, the terminal device 10A can present to the user the results of drawing the avatar corresponding to the user from a plurality of viewpoints.

As illustrated, the terminal device 10A may present the avatar to the user, but may not present the result of estimating the skeleton (bones). For example, it is assumed that a user has fewer opportunities to visually confirm the bone estimation result itself with the naked eye on a daily basis, compared to observing himself or herself in a mirror or the like. Therefore, the terminal device 10A presents an avatar to the user but does not display the bone estimation results, so that the user can see an avatar on the screen that is similar to his or her own body shape that can be seen with the naked eye on a daily basis. The user can be immersed in the avatar projected on the screen as if it were the user himself/herself.

The terminal device 10A arranges the three-dimensional object data of the auxiliary lines LN_01A and LN_02A in association with the three-dimensional object data corresponding to the user in the virtual space. Therefore, the terminal device 10A can display the auxiliary line together with the avatar corresponding to the user by performing rendering according to the settings of the virtual camera.

Specifically, the terminal device 10A draws an auxiliary line LN_01A and an auxiliary line LN_02A in association with the avatar AVT_A on the left side surface 3403A, and displays them to the user. The terminal device 10A draws an auxiliary line LN_01B and an auxiliary line LN_02B in association with the avatar AVT_B on the front surface 3403B and displays them to the user. The terminal device 10A draws an auxiliary line LN_01C and an auxiliary line LN_02C in association with the avatar AVT_C on the right side surface 3403C and displays them to the user.

(2-3) Method for evaluating user's body movement (i) As described above, the terminal device 10A displays the avatar without displaying the bone estimation results to the user. They are made to realize that they are a projection of themselves, and are immersed in moving their bodies, training, etc. On the other hand, the terminal device 10A evaluates the user's body movement to evaluate whether the user's posture is desirable and whether the user's body movement is ideal depending on the type of body movement. , the estimation result of the user's skeleton (bone) may be used.

For example, in the example of FIG. 34, if it is desirable that the user's knees be vertical, the terminal device 10A may determine whether the knees are vertical based on the estimation results of the knees among the user's skeleton (bones). It can be determined whether the user's posture is desirable or not.
(ii) In order to evaluate the user's body movement, the terminal device 10A corresponds to three-dimensional object data (for example, point cloud data) constituting the avatar that is the result of sensing the user's body movement, and auxiliary lines. It may also be based on three-dimensional object data.

For example, the terminal device 10A may cause a collision between three-dimensional object data constituting the user's avatar (for example, avatar AVT_A on the left side surface 3403A) and three-dimensional object data corresponding to an auxiliary line (for example, the auxiliary line LN_01A on the left side surface 3403A). It may also be possible to evaluate whether or not the user's body movement is ideal by determining that the user has moved within a certain distance, or determining that the user has approached within a certain distance.

When the terminal device 10A senses the movement of the user's body without using point cloud data (for example, when photographing the user as a two-dimensional image), based on the sensing result (two-dimensional photographed image data), Three-dimensional object data corresponding to the user's body may be generated. The terminal device 10A evaluates whether the user's body movement is ideal based on whether the 3D object data and the 3D object data corresponding to the auxiliary line collide or approach a certain distance. You can also do it.

When sensing the movement of the user's body without using point cloud data (when photographing the user as a two-dimensional image), the terminal device 10A detects the user's posture (of the joints) in the two-dimensional image as described above. position, skeleton) (using a trained model that estimates the posture as described above), and based on the estimated results, auxiliary lines may be drawn as object data in the two-dimensional image. For example, the terminal device 10A generates object data in the two-dimensional image based on the position of the user's joints (hip joints, shoulder joints, etc.) estimated in the two-dimensional image, based on the type of exercise performed by the user. Decide where to draw the auxiliary line. The terminal device 10A determines whether the user's body movement is ideal based on whether the two-dimensional image corresponding to the user and the auxiliary line drawn as object data collide or approach a certain distance. It is also possible to evaluate the

(2-4) Screen example presented to the user to measure specific body movements (a) Standing Alignment
For example, while the user is in a specific posture, such as standing, the user can maintain a good posture by allowing the user to adjust the user's body so that it does not tilt forward, backward, left, or right.

FIG. 35 is a diagram illustrating an example of a screen on which the terminal device 10A is used to adjust the tilt of the terminal device 10A in the front, rear, right, and left directions while the user is in a standing position.

FIG. 36 is a diagram illustrating a process of specifying a position at which a graphical object that is a guide to the movement of the user's body is drawn based on the result of estimating the position of a specific body part of the user.

As shown in FIG. 35, the terminal device 10A draws the avatar corresponding to the user from multiple viewpoints, such as the left side 3501A and the front 3501B, as a result of sensing the user's body movements on the screen 3500. Place the results on the same screen. The terminal device 10A displays the positions of multiple parts of the user's body on the screen 3501C so that the positional relationships in the horizontal plane can be compared.

The terminal device 10A displays an avatar AVT_35A corresponding to the user (corresponding to when the user is viewed from the left side) on the left side 3501A. The terminal device 10A displays the auxiliary line VL_35A together with the avatar AVT_35A as a graphic object.

Here, the process by which the terminal device 10A draws the auxiliary line VL_35A will be described.
As shown in the example (A) of FIG. 36, the terminal device 10A estimates the position JNT_35D of the ankle (heel) as a specific part of the user's body.

The terminal device 10A can estimate the positions of the right ankle, right foot, right toe, left ankle, left foot, and left toe based on point cloud data obtained by measuring the user's body (for example, the example in FIG. 12). For example, based on the estimation result of the position of the right ankle or the left ankle, the terminal device 10A draws the auxiliary line VL_35A as a graphic object so that it passes through the estimation result of the ankle position and becomes perpendicular to the ground. The drawing position may also be specified.

Thereby, the terminal device 10A updates the display of the avatar AVT_35A to the user each time it acquires sensing results of the user's body movement (that is, updates the display of the avatar AVT_35A based on point cloud data that measures the user's body in real time). While drawing the avatar AVT_35A), the positional relationship with the auxiliary line VL_35A can be recognized. The user can recognize whether or not the body is shifted forward, backward, left, or right by comparing the avatar AVT_35A and the auxiliary line VL_35A. For example, if the avatar AVT_35A significantly protrudes from the position of the auxiliary line VL_35A, it may mean that the user's body has shifted significantly in either the front, back, left, or right direction.

Note that in order to draw the auxiliary line VL_35A so that the user can compare it with the avatar AVT_35A seen from the left side as described above, the terminal device 10A draws a position other than the ankle as a specific part of the user's body. Drawing may be performed based on the estimated result (for example, based on the estimated result of the pelvis position).

As shown in the front view 3501B of FIG. 35, the terminal device 10A displays an avatar AVT_35B (corresponding to when the user is viewed from the front) corresponding to the user. Terminal device 10A displays auxiliary line LN_35A, auxiliary line LN_35B, auxiliary line LN_35C, and auxiliary line VL_35B as graphic objects in association with avatar AVT_35B.

Here, a process in which the terminal device 10A draws the auxiliary line LN_35A and the like on the front surface 3501B will be described.

As shown in the example (B) of FIG. 36, the terminal device 10A selects a head position JNT_35A (“head” in the example of FIG. 12) and a shoulder position JNT_35B (“head” in the example of FIG. 12) as specific parts of the user's body. ``shoulder center'' in the example), and the waist position JNT_35C (``pelvis center'', ``right pelvis'', ``left pelvis'', etc. in the example of FIG. 12) are estimated. The terminal device 10A determines the drawing position so that the auxiliary line VL_35B passes through at least one of the head position JNT_35A, the shoulder position JNT_35B, and the waist position JNT_35C and becomes a line perpendicular to the ground. Identify. Here, the terminal device 10A sets the auxiliary line VL_35B based on the position of a specific part other than these (for example, it passes through the midpoint between the right heel position and the left heel position and becomes perpendicular to the ground. line), or specifying the position to draw.

The terminal device 10A specifies the position where the auxiliary line LN_35A etc. are drawn based on the estimation results of the positions of these specific parts. For example, the terminal device 10A sets the auxiliary line LN_35A to a predetermined position in the horizontal direction (assuming that the direction toward the ground is vertical) from the head position JNT_35A, which is the estimated result of the user's head position. It is specified that the position is separated by a distance (distance DIST_35A in the example of FIG. 36). The terminal device 10A specifies that the auxiliary line LN_35B should be drawn at a position horizontally separated by a predetermined distance from the shoulder position JNT_35B. The terminal device 10A specifies that the auxiliary line LN_35C should be drawn at a position horizontally separated by a predetermined distance from the waist position JNT_35C.

The terminal device 10A draws these auxiliary lines LN_35A, LN_35B, and LN_35C in different ways (eg, different colors, different shapes, etc.). This allows the user to easily recognize which part of the user's body each drawing mode corresponds to. Here, the terminal device 10A draws the auxiliary circles RD_35A, RD_35B, and RD_35C on the screen 3501C according to the drawing mode of each of the auxiliary lines LN_35A, LN_35B, and LNN_35C.

The terminal device 10A draws an auxiliary circle RD_35D, which is a graphic object based on the position of the user's heel, on the screen 3501C. Example (C) of FIG. 36 shows a mode in which the auxiliary circles RD_35A, RD_35B, and RD_35C are not drawn on the screen 3501C of FIG. 35. The terminal device 10A draws an auxiliary circle RD_35A corresponding to the position of the user's head, an auxiliary circle RD_35B corresponding to the shoulder, and an auxiliary circle RD_35C corresponding to the waist on the screen 3501C. These auxiliary circles RD_35A, RD_35B, RD_35C, and auxiliary circle RD_35D may each have the same shape or may have similar shapes. The terminal device 10A displays these auxiliary circles RD_35A, RD_35B, and RD_35C at the relative positions of the head, shoulders, and hips to be compared with the position in the horizontal plane of a specific part that is a reference. As a result, the user can recognize whether the body is tilted forward, backward, left, or right, and use the multiple graphical objects (auxiliary circles RD_35A, RD_35B, and RD_35C) that serve as a reference for these multiple body parts. You will be prompted to match it to circle RD_35D.

As described above, the terminal device 10A sequentially senses the movement of the user's body as point cloud data, etc., and detects at least three specific parts of the user (the In the examples shown in FIGS. 35 and 36, the positions of four parts of the head, shoulders, hips, and heels are estimated. The terminal device 10A uses any one of at least three specific parts as a reference (in the examples of FIGS. 35 and 36, the heel position is used as a reference, but it may be any part along the body axis such as the waist). ), determines a position in the horizontal plane to draw a graphic object indicating the positional relationship of other specific parts, and displays the graphic object at the determined position.

The terminal device 10A displays, in an object 3501D in FIG. 35, an estimate of the remaining time during which the user performs an action to adjust the front, back, left, and right balance in a standing state. Thereby, the user is prompted to adjust the balance in the front, back, left, and right directions as much as possible within the remaining time, and the user's posture can be further improved.

(b) Lateral elevation Evaluate the user's good posture by measuring the user's movement of raising both arms in a standing position without extending them out in front of the sides of the body. can do. In the lateral lifting motion, the timing when the user fully raises both arms in a standing position will be described as the peak time of the motion.

FIG. 37 is a diagram showing an example of a screen on which the terminal device 10A causes the user to perform a lateral lifting motion.

FIG. 38 is a diagram illustrating a process of specifying a position at which a graphic object serving as a guide for the movement of the user's body is to be drawn, based on the result of estimating the position of a specific body part of the user.
FIG. 39 is a diagram showing a process in which the user moves his or her body.
FIG. 40 is a diagram showing a state in which the user fully raises both arms, which is the peak timing of the lateral lifting motion.
FIG. 41 is a diagram illustrating an example in which the user saves the sensing results at the peak timing of the lateral lifting motion and displays them as a comparable avatar.

As shown in FIG. 37, before starting the lateral lifting motion, the user assumes an initial posture by standing with both arms attached to the sides of the body. The terminal device 10A may notify the user by voice or image to take the initial posture.

The terminal device 10A draws a graphic object based on the user's initial posture and the result of estimating the position of a specific part. In the illustrated example, the terminal device 10A displays an avatar AVT_37A corresponding to the user (corresponding to the case viewed from the left side of the user) on the left side 3701A of the screen 3700. The terminal device 10A draws the following graphical object on the left side surface 3701A.

- Auxiliary line VL_37A, which is a graphical object to make it easier for the user to perceive the shift in front and back. It remains stationary and remains stationary while the user moves his or her body.
- An auxiliary line VL_37D indicating the position of the surface of the user's head in the initial posture. While the user is performing a lateral lifting motion, the user's body moves back and forth by making the user perceive the positional relationship between the avatar AVT_37A, which is the sensing result of the user's body movement, and the auxiliary line VL_37D. This makes it easier to recognize by focusing on the head. It remains stationary and remains stationary while the user moves his or her body.
- An auxiliary line VL_37E indicating the position of the surface of the user's abdomen in the initial posture. While the user is performing a lateral lifting motion, the user's body moves back and forth by making the user perceive the positional relationship between the avatar AVT_37A, which is the sensing result of the user's body movement, and the auxiliary line VL_37E. This makes it easier to recognize whether or not the patient is in a position by focusing on the abdomen. It remains stationary and remains stationary while the user moves his or her body.
- An auxiliary line HL_37A indicating the length between the auxiliary line VL_37A and the auxiliary line VL_37D. It remains stationary and remains stationary while the user moves his or her body.
- Auxiliary line HL_37B to make it easier to recognize the position of the user's shoulders in the initial posture. It remains stationary and remains stationary while the user moves his or her body.
- An auxiliary line HL_37C indicating the length between the auxiliary line VL_37A and the auxiliary line VL_37E. It remains stationary and remains stationary while the user moves his or her body.

The terminal device 10A displays avatar AVT_37C (corresponding to when the user is viewed from the right side) and auxiliary lines VL_37C, VL_37F, VL_37G, HL_37D, HL_37E, and HL_37F on the right side 3701C.
Auxiliary line VL_37C corresponds to auxiliary line VL_37A.
Auxiliary line VL_37F corresponds to auxiliary line VL_37D.
Auxiliary line VL_37G corresponds to auxiliary line VL_37E.
Auxiliary line HL_37D corresponds to auxiliary line HL_37A.
Auxiliary line HL_37E corresponds to auxiliary line HL_37B.
Auxiliary line HL_37F corresponds to auxiliary line HL_37C.

The terminal device 10A displays an avatar AVT_37B corresponding to the user (corresponding to when the user is viewed from the front) on the front side 3701B. The terminal device 10A displays the following graphical objects on the front surface 3701B.
- Auxiliary line VL_37B, which is a graphic object to help the user perceive the left-right shift. It remains stationary and remains stationary while the user moves his or her body.
- In association with avatar AVT_37B, auxiliary line LN_37L (corresponding to the user's left arm) and auxiliary line LN_37R (corresponding to the user's (corresponds to the right arm). Each of the terminal devices 10A updates the position at which the graphic object is drawn in conjunction with the sensing result of the user's body movement. That is, the terminal device 10A updates the positions at which the auxiliary lines LN_37L and LN_37R are drawn in conjunction with the movement of the avatar.
- Polygon QB_37A (corresponding to the left arm) indicating the end point of the auxiliary line LN_37L, and polygon QB_37B (corresponding to the right arm) indicating the end point of the auxiliary line LN_37R. If the user bends his elbow during a lateral movement, these polygons QB_37B or QB_37A will be drawn at a position away from the avatar AVT_37B. Alternatively, by recognizing the positional relationship with QB_37B, the bending of the elbow can be easily recognized.
・Polygon QB_37C (corresponding to the left arm), polygon QB_37C (corresponding to the left arm), to make the user perceive whether the user is shrugging his/her shoulders (is the arm raised using the shoulder joint) when the user fully raises the arm? Square QB_37D (corresponds to right arm). As will be described later, the terminal device 10A raises both arms so that when the user raises both arms without shrugging, polygons QB_37C and QB_37A overlap for the left arm, and polygons QB_37A for the right arm overlap. Polygon QB_37C and polygon QB_37D are drawn at positions where polygon QB_37D and polygon QB_37B overlap.

When prompting the user to perform a lateral lifting operation, the terminal device 10A displays an icon 3701D indicating that the user is in the initial posture, an icon 3701E indicating that the lateral lifting operation is in progress, and the measurement is completed. An icon 3701F is displayed to indicate that this has been done.

The process of drawing these graphical objects will be explained.
As shown in the example (A) of FIG. 38, the terminal device 10A identifies specific parts of the user's body, such as the position of the waist JNT_38B (such as the "center of the pelvis" in the example of FIG. 12), the position between the waist and the shoulder. JNT_38C (such as "chest and waist" in the example of FIG. 12), shoulder position JNT_37D, and head position JNT_38E are estimated.

The terminal device 10A specifies the drawing position of the auxiliary line VL_37A so that it passes through the positions such as the waist position JNT_38B and is perpendicular to the ground.

The terminal device 10A specifies the drawing position of the auxiliary line VL_37D based on the position of the body surface when passing through the head position JNT_38E and going from the user's back to the front in the horizontal direction. That is, the terminal device 10A specifies the position at which the graphic object is drawn based on the result of estimating the position of the body part and the sensing result of the user's body movement sensed as point cloud data.

The terminal device 10A passes through an intermediate position between the waist position JNT_38B and the position JNT_38C between the waist and shoulders with respect to the auxiliary line VL_37E, and the body surface when moving from the back of the user to the front in the horizontal direction. Identify the location to draw based on the location.
The method for drawing each auxiliary line shown in example (C) may be the same as in example (A).

As shown in example (B), the terminal device 10A draws the auxiliary line VL_37B based on the result of estimating the position of the user's body parts, such as the waist and pelvis. Here, based on the results of estimating the positions of the heel of the user's right foot and the heel of the left foot, the terminal device 10A directs the auxiliary line VL_37B to pass through a position intermediate between these positions and become perpendicular to the ground. You can also draw it like this.

The terminal device 10A sets the auxiliary line LN_37L from the left wrist as a starting point based on the results of estimating the user's wrist position JNT_38F (left wrist) and the user's elbow position JNT_38G (left elbow). The drawing position is specified so that the distance from to the elbow is further extended from the elbow to the end point (polygon QB_37A).

The terminal device 10A sets the auxiliary line LN_37R as the auxiliary line LN_37R in the same way as the auxiliary line LN_37L based on the results of estimating the user's wrist position JNT_38I (right wrist) and the user's elbow position JNT_38J (right elbow). Define the length and specify the position to draw.

Based on the shoulder position JNT_38H (left shoulder), the terminal device 10A identifies coordinates located a predetermined distance outside and above the user's body. The coordinates are set to correspond to the position of the shoulder that moves when the user reaches the peak of the lateral lift motion. The terminal device 10A may specify the coordinates based on the user's physical size (for example, the user's height, shoulder width, etc.). The terminal device 10A specifies the position where polygon QB_37C is drawn based on the specified coordinate values.

The terminal device 10A specifies the position to draw polygon QB_37D based on the user's shoulder position JNT_38K (right shoulder) in the same manner as the method for specifying the position to draw polygon QB_37C.

As shown in FIG. 39, when the user raises his arm, the terminal device 10A determines the trajectory of the hand movement based on the history of sensing results of the user's body movement and the history of identifying the position of the wrist. are displayed as a trajectory LN_39A (trajectory of left hand motion) and a trajectory LN_39C (trajectory of right hand motion).

As shown in FIG. 40, it is assumed that the user has raised both arms and reached the peak. The terminal device 10A detects that the user's lateral lifting motion has reached its peak (for example, detects that the user has raised both arms based on the estimation result of the position of a specific part of the user). ), and draws an avatar corresponding to the sensing result (user's posture) at the relevant timing. Here, if the polygon QB_37A, whose drawing is updated in conjunction with the user's body movements, and the polygon QB_37C, which serves as a guideline for the movement when both arms are fully raised, have a specific positional relationship (for example, When these polygons overlap), at least one of these polygons QB_37A and polygon QB_37C may be displayed in a different manner than when they do not overlap. Thereby, the terminal device 10A can easily make the user perceive that the lateral lifting motion is appropriate (such as that the shoulders are not hunched). Similarly, the terminal device 10A may display the display in a different manner when polygon QB_37B and polygon QB_37D have a specific positional relationship than when they do not.

As shown in FIG. 41, the terminal device 10A displays the history avatar AVT_37D, which is the state of the avatar at its peak, along with the avatars AVT_37A and AVT_37C whose drawings are updated in conjunction with the sensing results of the user's body movements. , display history avatar AVT_37E.

As shown in FIG. 41, when the user performs a lateral lifting motion, the avatar with both arms fully raised, the trajectory of the user raising both arms (trajectories LN_39A, LN_39C), and the user's Since the real-time postures of avatars AVT_37A and AVT_37C are displayed side by side, it is possible to provide the user with material for thinking about how to perform an action.

(C) Crouching FIG. 42 is a diagram showing an example of a screen on which the terminal device 10A allows the user to perform a crouching motion.
FIG. 43 is a diagram illustrating a process of specifying a position at which a graphic object that is a guide to the movement of the user's body is to be drawn, based on the result of estimating the position of a specific body part of the user.
FIG. 44 is a diagram showing a process in which the user moves his or her body.
FIG. 45 is a diagram showing a state in which the user has crouched down, which is the peak timing of the crouching motion.
FIG. 46 is a diagram illustrating an example in which the sensing results at the peak timing of the user's crouching motion are saved and displayed as a comparable avatar.

As shown in FIG. 42, in the initial posture before starting the crouching motion, the terminal device 10A draws a graphic object based on the initial posture and the result of estimating the position of the specific part.

In the illustrated example, the terminal device 10A draws the following auxiliary line on the left side 4201A of the screen 4200 while drawing the avatar AVT_42A.
- Auxiliary line LN_42A to visualize the angle of the user's back
- Auxiliary line LN_42B to visualize the angle of the user's shins

The terminal device 10A draws the following auxiliary line while drawing the avatar AVT_42C on the right side surface 4201C.
- Auxiliary line VL_42C for making the user perceive whether the user's posture is tilted forward or backward. Displayed in a fixed position while measuring the user's body movements.

The terminal device 10A draws the following auxiliary lines while drawing the avatar AVT_42B on the front side 4201B.
- Auxiliary line VL_42B for making the user perceive whether the user's posture is tilted to the left or right. Displayed in a fixed position while measuring the user's body movements.
- Auxiliary line LN_37L to make the user perceive whether the user's left foot is spread to the left or right.
- Auxiliary line LN_37R to make the user perceive whether the user's right foot is spread to the left or right.

The terminal device 10A displays an object 4201D indicating the state before starting the crouching motion, an object 4201E indicating that the crouching motion is being measured, and an object 4201F indicating that the measurement has ended.

As shown in the example (A) of FIG. 43, the terminal device 10A uses the heel position JNT_43A as a starting point based on the heel position JNT_43A and the knee position JNT_43B as a result of estimating the position of the user's body parts. A line segment passing through the knee position JNT_43B is drawn as an auxiliary line LN_42B.

The terminal device 10A draws a line segment starting from the waist position JNT_43C and passing through the shoulder position JNT_43D as an auxiliary line LN_42A, based on the waist (pelvis, etc.) position JNT_43C and the shoulder position JNT_43D.
Here, the terminal device 10A may determine the lengths of these auxiliary lines so that the heights of the end points of the auxiliary lines LN_42A and LN_42B are the same.

As shown in example (C), the terminal device 10A specifies the position where the auxiliary line VL_42C is drawn so as to be perpendicular to the ground based on the heel position JNT_43J. Here, the terminal device 10A may draw the auxiliary line VL_42C based on the position of the waist, etc., separately from the heel position JNT_43J.

As shown in example (B), the terminal device 10A uses the heel position as a starting point and calculates the length of the knee from the heel based on the heel position JNT_43E (left heel) and the knee position JNT_43F (left knee). is measured, and the position where the auxiliary line LN_37L is drawn is specified, with the end point being a point further extended by the length from the knee.

The terminal device 10A specifies a position to draw the auxiliary line VL_42B so that it passes through the waist position JNT_43I and is perpendicular to the ground. Furthermore, the terminal device 10A specifies the position at which the auxiliary line VL_42B is drawn, based on the positions of the left and right heels (JNT_43E, JNT_43G), so that it passes through the intermediate position between these and is perpendicular to the ground. You may.

As shown in FIG. 44, when the user starts crouching, the terminal device 10A draws a locus LN_42M of the user crouching on the front 4201B. For example, the terminal device 10A may draw the trajectory LN_42M by retaining the result of estimating the user's neck position (or head or shoulders) as a history.

The terminal device 10A traces the user's crouching trajectory LN_42 on the right side 4201C.
N may also be displayed. For example, the terminal device 10A may draw the trajectory LN_42N by retaining the results of estimating the user's neck position and the like as a history.

As a result, the user can compare the trajectory LN_42M and the auxiliary line VL_42B to determine whether the user is squatting without shaking left or right, or compare the trajectory LN_42N and the auxiliary line VL_42C to determine whether the user is squatting without shaking back or forth. can be easily recognized.

As shown in FIG. 45, the user reaches a peak in his/her crouching motion by squatting down completely. By detecting that the user's crouching motion has reached its peak (for example, based on the result of estimating the position of a specific part, the terminal device 10A compares the posture with the posture when the user's crouching motion has reached its peak). (to detect that the peak has been reached), and retain the sensing results (point cloud data, etc.) at the timing of the peak.

Here, in the fully crouched state, the terminal device 10A may determine whether the auxiliary line LN_42B (that is, the angle of the knee) and the auxiliary line LN_42A (that is, the angle of the back) are open. In the example of FIG. 45, the auxiliary line LN_42B and the auxiliary line LN_42A are in an open state based on the starting point (heel or waist), and are not bent forward, so it may be determined that they are in good condition. . In addition, the terminal device 10A is configured such that the end point of the auxiliary line LN_42B (when the heel is the starting point) and the end point of the auxiliary line LN_42A (when the waist is the starting point) are at a height from the ground when the user is fully crouched down. It may also be drawn in a waxy manner. Further, the terminal device 10A may determine whether the user's movement is good depending on whether the distance between the end point of the auxiliary line LN_42A and the end point of the auxiliary line LN_42B is a certain distance or more ( If the person is used to the place, it is judged to be good).

As shown in FIG. 46, when the user stands up after crouching, the terminal device 10A displays the sensing results of the user's crouched state on the left side 4201A, the avatar AVT_42D, the front 4201B, the avatar AVT_42E, and the right side 4201C. Displayed as avatar AVT_42F.

Terminal device 10A displays auxiliary lines LN_42A and LN_42B in a fully crouched state together with avatar AVT_42D on left side surface 4201A.

The terminal device 10A draws a trajectory LN_42P (for example, a trajectory is drawn based on the neck position) when standing up after crouching down (after reaching the peak timing) on the front surface 4201B.

The terminal device 10A draws a trajectory LN_42Q (for example, a trajectory is drawn based on the neck position) when standing up after crouching down (after reaching the peak timing) on the right side surface 4201C.

(D) Motion of lowering one's hips while taking a step FIG. 47 is a diagram showing an example of a screen on which the terminal device 10A causes the user to lower one's hips while taking a step.
FIG. 48 is a diagram showing an example of a screen in which the user is shown that the user's initial posture is appropriate.
FIG. 49 is a diagram illustrating a phase in which measurement of a user's motion is started.
FIG. 50 is a diagram illustrating a situation in which it is determined whether the user's motion is appropriate and the determined result is presented to the user.
FIG. 51 is a diagram showing a situation in which it is determined whether the user's motion is appropriate and the determined result is presented to the user.

As shown in FIG. 47, the terminal device 10A displays an avatar AVT_47B (corresponding to the front) corresponding to the user on the front 4701B of the screen 4700. The terminal device 10A displays, on the right side surface 4701C, an avatar AVT_47C (right side surface) corresponding to the user, and an auxiliary line LN_47A and an auxiliary line LN_47B that serve as a guide for the position where the user steps both feet.

In the illustrated example, the terminal device 10A displays an object on an arrow that prompts the user to lower his or her hind legs further back.

In the example of FIG. 47, the terminal device 10A measures the length of the user's legs based on the result of estimating the position of the user's specific region. For example, the terminal device 10A measures the length of the user's legs based on the position of the user's heel and the position of the hip (pelvis). Thereby, the terminal device 10A specifies the distance at which the user should spread his/her legs when performing an action. The terminal device 10A may adjust the distance at which the user should spread his or her legs according to the user's physique (for example, the length of the legs). The terminal device 10A determines the interval between the auxiliary line LN_47A and the auxiliary line LN_47B based on the result of measuring the length of the user's foot, and draws it on the right side surface 4701C.

In the illustrated example, the terminal device 10A displays an object 4701D for visualizing the user's hip rotation and shoulder rotation on the screen 4700. The terminal device 10A specifies the rotation of the user's waist based on the results of estimating the positions of the pelvis (the positions of the "left pelvis" and "right pelvis" in the example of FIG. 12). The terminal device 10A identifies the rotation of the user's shoulders based on the results of estimating the positions of the shoulders (“left shoulder” and “right shoulder” in the example of FIG. 12).

The terminal device 10A displays, in the object 4701D, an auxiliary line LN_47SB indicating the rotation of the waist and an auxiliary line LN_47SH indicating the rotation of the shoulder, based on the position of the waist (such as the center of the pelvis), for example. This makes it easy for the user to perceive how much the hips and shoulders are rotating when the user lowers the hips while taking a step.

As shown in FIG. 48, before measuring the movement of the user opening his/her legs, lowering his or her hips, and moving the user up and down, the terminal device 10A determines whether or not the user has sufficiently opened both legs.

The terminal device 10A may determine whether the user's legs are appropriately spread forward and backward in the following manner.
- Based on the results of estimating the positions of the user's body parts, the angle at which the legs are apart (such as the angle formed by the line segment from the left knee to the pelvis and the line segment from the right knee to the pelvis) is calculated. When it is more than a predetermined value - When a specific part of the foot (heel, etc.) that the user steps forward and backward, and the auxiliary line LN_47A and the auxiliary line LN_47B have a predetermined positional relationship. For example, when the result of estimating the position of the heel of the user's front foot and the auxiliary line LN_47A overlap or are within a certain distance. Similarly, the case where the result of estimating the position of the heel of the user's hind foot and the auxiliary line LN_47B overlap or are within a certain distance.

When the front legs and the hind legs are opened appropriately, the terminal device 10A displays an object OBJ_47A and an object OBJ_47B indicating this. If the legs are not opened properly, the terminal device 10A displays a message to that effect (for example, "NG") in the object OBJ_47A or the object OBJ_47B.

The terminal device 10A may draw the object OBG_47A or the object OBJ_47B based on the result of estimating the position of the user's specific part. For example, the terminal device 10A may draw the object OBJ_47A in association with the coordinates of the result of estimating the position of the user's heel (for example, draw the speech bubble of the object OBJ_47A based on the position of the user's heel. )

As described above, in response to detecting that the user's posture before performing an action is appropriate, the terminal device 10A provides a notification (for example, " It is also possible to provide a notification that reminds the user of specific actions, such as "Please lower your back slowly and then return to your original position." This allows the user to confirm that the user's posture before performing the movement is appropriate, making it easier for the user to start the movement of moving the user's hips up and down.

As shown in FIG. 49, when the user moves his or her hips up and down, the terminal device 10A is fixed (at a fixed position) along with the avatar AVT_47B corresponding to the user on the front 4701B, regardless of the user's movements. Draw auxiliary lines VL_47B and VL_47E to be displayed.

The auxiliary line VL_47B is an auxiliary line that allows the user to perceive whether or not the user is moving from side to side while moving the user's hips up and down. For example, the terminal device 10A draws the auxiliary line VL_47B based on the result of estimating the position of the user's waist before the user starts the motion of lowering the waist.

The auxiliary line VL_47E is an auxiliary line that allows the user to perceive whether or not the user's feet (for example, the front foot) are shifting from side to side while the user is moving his or her hips up and down. For example, the terminal device 10A draws the auxiliary line VL_47B based on the results of estimating the positions of the heel and knee of the user's front foot before the user starts bending down.

The auxiliary line VL_47F is an auxiliary line for making the user feel whether the user's shoulders are shifting from side to side while the user is moving his or her hips up and down. The terminal device 10A draws the auxiliary line VL_47F in conjunction with the user's bending down motion.

The terminal device 10A displays, on the right side surface 4701C, an avatar AVT_47C corresponding to the user as well as auxiliary lines VL_47C and LN_47C that are fixedly displayed regardless of the user's movements.

The auxiliary line VL_47C is an auxiliary line that allows the user to perceive whether or not the user can lower the waist vertically without shaking back and forth while the user is moving the waist up and down. For example, the terminal device 10A draws the auxiliary line VL_47C based on the result of estimating the position of the user's pelvis, etc., before the user starts the motion of lowering the waist.

The auxiliary line LN_47C is used to make the user perceive whether the user is lowering the hips too much forward or backward (for example, whether the knees are too far forward) while the user is moving the hips up and down. This is an auxiliary line. For example, the terminal device 10A draws the auxiliary line LN_47C based on the result of estimating the position of the user's heel, etc., before the user starts the motion of lowering his or her hips.

The terminal device 10A draws an object OBJ_47E and an object OBJ_47F, which serve as a guide for the position and speed at which the user lowers his or her hips, in conjunction with the speed at which the user moves his or her hips up and down. These objects OBJ_47E and OBJ_47F move along the auxiliary line VL_47C in conjunction with the speed at which the user moves his or her hips up and down.

Object OBJ_47E is an object that allows the user to perceive the approximate position of the neck and shoulders while the user moves his/her waist up and down.

Object OBJ_47F is an object that allows the user to perceive the approximate position of the waist while the user moves the waist up and down.

As shown in FIG. 50, it is assumed that the user is leaning forward slightly in an attempt to lower his or her back. As shown in the figure, the terminal device 10A displays, on the right side 4701C, the estimated result of the actual position where the user is moving the body (the position of the neck, shoulders, head, etc.) with respect to the object OBJ_47E, which is a guideline for ideal movement. ) is drawn.

The terminal device 10A draws an object OBJ_47K that indicates the actual position where the user is moving his or her body (estimation result of the position of the waist, pelvis, etc.) with respect to the object OBJ_47F that serves as a guide for ideal movement.

The terminal device 10A allows the object OBJ_47E and the object OBJ_47J to be in a predetermined positional relationship (overlapping, within a certain distance, etc.) so that the user can easily perceive whether or not the user is vertically moving the waist up and down. In some cases, the display may be rendered in a different manner than in other cases. Similarly, the terminal device 10A may draw in a different display mode when the object OBJ_47F and the object OBJ_47K are in a predetermined positional relationship than when they are not. This makes it easy to perceive in real time whether the user is moving his or her hips up and down in an ideal manner, and whether the user is moving his or her body in accordance with ideal movement guidelines.

The terminal device 10A determines whether or not the front legs are being moved appropriately, and displays the determined result using the object OBJ_47P. For example, the terminal device 10A may determine whether the front legs are being moved appropriately as follows.
- Based on the angle of the user's front legs, such as the heel, knee, and pelvis, it determines whether the user is moving the front legs appropriately (for example, whether the user leans too far forward) - Parts of the user's lower body It is determined whether the estimated position of (heel, knee, etc.) and the auxiliary line LN_47C have a predetermined positional relationship. (For example, if the heel or knee position exceeds the auxiliary line LN_47C, you are leaning forward, etc.) Also, if the line segment (knee angle) connecting the heel position and the knee position exceeds the auxiliary line LN_47C and the specified If it is within the angle, you are lowering your hips well, etc.)

The terminal device 10A may specify the position where the object OBJ_47P is drawn based on the result of estimating the position of the user's body part. For example, the terminal device 10A may draw the object OBJ_47P based on the position of the user's knee. As a result, the object OBJ_47P is drawn while moving in real time along with the user's body movements, so it is possible to easily make the user perceive whether the user is moving his or her body appropriately.

The terminal device 10A determines whether or not the hind legs are being moved appropriately, and displays the determined result using the object OBJ_47N. As in the case of the front legs described above, it may be determined whether the hind legs are being moved appropriately.

The terminal device 10A may determine whether or not the front legs are moving from side to side in the front view 4701B, and may display the determination result using the object OBJ_47M. For example, based on the results of estimating the heel position and the knee position, the terminal device 10A, if the angle of the knee is not perpendicular (for example, if it is at a predetermined angle or more with respect to the auxiliary line VL_47E), It may also be determined that the front legs are moving left and right. In the example of FIG. 50, the front legs do not move left or right, and "OK" is displayed in object OBJ_47M.

As shown in FIG. 51, on the right side surface 4701C, the user is vertically moving his or her waist vertically up and down, and "OK" is displayed in object OBJ_47P and object OBJ_47N.

(3) Additional notes The matters explained in the above embodiments are added below.

(Additional note 1)
A program for operating a computer, the program includes a step of causing a processor of the computer to sequentially sense the movement of the user's body, and a real-time photograph of the user's appearance based on the sensing results. a step of presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the user's body surface rather than an image; and a step of associating the avatar with the avatar based on the sensing results and serving as a guide for the user's body movement. A program that executes the steps of specifying a position to draw a graphical object and displaying the graphical object at the specified position.
(Additional note 2)
The program according to appendix 1, wherein in the step of presenting the avatar, the avatar is presented to the user in a manner with less information than a live-action image.
(Additional note 3)
The program according to appendix 1 or 2, wherein in the step of presenting the avatar, the avatar is presented to the user each time a sensing result is obtained.
(Additional note 4)
In the step of presenting the avatar, based on the sensing results, an avatar depicting the user from a first direction and an avatar depicting the user from a second direction different from the first direction are both rendered side by side. The program according to any one of Supplementary Notes 1 to 3, which presents an avatar to a user by doing so.
(Appendix 5)
At the sensing step, the movement of the user's body is sensed as point cloud data, and in the step of presenting the avatar, the avatar is presented to the user by drawing the avatar as a collection of point cloud data. The program according to any one of 4 to 4.
(Appendix 6)
The program further instructs the processor to estimate the positions of a plurality of specific parts of the user based on the results of sensing the user's body movements, and to estimate the positions of the specific parts while drawing the avatar. 6. The program according to any one of appendices 1 to 5, which executes the step of evaluating how the user moves his or her body based on the estimation result of the position of a specific part without drawing the result.
(Appendix 7)
In the step of presenting the avatar, the drawing of the avatar is updated according to the sensing result of the user's body movement, and in the step of displaying the avatar, the graphic object is displayed at a fixed position regardless of the user's body movement. The program according to any one of 1 to 6.
(Appendix 8)
The program further directs the processor to the step of estimating the position of a specific body part of the user based on the result of sensing the body movement of the user; The program according to appendix 7, which executes the step of drawing the sensing history in chronological order along with the avatar regarding the position.
(Appendix 9)
In the identifying step, based on the result of sensing the user, it is specified that a linear graphic object is drawn at a fixed position parallel to the user's body axis regardless of the user's body movements, as described in Appendix 7. program.
(Appendix 10)
In the identifying step, based on the sensing result of the user's first posture before the user starts the specific action, a position where a graphical object is drawn that is a guide when the user performs the specific action is identified. , the program described in Appendix 1.
(Appendix 11)
The specific motion is a predetermined motion for measuring the movement of the user's body, and in the identifying step, a guideline for the movement of the user's body at a predetermined timing of the specific motion is determined based on the first posture. The program according to appendix 10, which specifies a position at which a graphic object is drawn.
(Appendix 12)
According to appendix 10, in the identifying step, a position for drawing a graphical object serving as a guide for at least one of the group consisting of front-back balance and left-right balance of the user's body is identified based on the first posture. program.
(Appendix 13)
According to appendix 10, in the identifying step, a position for drawing a graphical object that is a guideline for a second posture that the user may take while performing a specific action is specified based on the first posture. program.
(Appendix 14)
A specific action is an action that the user performs with both feet spread apart, and in the specified step, the position at which a graphic object that is a guide for the state in which the user steps in with his/her front feet is drawn is set to the first posture. The program according to appendix 13, specified based on the program.
(Appendix 15)
A step of estimating the position of the user's lower limbs based on the result of sensing the user's body movement is further executed, and in the identifying step, the position where the user steps with his/her legs apart is determined based on the estimated result. The program according to appendix 14, which specifies a position at which a graphical object serving as a reference is to be drawn.
(Appendix 16)
The step of estimating the position of a specific part of the user based on the result of sensing the movement of the user's body; The program according to appendix 13, which executes the step of evaluating whether or not the posture described in item 2 is properly performed.
(Appendix 17)
17. The program according to supplementary note 16, which, when evaluating that the second posture is properly performed in the evaluating step, presents a notification to that effect to the user.
(Appendix 18)
The specific motion is accompanied by a body movement performed by the user in the second posture, and in the evaluating step, the specific motion is performed in response to the evaluation that the second posture is properly performed. The program according to supplementary note 16, which starts measurement and presents to the user that the measurement is to be started.
(Appendix 19)
A step of estimating the positions of a plurality of specific parts of the user based on the results of sensing the user's body movements is further executed, and in the identifying step, the positions of the specific parts are estimated based on the estimated positions of the specific parts. The program according to supplementary note 1, which specifies a position at which a linear graphic object is drawn to make the user perceive whether a part can be moved appropriately during the movement of the user's body.
(Additional note 20)
The program according to appendix 19, in which, in the step of displaying, a line segment that is an extension of a line going from the surrounding parts of the specific part to the specific part is displayed as a graphic object together with the avatar.
(Additional note 21)
The program according to appendix 20, wherein in the displaying step, a graphic object having a larger size than the width of the line segment is drawn and displayed at the end point of the line segment.
(Additional note 22)
In the displaying step, a graphic object corresponding to the end point of the line segment is displayed so as to move in conjunction with the sensing result of the user's body movement, and the graphic object is fixedly placed regardless of the user's body movement. The program according to attachment 21, which changes and displays the display mode of the graphical object based on the positional relationship between the object and the graphical object corresponding to the end point.
(Additional note 23)
further performing and displaying the step of estimating the positions of a plurality of specific parts of the user based on the results of sensing the movement of the user's body, a graphical object indicating the inclination of the first part of the user; , the program according to appendix 19, which draws and displays graphical objects indicating the inclination of a second region different from the first region in association with the first region and the second region.
(Additional note 24)
A step of estimating the position of a specific part of the user based on the result of sensing the movement of the user's body is further executed, and in the identifying step, the position of the specific part estimated in the previous step and the shape of the surface of the user's body are The program according to appendix 19, which specifies a position at which a graphical object is drawn based on.
(Additional note 25)
The specific motion is a predetermined motion for measuring the movement of the user's body, and in the presenting step, an avatar is presented to the user each time a sensing result is obtained, and a predetermined motion that is a break between the specific motions is displayed. The program according to Supplementary Note 1, wherein the avatars at the predetermined timing are arranged and presented to the user by storing the sensing results of the user at the timing in the storage unit.
(Additional note 26)
In the step of presenting the avatar, a step of estimating the positions of at least three specific parts of the user based on the results of sensing the user's body movements is performed, and any one of the at least three specific parts is executed. determining a position at which to draw an object indicating the positional relationship of other specific parts in a horizontal plane as a reference, and displaying the object at the determined position; The program according to supplementary note 1, which specifies a position at which a graphical object is drawn in association with an avatar based on the estimation result.
(Additional note 27)
A program for operating a computer, the program includes the steps of: sequentially sensing a user's body movements in a processor of the computer; estimating the position of the three specific parts, and determines the position at which to draw a graphic object indicating the positional relationship of the other specific parts in the horizontal plane, with one of the at least three specific parts as a reference. A program that executes the steps of displaying a graphical object at the determined position.
(Additional note 28)
28. The program according to appendix 27, wherein in the step of displaying a graphical object, graphical objects representing a specific part that is not a reference are displayed as graphical objects each including a graphical object having the same or similar shape.
(Additional note 29)
A method for operating a computer, the method comprising: a processor of the computer;
Steps of sequentially sensing the movement of the user's body, and responding to the user in a manner that allows the user to perceive the shape of the user's body surface, rather than a real image of the user's appearance, based on the sensing results. a step of presenting the avatar to the user, a step of identifying a position to draw a graphical object that is a guide to the user's body movement in association with the avatar based on the sensing results, and a step of drawing the graphical object at the identified position. The steps to display and how to perform them.
(Additional note 30)
The information processing device includes a step in which the control unit sequentially senses the movement of the user's body, and a step in which the control unit sequentially senses the movement of the user's body, and based on the sensing result, takes a photograph of the user's appearance instead of a real image. presenting an avatar corresponding to the user in a manner that allows the user to perceive the shape of the user's body surface; and based on the sensing results, a graphic object is associated with the avatar and serves as a guide for the user's body movement. An information processing device that executes the steps of specifying a drawing position and displaying a graphic object at the specified position.

10 terminal device, 20 server, 30 measurement unit, 31 measurement device, 32 mat, 33 pole, 132 display, 181 user information, 182 measurement item information, 183 measurement history information, 184 friend list, 185 training item information, 281 user information Database, 282 Measurement item information, 283 Measurement history database, 284 Friend list, 285 Training item information, 287 Measurement result database, 288 Evaluation result database, 289 User questionnaire information, 290 Evaluation result database, 2601 Trainee information database, 2602 Trainer information database , 2603 Training history database

Claims (30)

A program for operating a computer, the program causing a processor of the computer to:
sequentially sensing the user's body movements;
Based on the sensing result, presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the user's body surface rather than a real image taken of the user's appearance;
Based on the sensing results, identifying a position to draw a graphic object that is a guide for the user's body movement in association with the avatar;
displaying the graphical object at the specified position;
A program to run. 2. The program according to claim 1, wherein in the step of presenting the avatar, the avatar is presented to the user in a manner with less information than the live-action image. 3. The program according to claim 1, wherein in the step of presenting the avatar, the avatar is presented to the user each time the sensing result is obtained. In the step of presenting the avatar,
Based on the sensing results, an avatar of the user drawn from a first direction and an avatar of the user drawn from a second direction different from the first direction are drawn side by side. 4. The program according to claim 1, wherein the program presents the avatar to the user. In the sensing step, sensing the user's body movement as point cloud data,
5. The program according to claim 1, wherein in the step of presenting the avatar, the avatar is presented to the user by drawing the avatar as a collection of the point cloud data. The program further causes the processor to:
estimating the positions of a plurality of specific parts of the user based on the results of sensing the user's body movements;
while drawing the avatar, evaluating how the user's body moves based on the estimation result of the position of the specific part without drawing the result of estimating the position of the specific part. The program according to any one of claims 1 to 5. In the step of presenting the avatar, updating the drawing of the avatar according to the sensing result of the user's body movement,
7. The program according to claim 1, wherein in the displaying step, the graphical object is displayed at a fixed position regardless of the movement of the user's body. The program further causes the processor to:
estimating the position of a specific part of the user based on a result of sensing the movement of the user's body;
8. The program according to claim 7, causing the program to execute the step of drawing the sensed history together with the avatar in chronological order regarding the position of a first part of the user whose body movement is to be measured. In the identifying step, based on the result of sensing the user, it is determined that the linear graphic object is drawn in a fixed position parallel to the user's body axis regardless of the user's body movement. , The program according to claim 7. In the identifying step, based on the sensing result of the first posture of the user before the user starts the specific action, the graphical object is drawn as a guide when the user performs the specific action. 2. The program according to claim 1, which specifies a location where the program is performed. The specific motion is a predetermined motion for measuring the movement of the user's body,
According to claim 10, in the identifying step, a position at which the graphical object is drawn, which is a guide for the movement of the user's body at a predetermined timing of the specific motion, is identified based on the first posture. program. In the identifying step, a position at which the graphical object is drawn is identified based on the first posture as a guide for at least one of a group consisting of a front-back balance and a left-right balance of the user's body. The program described in item 10. In the identifying step, a position where the graphical object is drawn, which is a guideline for a second posture that the user may take while the user is performing the specific action, is specified based on the first posture. , The program according to claim 10. The specific action is an action performed by the user with both feet spread apart,
14. The program according to claim 13, wherein, in the identifying step, a position at which the graphical object is drawn, which is a measure of the state in which the user steps in with his/her front foot, is identified based on the first posture. further performing the step of estimating the position of the lower limb region of the user based on the result of sensing the movement of the user's body;
15. The program according to claim 14, wherein, in the specifying step, a position where the graphical object is drawn is specified as a guideline for a position where the user steps with both feet apart, based on the estimated result. estimating the position of a specific part of the user based on a result of sensing the movement of the user's body;
A step of evaluating whether the user can appropriately perform the second posture based on a positional relationship between the estimated position of the specific part and the position of the graphic object is performed. The program described in 13. 17. The program according to claim 16, wherein when the evaluating step evaluates that the second posture is properly performed, the program presents the user with a notification to that effect. The specific action is accompanied by a body movement performed by the user in the second posture,
In the step of evaluating, in response to evaluating that the second posture is properly performed, starting measurement of the specific motion, and presenting to the user that the measurement is to be started. The program described in item 16. further performing the step of estimating the positions of a plurality of specific parts of the user based on the results of sensing the user's body movements;
In the identifying step, the linear figure is used to make the user perceive whether the specific part can be appropriately moved during the user's body movement based on the estimated position of the specific part. The program according to claim 1, which specifies a position at which an object is drawn. 20. The program according to claim 19, wherein in the displaying step, a line segment that is an extension of a line going from the surrounding parts of the specific part to the specific part is displayed as the graphical object together with the avatar. 21. The program according to claim 20, wherein in the displaying step, the graphical object having a size larger than the width of the line segment is drawn and displayed at the end point of the line segment. In the displaying step, the graphical object corresponding to the end point of the line segment is displayed so as to be moved in conjunction with the sensing result of the user's body movement;
The display mode of the graphical object is changed and displayed based on the positional relationship between the graphical object, which is fixedly arranged regardless of the movement of the user's body, and the graphical object corresponding to the end point. The program described in item 21. further performing the step of estimating the positions of a plurality of specific parts of the user based on the results of sensing the user's body movements;
In the step of displaying, the graphic object indicating the inclination of the first body part of the user and the graphic object indicating the tilt of a second body part different from the first body part, The program according to claim 19, wherein the program draws and displays each of the parts in association with the second part. further performing the step of estimating the position of a specific part of the user based on the result of sensing the movement of the user's body;
20. The program according to claim 19, wherein in the specifying step, a position at which the graphic object is drawn is specified based on the previously estimated position of the specific part and the shape of the user's body surface. The specific motion is a predetermined motion for measuring the movement of the user's body,
In the presenting step, each time the sensing result is obtained, the avatar is presented to the user, and the sensing result of the user at a predetermined timing that is a break between the specific actions is stored in a storage unit. 2. The program according to claim 1, wherein the avatars at the predetermined timing are arranged and presented to the user. In the step of presenting the avatar,
estimating the positions of at least three specific parts of the user based on the results of sensing the movement of the user's body;
Determining a position to draw an object indicating the positional relationship of other specific parts in a horizontal plane with respect to any one of the at least three specific parts as a reference, and displaying the object at the determined position. let me,
2. The program according to claim 1, wherein in the specifying step, a position at which the graphic object is drawn in association with the avatar is specified based on the estimation results of the positions of the at least three body parts. A program for operating a computer, the program causing a processor of the computer to:
sequentially sensing the user's body movements;
estimating the positions of at least three specific parts of the user based on the results of sensing the movement of the user's body;
Determining a position to draw a graphic object indicating the positional relationship of other specific parts in a horizontal plane with respect to any one of the at least three specific parts, and displaying the graphic object at the determined position; A program to run. In the step of displaying the graphical object,
28. The program according to claim 27, wherein the graphical objects representing the specific parts that are not the reference are displayed as graphical objects each including a graphical object having the same or similar shape. A method for operating a computer, the method comprising: a processor of the computer:
sequentially sensing the user's body movements;
Based on the sensing result, presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the user's body surface rather than a real image taken of the user's appearance;
Based on the sensing results, identifying a position to draw a graphic object that is a guide for the user's body movement in association with the avatar;
displaying the graphical object at the specified position;
How to do it. An information processing device, wherein the information processing device includes a control unit that includes:
sequentially sensing the user's body movements;
Based on the sensing result, presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the user's body surface rather than a real image taken of the user's appearance;
Based on the sensing results, identifying a position to draw a graphic object that is a guide for the user's body movement in association with the avatar;
displaying the graphical object at the specified position;
An information processing device that executes.