JP2022187952A - Program, method, and information processing device - Google Patents

Abstract

To provide another technology for a user himself/herself to easily check his or her body motion.SOLUTION: There is provided a program for causing a computer to be operated. The program causes a processor of the computer to execute the steps of: sequentially sensing a body motion of a user; presenting to the user an avatar corresponding to the user in a mode to cause the shape of the body surface of the user to be perceived instead of an actually taken image when the user's external appearance is photographed, on the basis of a result of the sensing; specifying a position for drawing a graphic object as an indication of the body motion of the user in association with the avatar on the basis of the result of the sensing; and displaying the graphic object at the specified position.SELECTED DRAWING: Figure 37

Description

The present disclosure relates to programs, methods, and information processing apparatuses.

2. Description of the Related Art A technique is known in which a user's movement is captured by a camera or the like, and the user's movement is evaluated based on the captured moving image.

Japanese Patent Application Laid-Open No. 2016-81504 discloses a technology related to a device that recognizes the type of competition based on the skeleton of the human body using a depth camera, automatically evaluates the movement of the user using this, and determines the presence or absence of a step increase. Are listed.

JP 2016-81504 A

There is a need for another technique that allows users to easily check their own body movements.

According to one embodiment, a program is provided for operating a computer. The program causes the computer processor to sequentially sense the movement of the user's body, and based on the sensing results, perceive the shape of the user's body surface instead of the actual image when the user's appearance is captured. a step of presenting an avatar corresponding to the user to the user in a manner that allows the user to see the avatar; and a step of identifying a position for drawing a graphical object that serves as a guideline for movement of the user's body in association with the avatar based on the sensing result. , and displaying the graphic object at the identified position.

According to the present disclosure, while the user recognizes the avatar presented to the user based on the sensing result of the user's body movement, the user can also recognize the graphical object that serves as a guideline for the user's body movement. Therefore, the user himself/herself can easily check the movement of the body.

1 is a diagram showing the overall configuration of a measurement system 1; FIG. 2 is a block diagram of a terminal device 10 configuring the measurement system 1 of Embodiment 1. FIG. 3 is a diagram showing a functional configuration of a server 20; FIG. 3 is a diagram showing data structures of a user information database 281, measurement item information 282, a measurement history database 283, and a friend list 284 stored by the server 20. FIG. The measurement system 1 shows the phase in which the user measures body movements. 4 is a diagram showing details of a mat 32; FIG. 6 is a flow chart showing a process in which the terminal device 10 performs a ROM check and updates the measurement history. 4 is a flow chart showing a process in which the terminal device 10 evaluates the range of motion of the joint by measuring the user's stepping motion. 4 is a flow chart showing a process in which the terminal device 10 evaluates the joint range of motion by measuring the user's upper body twisting motion. FIG. 10 is a diagram showing an example of the notification content of the terminal device 10 in the phase of measuring the movement of the user's body; A situation in which a user's walking motion is measured by the measurement system 1 according to the second embodiment is shown. Each part of the user's body specified by the terminal device 10 is shown based on the measurement result of the measurement device 31 . FIG. 10 is a diagram showing data structures of a measurement result database 287 and an evaluation result database 288 stored in the server 20 according to the second embodiment; 9 is a flow chart showing a process of measuring and evaluating a user's walking motion by the terminal device 10 in the second embodiment. FIG. 10 is a diagram showing an output example when the terminal device 10 outputs a measurement result of measuring the user's walking motion and an evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. FIG. 10 is a diagram showing an output example when the terminal device 10 outputs a measurement result of measuring the user's walking motion and an evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. FIG. 4 is a diagram showing an example in which the terminal device 10 outputs an evaluation result of the load on the knees in walking motion of the user. This corresponds to step S1409 in FIG. 10 is a flow chart showing a process of measuring and evaluating the balance of the user's trunk by the terminal device 10 in the third embodiment. FIG. 10 is a diagram showing an example in which the terminal device 10 outputs a measurement result of measuring the balance of the user's trunk and an evaluation result of the balance of the trunk in the third embodiment. FIG. 10 is a diagram showing data structures of various data stored in a server 20 according to the third embodiment; FIG. It is a screen example of a situation in which the terminal device 10 presents possible troubles to the user's body based on the measurement results and evaluation results of the user's bodily movements, and proposes an exercise menu corresponding to the troubles. 4 shows the processing flow of each device in the fourth embodiment. It is a figure which shows the data structure of the various data which the server 20 memorize|stores in the modification of 4th Embodiment. It is a figure which shows the flow of the operation|movement in the modification of 4th Embodiment. In a phase where the terminal device 10 presents possible physical concerns of the user based on the measurement results and evaluation results of the user's body movements and proposes an exercise menu corresponding to the concerns, the user answers a questionnaire. It is a screen example of a situation. FIG. 14 is a diagram showing data structures of a trainee information database 2601, a trainer information database 2602, and a training history database 2603 stored by the server 20 in the fifth embodiment. It is a flowchart which shows a series of processes which sense a 1st user's body motion as point cloud data, and transmit the information based on the 1st user's point cloud data obtained by sensing to a 2nd user's terminal. 4 is a flow chart showing a flow of processing for receiving an input of a trainer's voice from a trainer's terminal and transmitting it to a trainee's terminal. Sensing the movement of the first user's body as point cloud data, and specifying the instruction content for the first user based on the information based on the first user's point cloud data obtained by sensing and the trained model. 4 is a flow chart showing a series of processes to be performed. When the first user's terminal device 10A senses the movement of the first user's body as point cloud data and transmits information based on the first user's point cloud data obtained by sensing to the second user's terminal is a screen example of The first user including the overall configuration when sensing the first user's body movement as point cloud data and transmitting information based on the first user's point cloud data obtained by sensing to the second user's terminal and a screen example of a second user. It is an example of a screen when sensing the body movements of a plurality of users as point cloud data and transmitting information based on the point cloud data of each user obtained by sensing to a defined virtual space. 10A is a diagram showing a concept of processing for drawing an auxiliary line based on the estimation result of the user's posture by the terminal device 10A; FIG. 4 shows an example of a screen presented to the user. 10A is a diagram showing an example of a screen used by the terminal device 10A to adjust the tilt in the front, rear, left, and right directions when the user is standing. FIG. FIG. 10 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guideline for user's body movement based on the result of estimating the position of a specific part of the user; 10A is a diagram showing an example of a screen on which the terminal device 10A causes the user to perform a side-lifting motion. FIG. FIG. 10 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guideline for user's body movement based on the result of estimating the position of a specific part of the user; It is a figure which shows the process in which the user is moving the body. FIG. 10 is a diagram showing a state in which the user raises both arms at the peak timing of side elevation. FIG. 10 is a diagram showing an example in which the user saves the sensing results at the peak timing of the side lifting motion and displays them as comparable avatars. 10A is a diagram showing an example of a screen for causing the user to crouch. FIG. FIG. 10 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guideline for user's body movement based on the result of estimating the position of a specific part of the user; It is a figure which shows the process in which the user is moving the body. FIG. 10 is a diagram showing a fully crouched state, which is the peak timing of the user's crouching motion. FIG. 10 is a diagram showing an example in which the user saves the sensing results at the peak timing of the crouching motion and displays them as comparable avatars. 10A is a diagram showing an example of a screen for the terminal device 10A to cause the user to take a stepping motion and lower back. FIG. FIG. 10 is a diagram showing a screen example of a situation that presents to the user that the user's initial posture is appropriate; FIG. 10 illustrates the phase of initiating measurement of a user's actions; FIG. 10 is a diagram showing a situation in which it is determined whether or not a user's action is appropriate, and the determined result is presented to the user; FIG. 10 is a diagram showing a situation in which it is determined whether or not a user's action is appropriate, and the determined result is presented to the user;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<First embodiment>
<Overview>
In the following embodiments, by measuring the balance and movement of the user's body, the suppleness of the user's body is evaluated, and an exercise menu is provided so that the user can acquire a beautiful posture (for example, a beautiful walking posture). A measurement system 1 for providing will be described.

If the walking posture of the user is not in order, the user will perform a motion that is not necessarily required for walking, so the user's strength will be unnecessarily consumed, and fatigue will easily accumulate. By improving the user's walking posture, the user can move the body functionally, and as a result, the user can walk with less fatigue. Therefore, the measurement system 1 (1) measures the motion of the user during walking to evaluate the walking posture (evaluates whether the user can walk beautifully and with good balance); (3) Evaluating the user's muscle strength and providing a training menu. .

When the user is able to walk in a well-balanced manner (the way the body moves left and right is not biased, the movement of the muscles during walking is ideal, etc.), the way the user moves the body becomes functional. Here, the functional way of moving means that the user can move the muscles for walking without consuming unnecessary power. In other words, the user moves the muscles to obtain the driving force for walking, and the movement of the muscles leads to the driving force with high efficiency. As a result, it is possible to give the impression that the user is walking beautifully. It is effective to evaluate the joint range of motion in order to allow the user to learn such a well-balanced walking style. To provide a user with muscle training necessary for making a walking posture look beautiful on the premise.

As described above, the measurement system 1 provides measurement (checking) of the user's body movement and exercises for moving the user's body.

The measurement system 1 can be installed, for example, in an office of a business company, in a facility of a commercial facility, in a store space, in a studio, in an individual's home, or the like. For example, a business company may introduce the measurement system 1 in order to further encourage employees to manage their health. Also, by deploying the measurement system 1 in stores of commercial facilities, health management services may be provided to consumers.

<1 Configuration diagram of the entire system>
FIG. 1 is a diagram showing the overall configuration of a measurement system 1. As shown in FIG.

As shown in FIG. 1, the measurement system 1 includes a plurality of measurement units (measurement units 30A, 30B, and 30C in the illustrated example) configured by various sensors that measure the movement of the user's body. Each measurement unit is sometimes referred to as a “measurement unit 30”), and a terminal device 10 that controls the operation of the measurement unit (in the illustrated example, the terminal device 10 is shown). When describing a device, it may be described as "terminal device 10A", "terminal device 10B", etc. In addition, each element (such as a processor) that constitutes a terminal device may also be described in the same way.) and a server 20 that accumulates the measurement results of the user's body movements.

The measurement unit 30 is connected to the terminal device 10 by wire or wirelessly. As shown in the figure, a plurality of measurement units 30 may be connected to one terminal device 10, or the terminal device 10 may be connected to each measurement unit 30 (that is, one measurement unit 30 is connected to one measurement unit 30). It is also possible to prepare two terminal devices 10 and measure users.

The measurement unit 30 includes a measurement device (in the illustrated example, measurement devices 31A, 31B, and 31C are shown. These may be collectively referred to as “measurement device 31”), and a mat (in the illustrated example, , mats 32A, 32B, and 32C, which may be collectively referred to as “mats 32”) and three-dimensional poles (poles 33A, 33B, and 33C are shown in the illustrated example). These are sometimes collectively referred to as “pole 33”).

The measuring device 31 is for measuring the movement of the user's body, and detects the distance and direction to the measuring object by, for example, an RGB-D camera (depth sensor). The measurement device 31 may also include a microphone array or the like to detect the user's voice, the direction from which the voice arrives, and the like. For example, the measurement device 31 may be a device such as Kinect (registered trademark) that can accept user gestures and voice recognition operations. By detecting the user's body, the measuring device 31 detects the position of the user and the positions of the joints of the user's body. Thereby, the measuring device 31 sets a virtual bone (bone) that virtually connects each joint that constitutes the user, and measures the movement of the user's body. In addition, the measuring device 31 may be a terminal such as a smart phone in addition to the Kinect (registered trademark). For example, when a smartphone is provided with a depth camera, the smartphone can acquire the user's body as point cloud data (point cloud data).

In addition, even if the measuring device 31 does not have a function to measure the distance, the measurement device 31 can identify the subject based on the captured image and use a learned model for estimating the distance between the subject and the camera. The distance to the subject (for example, the distance to each joint of the subject) may be obtained based on the above. Also, even if the measuring device 31 cannot measure the distance (such as with an RGB camera), each part of the subject may be specified based on the captured image to estimate the skeleton (bone). Alternatively, a plurality of cameras may be provided as the measuring device 31 to obtain captured images with parallax, thereby estimating the distance to the subject.

The mat 32 is for displaying an indication of the user's standing position in order to measure the movement of the user's body. The mat 32 is arranged at a position having a predetermined positional relationship with respect to the measuring device 31 . That is, the mat 32 is placed within a range where the measurement device 31 can measure the movement of the user's body. The mat 32 includes a plurality of patterns for guiding the user's standing position, as will be described later. While viewing the pattern, the user adjusts the position of the foot or makes a stepping motion.

The poles 33 are used with the mat 32 to measure the motion of the user's body, eg, the user's twisting motion. The poles 33 are arranged on the mat 32 at positions associated with patterns indicating the standing positions on which the user's feet are placed. While keeping both feet on the mat 32 (that is, without moving the standing position of both feet), the user twists the body and touches the pair of poles placed on both sides of the mat 32 with the hands. Therefore, the mat 32 is used as a target point for moving the arm in order for the user to make a twisting motion. The mat 32 may be set to a height corresponding to the height of the user so that the user can hold it by hand when twisting the body.

As described above, the combination of the measuring device 31, the mat 32, and the poles 33 that constitute the measuring unit 30 has been described. In the following description, it is assumed that one user uses one measurement unit 30 . However, one measuring device 31 may measure the body movements of a plurality of users. In other words, a plurality of users may move their bodies within the measurement range of one measurement device 31 .

The terminal device 10 and the server 20 are connected for communication via the network 80 . Further, in the present embodiment, each device (terminal device, server, etc.) can also be understood as an information processing device. In other words, a set of devices can be understood as one "information processing device", and the measurement system 1 may be formed as a set of a plurality of devices. The method of distributing a plurality of functions required to implement the measurement system 1 according to this embodiment to one or a plurality of pieces of hardware depends on the processing capability of each piece of hardware and/or the specifications required for the measurement system 1. can be determined as appropriate in view of the above.

The terminal device 10 is a device operated by each user. The terminal device 10 may be, for example, a stationary PC (Personal Computer) or a laptop PC. In addition, the terminal device 10 is implemented by a mobile terminal such as a smart phone or a tablet compatible with a mobile communication system. As shown as terminal device 10B in FIG. The server 20 includes a communication IF 22 , an input/output IF 23 , a memory 25 , a storage 26 and a processor 29 .

The terminal device 10 is communicably connected to the server 20 via the network 80 . The terminal device 10 is compatible with communication standards such as 5G and LTE (Long Term Evolution), and wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. is connected to the network 80 by communicating with the

The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 communicates with an external device. The input device 13 is an input device (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving an input operation from a user. The output device 14 is an output device (display, speaker, etc.) for presenting information to the user. The memory 15 temporarily stores programs and data processed by the programs, and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for storing data, and is, for example, a flash memory or a HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set described in a program, and is composed of arithmetic units, registers, peripheral circuits, and the like.

As illustrated, the terminal device 10 is connected to the measurement unit 30 by wire or wirelessly. By measuring the movement of the user's body with the measurement unit 30, it is possible to evaluate the flexibility of the user's body and the extent of the joint range of motion.

The server 20 manages information of each user. The server 20 manages, as user information, parameters (such as height) that are referred to when measuring the body movements of each user.

Specifically, the server 20 performs processing for promoting interaction between users, such as user matching and the like, in order to assist a plurality of users in moving their bodies for health management and in measuring their body movements. I do. The server 20 also performs transmission and reception of messages between users.

The server 20 may manage virtual points owned by each user. For example, suppose there is an in-house point system that each employee can use like virtual currency within a business company. The points can be used for in-house purchase processing (can be used for payment at in-house vending machines, etc.) or given to other employees, like virtual currency. For example, the server 20 may give the points to the user when the user measures with the measurement system 1, or the user measures with the measurement system 1 together with other users (for example, the same The point may be given to the user when the measurement is performed at the same place, the measurement is performed at the same place at the same time, etc.), or the point is given when the evaluation result of the joint range of motion exceeds a predetermined value. good too.

The communication IF 22 is an interface for inputting and outputting signals for the server 20 to communicate with an external device. The input/output IF 23 functions as an interface with an input device for receiving input operations from the user and an output device for presenting information to the user. The memory 25 temporarily stores programs and data processed by the programs, and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for storing data, such as a flash memory or HDD (Hard Disc Drive). The processor 29 is hardware for executing an instruction set described in a program, and is composed of arithmetic units, registers, peripheral circuits, and the like.

<1.1 Configuration of terminal device 10>
FIG. 2 is a block diagram of the terminal device 10 configuring the measurement system 1 of the first embodiment. As shown in FIG. 2, the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to each antenna, and an operation unit. a reception unit 130 (including a touch-sensitive device 131; in addition, it may include various pointing devices such as a mouse and a touchpad; it may also include a keyboard and a gesture detection device); , an audio processing unit 140 , a microphone 141 , a speaker 142 , a position information sensor 150 , a camera 160 , a storage unit 180 and a control unit 190 . The terminal device 10 also has functions and configurations not particularly shown in FIG. 2 (for example, a battery for retaining power, a power supply circuit for controlling power supply from the battery to each circuit, etc.). As shown in FIG. 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

The antenna 111 radiates a signal emitted by the terminal device 10 as radio waves. Antenna 111 also receives radio waves from space and provides a received signal to first wireless communication section 121 .

The antenna 112 radiates a signal emitted by the terminal device 10 as radio waves. Antenna 112 also receives radio waves from space and provides a received signal to second wireless communication section 122 .

The first wireless communication unit 121 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 111 so that the terminal device 10 communicates with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 112 so that the terminal device 10 communicates with other wireless devices. The first radio communication unit 121 and the second radio communication unit 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high frequency circuit, and the like. The first radio communication unit 121 and the second radio communication unit 122 perform modulation/demodulation and frequency conversion of radio signals transmitted/received by the terminal device 10 and provide received signals to the control unit 190 .

The operation accepting unit 130 has a mechanism for accepting user input operations. Specifically, operation accepting unit 130 includes a touch-sensitive device 131 or the like as a pointing device. The touch-sensitive device 131 receives input operations of the user of the terminal device 10 . The touch-sensitive device 131 detects the user's touch position on the touch panel by using, for example, a capacitive touch panel. The touch-sensitive device 131 outputs a signal indicating the user's touch position detected by the touch panel to the control unit 190 as an input operation. The terminal device 10 may communicate with the measuring device 31 and receive the measurement result of the measuring device 31 to receive the user's input operation. For example, when the measurement device 31 acquires point cloud data, the user's body and each joint that constitutes the user's body are detected based on the point cloud data, and based on the detection result, the user's gesture is used as an input. accept. Gestures that can be input by the user include body movements such as user's hand movements and head movements. For example, the operation reception unit 130 may detect that the user takes a predetermined pose (raise both hands horizontally to take a “T”-shaped pose), or may detect that the user has made the hand into a predetermined shape. It is good also as detecting that it does.

The display 132 displays data such as images, moving images, and text under the control of the control unit 190 . The display 132 is implemented by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The terminal device 10 and the output device (display 132, speaker 142) may be implemented as separate devices.

The audio processing unit 140 modulates and demodulates an audio signal. Audio processing section 140 modulates the signal supplied from microphone 141 and supplies the modulated signal to control section 190 . Also, the audio processing unit 140 provides an audio signal to the speaker 142 . The audio processing unit 140 is implemented by, for example, a processor for audio processing. Microphone 141 receives a voice input and provides voice signal corresponding to the voice input to voice processing section 140 . The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10 .

The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. The satellite positioning system receives signals from at least three or four satellites, and detects the current position of the terminal device 10 equipped with a GPS module based on the received signals. For example, in the measurement system 1, when recording the place where the user performed the measurement, the detection result of the position information sensor 150 (latitude/longitude information, facility information specified by latitude/longitude and map information, associated with the facility) (such as information on the office where the measurement is performed) may be stored in the server 20 in association with the measurement result. In other words, the terminal device 10 may determine the location where the user performed the measurement based on the detection result of the position information sensor 150 without the user's operation of inputting the location.

The camera 160 is a device for receiving light with a light receiving element and outputting it as a photographed image. The camera 160 is, for example, a depth camera that can detect the distance from the camera 160 to the subject. In this embodiment, the measuring device 31 may be realized by the camera 160 of the terminal device 10. FIG. In other words, in addition to realizing the terminal device 10 and the measuring device 31 as separate devices, the terminal device 10 may be equipped with the functions of the measuring device 31 .

The storage unit 180 is configured by, for example, a flash memory or the like, and stores data and programs used by the terminal device 10 . In one aspect, storage unit 180 stores user information 181 , measurement item information 182 , measurement history information 183 , friend list 184 , and training item information 185 . The details of these will be described later, but the outline is as follows. The terminal device 10 may acquire these pieces of information from various databases of the server 20, which will be described later.

The user information 181 is user information in the measurement system 1 . The user information includes information for identifying the user, user's name, user's affiliation, user's physical information (height, weight, age, etc.) and the like.

The measurement item information 182 is information of each measurement item when ROM check is performed. The measurement item information 182 includes voice guidance information to be output to the user for each measurement item.

The measurement history information 183 indicates the history of the measurement result of measuring the user's body movement.

The friend list 184 shows information of each user registered as a friend between the users. Messages may be sent and received between users registered as friends.

The training item information 185 includes information on each training item when a muscle training menu is provided to the user. For example, it includes the name of the training, and video and audio information that guides the user to exercise. The terminal device 10 uses the measuring device 31 to measure the movement of the user's body based on the video and audio that serve as a guide, and sequentially displays the measurement results on the display 132 . On the display 132, the terminal device 10 compares the ideal way of moving the body with the measurement results (movements of each joint) when the user is moving the body, thereby obtaining the ideal way of moving the body. The user may be prompted to do so.

The control unit 190 controls the operation of the terminal device 10 by reading programs stored in the storage unit 180 and executing instructions included in the programs. The control unit 190 is, for example, an application processor. The control unit 190 functions as an input operation reception unit 191, a transmission/reception unit 192, a data processing unit 193, and a notification control unit 194 by operating according to a program.

The input operation reception unit 191 performs processing for receiving a user's input operation to the input device such as the touch sensitive device 131 . The input operation reception unit 191 determines whether the user's operation is a flick operation, a tap operation, or a drag (swipe) operation based on information on coordinates at which the user touches the touch-sensitive device 131 with a finger or the like. The type of operation, such as whether it is an operation or not, is determined.

The transmitting/receiving unit 192 performs processing for the terminal device 10 to transmit/receive data to/from external devices such as the server 20 and the measurement unit 30 according to a communication protocol.

The data processing unit 193 performs processing according to a program to perform computation on data input by the terminal device 10, and outputs the computation result to a memory or the like.

The notification control unit 194 performs processing for displaying a display image on the display 132, processing for outputting sound from the speaker 142, and the like. For example, the notification control unit 194 outputs audio guidance during a ROM check, and provides audio guidance and a video showing model body movements to the user when providing a training menu for exercise. Present.

<1.2 Functional Configuration of Server 20>
FIG. 3 is a diagram showing a functional configuration of the server 20. As shown in FIG. As shown in FIG. 3 , the server 20 functions as a communication section 201 , a storage section 202 and a control section 203 .

The communication unit 201 performs processing for the server 20 to communicate with an external device such as the terminal device 10 .

The storage unit 202 stores data and programs used by the server 20 . The storage unit 202 stores a user information database 281, measurement item information 282, a measurement history database 283, a friend list 284, and the like.

The user information database 281 shows information of each user who performs measurement using the measurement system 1 . Details will be described later.

The measurement item information 282 indicates information on each of a plurality of measurement items. Details will be described later.

The measurement history database 283 shows the history of measurements made by each user. Details will be described later.

The friend list 284 is a database for holding information on users registered as friends by each user.

The training item information 285 includes information on each training item when providing a training menu for exercise to the user. The training menu includes information on a plurality of training items according to the load on the user's body, the part of the body to be trained, and the like.

The control unit 203 exhibits functions shown as various modules when the processor of the server 20 performs processing according to a program.

The operation content acquisition module 2041 acquires the user's operation content. The operation content acquisition module 2041 performs, for example, an operation for the user to acquire past measurement results by the measurement unit 30 and an evaluation result of the user's body movement, an operation for transmitting and receiving messages with other users, and a measurement by the measurement unit 30. Accepts operations, etc. for applying to participate in measurement when there is an opportunity for

The reception control module 2042 controls processing for the server 20 to receive a signal from an external device according to a communication protocol.

The transmission control module 2043 controls the processing by which the server 20 transmits signals to external devices according to the communication protocol.

The matching module 2044 matches users together. The matching module 2044 may, for example, detect that multiple users have taken measurements using the measurement system 1 at the same measurement venue, match the multiple users, and prompt them to register as friends. In addition, the matching module 2044 may extract and match users with similar histories of measurements performed by the measurement system 1 , or extract and match users with similar ROM check results by the measurement unit 30 . It is also possible to let By matching a plurality of users and urging them to register as friends in this manner, it is possible to encourage a plurality of users to train together with the aim of achieving a beautiful posture. can support

The measurement progress module 2045 progresses the process of measuring the movement of the user's body by the measurement unit 30 . The measurement progress module 2045 reads out each measurement item shown in the measurement item information 282, and sequentially outputs to the user an audio guide or the like for measurement according to the measurement item. For example, the server 20 may transmit each measurement item to the terminal device 10 and cause the terminal device 10 to sequentially measure the movement of the user's body according to each measurement item.

The training progress module 2046 progresses the process of providing the user with a training menu by video and audio. The training progress module 2046 reads out each training item indicated in the training item information 285 and sequentially outputs audio guidance, video guidance, etc. for training according to the training item to the user.

<2 Data structure>
FIG. 4 shows the data structures of the user information database 281, measurement item information 282, measurement history database 283, and friend list 284 stored by the server 20. As shown in FIG.

As shown in FIG. 4, each record of the user information database 281 includes an item "user identification information (user ID)", an item "user name", an item "gender", an item "height", and an item " weight", item "affiliation", item "evaluation value", item "company point", and the like.

The item “user identification information (user ID)” is information for identifying each user.

The item "user name" is the name set by the user.

The item "gender" indicates the gender of the user. For example, the user may set the gender when registering user information to start using the measurement system 1 .

The item "height" indicates the user's height as information to be referred to when measuring the user's body movement.

The item "weight" indicates information on the weight of the user. In cases such as when weight information is not referred to when measuring the user's body movement, data for the item "weight" may not be retained.

The item "affiliation" indicates information on the user's affiliation, such as when the user belongs to an organization. For example, the item "Affiliation" includes information such as company name, organization name, and office name. The item "affiliation" may include information on the location of the office where the user works.

The item "evaluation value" includes information on an evaluation value evaluated based on the measurement results of the ROM check performed by the user in the measurement system 1, and an evaluation value based on the measurement results of the user's walking posture. The evaluation value includes, for example, classification result information when the terminal device 10 classifies the user's ROM check result based on the measurement result by the measurement unit 30 .

The item "in-house points" indicates the amount of points held by the user when there are virtual points used in the operating company to which the user belongs.

Each record of the measurement item information 282 includes the item “measurement item number (measurement item No.)”, the item “measurement site”, the item “guidance voice”, the item “foot position on the mat”, and the item “ part of the body to be measured".

The item "measurement item number (measurement item No.)" is information for identifying each measurement item.

The item "measurement site" indicates the site of the body to be measured. For example, there are the following parts as targets for evaluating joint range of motion.

(i) Measurement of Upper Limb Scapula: Directions of movement include flexion, extension, fist raising, and pulling down.
・Shoulder: Movement directions include flexion, extension, abduction, adduction, external rotation, and internal rotation.
・ Elbow: There are flexion and extension as motion directions.
・ Forearm: There are pronation and supination as directions of movement.
・ Hands: Movement directions include flexion, extension, flexion, and ulnar flexion.

(ii) Measurement of fingers - Thumb: Movement directions include flexor side abduction, ulnar side adduction, volar side abduction, volar side adduction, flexion, and extension.
・ Fingers: Movement directions include flexion, extension, abduction, and adduction.

(iii) Measurement of Lower Limbs · Crotch: Movement directions include flexion, extension, abduction, adduction, external rotation, and internal rotation.
・ Knee: Movement directions include flexion and extension.
・ Legs: Movement directions include flexion and extension.
・Foot: Movement directions include rolling, rolling, abduction, and adduction.
・ Thumb: Movement directions include flexion and extension.
・ Toes: Movement directions include flexion and extension.

(iv) Measurement of Trunk Neck: Movement directions include flexion, extension, rotation (left rotation, right rotation), and lateral bending (left bending, right bending).
・ Thoracic and lumbar region: Movement directions include flexion, extension, rotation (left rotation, right rotation), and lateral bending (left bending, right bending).

The item "guidance voice" indicates the content of the voice to be notified to the user when performing the measurement indicated in the measurement item. As a voice guide, there is a voice when the user does not put his/her feet on the prescribed positions on the mat 32 (for example, "Please place both feet on the prescribed positions on the mat." , the terminal device 10 detects that the user has placed his/her feet on the specified position, and prompts the user to move the body (for example, "Twist your body while placing both feet on the specified position on the mat"). , and a voice when the measurement data of the user's body movement is obtained based on the measurement result of the measurement device 31 (for example, "It was measured normally").

The item “position of feet on mat” includes information on where the user's feet are placed on the mat 32 (positions where the user places their feet on the mat 32) when performing the measurement. For example, the terminal device 10 detects whether the user's feet are placed on the specified positions on the mat, based on the information in the item "feet positions on the mat". By this detection, the terminal device 10 outputs to the user a sound indicating the content of the body motion for measurement.

The item “body part to be measured” indicates the part of the user's body that is to be measured in the measurement item.

Each record in the measurement history database 283 includes the item “measurement number (measurement No.)”, the item “date and time”, the item “user identification information (ID)”, the item “measurement location”, and the item “measurement data”. and the item "evaluation result". The server 20 receives from the terminal device 10 the result of measuring the user's body movement. The server 20 updates the record of the measurement history database 283 based on the information received from the terminal device 10 .

The item “measurement number (measurement No.)” indicates each measurement result when the user measures the movement of the body with the measurement unit 30 .

The item “date and time” indicates the timing at which the measurement was performed by the measurement unit 30 by the user. .

The item “user identification information (ID)” indicates information identifying the user whose body movement was measured by the measurement unit 30 .

The item “measurement place” indicates the place where the user performed the measurement with the measurement unit 30 . When the server 20 receives the measurement result from the terminal device 10, the server 20 refers to the user information associated with the measurement result, and the user performs the measurement based on the information associated with the user (user information database 281). You may specify the location where For example, based on the information on the user's affiliation shown in the user information database 281, the information on the place where the relevant affiliation is located may be used as the place where the measurement was performed by the measurement unit 30. FIG. Alternatively, the position of the terminal device 10 may be set as the location where the measurement is performed by the measurement unit 30 by the terminal device 10 acquiring the position information by the position information sensor 150 or the like.

The item “measurement data” indicates the measurement result of the user's body movement measured by the measurement unit 30 . For example, when the user's body movements are measured as time-series point cloud data, the time-series point cloud data may be the measurement results in the item "measurement data." In addition, with respect to the user's body measured by the measuring device 31, the terminal device 10 identifies the positions of the user's joints based on the measurement results, and the movement of each joint is recorded as the measurement result of the user's body movement in the server 20. It may be held. For example, the terminal device 10 can measure time-series movements of the user's joints by detecting specific parts of the user's body, such as shoulders and elbows, based on captured images of the user's body.

The item "evaluation result" indicates the evaluation result of evaluating the level of the user's body movement based on the measurement result indicated in the item "measurement data". For example, when the terminal device 10 holds the time-series data of the user's joint movements in the item "measurement data", the terminal device 10 evaluates the user's body movements by comparing the data with the reference body movement data. do. For example, the terminal device 10 may hold, as reference data, multiple stages of data according to the extent to which the user's body movement is favorable, thereby determining the most similar movement to the measurement result of the user's body movement. Data to be referenced can be specified.

Each of the records in the friend list 284 includes items such as the item “user 1”, the item “user 2”, the item “friend registration date”, the item “measurement history at the same timing”, and the item “message transmission/reception history”. include.

The item "user 1" indicates one of the users who are mutually registered as friends.

The item "user 2" indicates the other user among the users who are mutually registered as friends.

The item "friend registration date" indicates the date and time when the users were registered as friends.

The item “history of measurements at the same timing” indicates histories of measurements performed by each user registered as a friend while satisfying at least one of the same location or the same timing. For example, when the measurement unit 30 is arranged in an office space of a business establishment, a plurality of users may perform measurements during a certain period of time, and the plurality of users may perform measurements at the same timing. . The item “history of measurements at the same timing” may include information on locations where measurements were taken by a plurality of users. For example, by referring to the friend list 284, the server 20 can specify the frequency with which a plurality of users both measure their body movements with the measuring unit 30 and train with a training menu.

The item "message transmission/reception history" indicates the history of messages transmitted and received between users registered as friends.

<3 Configuration of measurement unit 30>
FIG. 5 shows a phase in which a user measures body movements with the measurement system 1 . FIG. 5 shows an example in which a plurality of measurement units 30 are installed and a plurality of users perform measurements. Also, an example in which one terminal device 10 is connected to a plurality of measurement units 30 is shown.

The measuring device 31A measures the body movement of the first user User_A in a sensing range 38A. In the detection range 38A, the mat 32A is installed at a position having a predetermined positional relationship with the measuring device 31A. Here, the position of the mat 32A may be, for example, at a certain distance (for example, several meters) from the front of the measuring device 31A. The mat 32A may be installed with respect to the measuring device 31A so that the measuring device 31A can measure the movement of the user's whole body when the first user User_A gets on the mat 32A. Poles 33A are installed on the right and left sides of the mat 32A, respectively. For example, if the mat 32A has a pattern indicating a predetermined position (which may be referred to as a home position) for the user to place both feet, a set of poles 33A are placed outside the mat 32A based on the pattern. It may be arranged. The pole 33A is installed as a reference for the user to grasp when the user makes a motion of twisting the upper limb on the mat 32A with both feet on the prescribed position. As a result, when the user twists the upper limbs, the user is urged to move the hand toward the pole 33A, so that the movement form of the upper limbs can be stabilized to some extent even when the measurement opportunities are increased. In other words, it is possible to suppress variations in time-series measurement results when measuring the user's body movement.

The measuring device 31B measures the body movement of the second user User_B in a sensing range 38B. In the detection range 38B, the mat 32B is installed at a position having a predetermined positional relationship with the measuring device 31B. Here, the position of the mat 32B may be, for example, a certain distance (about several meters) from the front of the measuring device 31B. The mat 32B may be installed with respect to the measuring device 31B so that the measuring device 31B can measure the motion of the user's whole body when the second user User_B gets on the mat 32B. Poles 33B are installed on the right and left sides of the mat 32B, respectively.

The first user User_A checks the position of the foot based on the pattern of the mat 32A and moves the body while viewing the video and audio displayed on the display 132A. Similarly, the second user User_B moves his or her body according to the video and audio guidance displayed on the display 132B. Thereby, the body movements of the first user User_A and the second user User_B can be measured by the measurement devices 31A and 31B, respectively.

In the illustrated example, based on the output from the display 132A, the first user User_A is about to start bending and stretching after placing both feet on the home position on the mat 32A. The second user User_B is making an action of stepping his right foot according to the output from the display 132B.

FIG. 6 is a diagram showing the details of the mat 32. As shown in FIG. As shown in the figure, patterns are arranged at a plurality of locations as guidelines for placing the user's right and left feet.

For example, for the right foot, there are a first home position 41R, a second home position 42R, and a third home position 43R where the right foot is placed at a position that serves as a guide for placing the user's feet over a plurality of measurement items. are shown multiple times.

For the left foot, a plurality of positions for placing the left foot are shown, such as a first home position 41L, a second home position 42L, and a third home position 43L.

In the following description, the home positions 41R and 41L may be collectively referred to as the home position 41 or the like. Similarly, the home positions 42R and 42L may be collectively referred to as the home position 42, and the home positions 43R and 43L may be collectively referred to as the home position 43.

For each measurement item (measurement item information 282), the user defines the distance between the right and left legs, and then the terminal device 10 guides the user to perform a predetermined body movement. For example, information on the position of the home position to be used may be provided in the audio guidance that accompanies the measurement of the measurement item.

When measuring the measurement items, the terminal device 10 determines whether or not the user is using a predetermined home position based on the measurement result of the measurement device 31 . For example, when the terminal device 10 identifies each pattern of the home position shown on the mat 32 based on the sensing result of the measuring device 31, the user's ankles and the like are detected based on the user's feet. It may be possible to identify the pattern that is present. If the user puts his/her feet in a position different from the assumed home position when measuring the measurement item, the user is prompted to place their feet in an appropriate position by voice guidance, video guidance, etc. Notification may be made.

In addition, the terminal device 10 includes, as a measurement item, the terminal device 10 measuring the movement of each joint when the user makes a stepping motion with the measurement unit 30 . The terminal device 10 can evaluate the goodness of balance, etc., when the user moves his/her body by taking a stepping motion, using the measurement items.

The mat 32 includes a pattern group 45R that serves as a target for stepping the right foot when there is a stepping action as a measurement item. Similarly, a pattern group 45L serving as a target for stepping the left foot is included.

These pattern groups 45R and 45L may be collectively referred to as "pattern group 45".

The pattern group 45 includes a plurality of patterns that serve as guides for the stepping position according to the height of the user. In the illustrated example, patterns are arranged in increments of 10 cm in the range of user's height from "140 cm" to "200 cm". By using a shape that imitates the toe portion of the foot as the pattern, it is possible to make the user easily recognize that it is a target position to step on. For example, the user places both feet at home positions 41R and 41L or patterns 44R and 44L (these may be collectively referred to as "pattern 44") as initial positions, and moves one foot at a time according to height. Then, step forward to the target pattern position. The terminal device 10 uses the measurement unit 30 to measure the movement of the user's body (motion of joints) when performing the stepping motion. When measuring the stepping action of the user, the terminal device 10 designates the home positions 41 to 43 or the pattern 44 as the initial position of the foot according to the distance that the user wants the user to step. It may be used as a guide.

The illustrated example also shows the position of the pole 33 . For example, the poles 33 (right side and left side) may be arranged so as to line up with the home positions 41 , 42 , 43 in a straight line. For example, the user tries to grasp the pole 33 by twisting the upper limb while placing both feet on one of the home positions 41 , 42 and 43 .

As shown in the figure, patterns of arrows may be arranged in the pattern groups 45R and 45L. In the illustrated example, the arrow pattern is superimposed on the pattern groups 45R and 45L corresponding to each height. This makes it possible for the user to be conscious of stepping forward, and reduces the possibility that the user will hesitate about how to move the body during measurement when the user looks at the mat 32.例文帳に追加

Also, as shown in the figure, regarding the size of the pattern, since the pattern serves as a guideline for the user to place the foot, the length in the vertical direction (the direction in which the foot is stepped) in the example shown in the figure is the size of a typical foot. It may be about the same as the size (about 22 cm to 28 cm in length). The shape of the pattern may be a footprint or a shape similar to a footprint so that the user can easily understand that a foot is placed on the pattern.

<4 Operation>
A measurement process in which the terminal device 10 measures the movement of the user's body will be described below.

FIG. 7 is a flow chart showing a process in which the terminal device 10 performs a ROM check and updates the measurement history.

In step S700, the control unit 190 of the terminal device 10 displays on the display 132 a screen for logging in by the user who performs the measurement, and receives an operation for logging in from the user. The terminal device 10 performs user authentication by, for example, reading information from a recording medium such as an IC card possessed by each user, reading a two-dimensional code displayed on a communication terminal possessed by the user, and the like. Get information. For example, by communicating with the server 20, the terminal device 10, as information on the user, includes information such as the user's height (user information database 281), past measurement history (measurement history database 283), registration of the user as a friend. Acquire information (friend list 284) of other users who have Further, the terminal device 10 acquires measurement item information (measurement item information 282 ) from the server 20 and stores it as measurement item information 182 .

In step S<b>701 , the terminal device 10 detects that the mat 32 and the poles 33 are installed within the measurement range of the measuring device 31 based on the measurement result of the measuring device 31 . For example, the terminal device 10 detects the mat 32 and the poles 33 by recognizing the measurement result of the measuring device 31 as a two-dimensional image. The terminal device 10 detects that the mat 32 and the poles 33 are at predetermined positions with respect to the measuring device 31 . In other words, the terminal device 10 detects that the mat 32 and the pole 33 are in positions suitable for measurement and in positions where the user can easily check the audio and video of the guide.

In step S703, the terminal device 10 refers to the measurement item information 182 and performs processing for sequentially measuring each measurement item. The terminal device 10 reads out the first measurement item and outputs information (such as the position where the user places his or her foot on the mat 32) that serves as a guide for measurement by audio and video. The terminal device 10 detects the joints of the user based on the measurement result of the measuring device 31, and detects that the user's feet are placed on predetermined positions (positions defined by the measurement items) on the mat.

In step S705, the terminal device 10 outputs a voice prompting the user to perform a predetermined physical action. The terminal device 10 measures the movement of the user's body by detecting the movement of the user's body with the measurement device 31 .

In step S707, the terminal device 10 sequentially reads out the measurement items from the measurement item information 182, outputs audio guidance, detects that the user's feet are at a specified position on the mat 32, and detects that the user's body is moving. to measure.

In step S709, when the measurement for each measurement item is completed, the terminal device 10 evaluates the range of motion of the user's joint based on the measurement result of the user's body movement. Specifically, the terminal device 10 identifies the position of each joint of the user based on the measurement results of the measurement device 31, and uses time-series data of the joint position (motion of the joint position that changes in time series) to measure Assess the range of motion of the target body part. The terminal device 10 outputs the evaluation result of the range of motion of the joint on the display 132 or the like, and transmits it to the server 20 .

In step S753, the server 20 receives the result of the user's ROM check from the terminal device 10, and updates the databases (measurement history database 283, user information database 281) based on the user's measurement result.

Next, processing of each device when measuring a user's stepping action as a measurement item will be described.

FIG. 8 is a flow chart showing a process in which the terminal device 10 evaluates the joint range of motion by measuring the user's stepping motion.

In step S801, the terminal device 10 confirms that both the user's feet are placed on the prescribed positions of the mat 32 (item "Position of feet on the mat" of the measurement item information 282) based on the measurement result of the measurement device 31. to detect The terminal device 10 may guide the user to put the feet on the prescribed positions when the user's feet are not on the prescribed positions.

In step S803, the terminal device 10 outputs audio and video guidance to the user to step the user's right foot from a prescribed position to a stepping position according to the height of the user.

In step S805, the terminal device 10 detects that the user's right foot has stepped from a specified position to another position on the mat based on the measurement result obtained by the measurement device 31, and measures the movement of the user's right foot stepping. Record the results. When the user's right foot is stepped to a position different from the target position to be stepped, the terminal device 10 guides the user to step the right foot again to the target position in order to re-measure. You can do it.

In step S807, the terminal device 10 outputs a voice guiding the user to step the left foot of the user from a prescribed position to a step position corresponding to the height (step length) of the user.

In step S809, the terminal device 10 detects that the user's left foot has stepped from the specified position to another position on the mat based on the measurement result of the measurement device 31, and measures the action of the user stepping the left foot. Record the results.

In step S<b>811 , the terminal device 10 evaluates the range of motion of the joints of the lower limbs based on the measurement result of the user's stepping motion on the mat 32 , outputs the evaluation result to the display 132 or the like, and transmits it to the server 20 .

Next, the processing of each device when measuring the user's upper body twisting motion will be described.

FIG. 9 is a flow chart showing a process in which the terminal device 10 evaluates the joint range of motion by measuring the user's upper body twisting motion.

In step S<b>901 , the terminal device 10 detects that the user's feet are placed on prescribed positions on the mat based on the measurement result of the measurement device 31 .

In step S903, the terminal device 10 determines whether the height of the pole 33 is appropriate based on the height information of the user. For example, the terminal device 10 associates and stores information on the height of the pole 33 that is appropriate according to the height of the user, so that the height of the pole 33 detected by the measurement result of the measuring device 31 is appropriate. Determine whether or not there is For example, the appropriate height of the pole 33 may be defined according to the position of the user's hands when the user stands upright. Here, the position of the hands when the user stands upright may vary depending on the height of the user. In other words, it is assumed that the higher the height, the higher the position of the hands when standing upright. In addition, the terminal device 10 detects the position of the user's hand (the height from the ground) in the upright posture based on the measurement result of the measuring device 31, thereby determining the appropriate height of the pole 33. It may be determined based on the position of the hand. That is, the height of the pole 33 may be adjusted according to the measurement result of the user's body by the measuring device 31 .

The height of the pole 33 is measured by specifying a first position where the pole 33 is in contact with the ground and a second position which is the top of the pole 33 based on the measurement result of the measuring device 31, for example. The determination may be made based on information on the distance from the position of the device 31 to the first and second positions of the poles 33 . In addition, when the pole 33 has a structure that can be extended, the pattern of the pole 33 includes a pattern indicating the height of the pole 33, and the terminal device 10 indicates the height of the pole 33. A pattern may be detected.

The terminal device 10 may determine whether or not the position where the pole 33 is installed is appropriate. For example, the terminal device 10 detects a pattern indicating a home position or the like on the pole 33, and determines whether the position of the pole 33 is appropriate based on the positional relationship between the position of the pattern and the position where the pole 33 is in contact with the ground. It may be determined whether or not there is.

In step S905, the terminal device 10 twists the user's body to the right while keeping both feet on the mat at prescribed positions, and the left hand is placed on the right pole (the pole 33 placed on the right) and the right hand is placed on the left pole (the left pole). It outputs voice and video guidance to grasp the pole 33).

In step S<b>907 , the terminal device 10 records the measurement result of the user's body twisting motion to the right based on the measurement result of the measurement device 31 .

In step S909, the terminal device 10 outputs a voice guiding the user to twist the user's body to the left while keeping both feet on the mat, and grasp the left pole with the right hand and the right pole with the left hand.

In step S911, the terminal device 10 records the measurement result of the user's body twisting motion to the left based on the measurement result.

In step S913, the terminal device 10 evaluates the range of motion of each joint of the body based on the measurement result of the user's twisting motion, and outputs the evaluation result. As the evaluation of the range of motion of the joint, whether or not the user was able to grasp the pole 33 by twisting the upper body may also be used as an evaluation item.

As described above, by setting the height of the pole 33 to be approximately the same as the height of the user's hand when the user stands upright, the user can visually recognize the pole while looking back when the user twists the upper body. With 33 as the target, try to move the hands and arms. As a result, it is possible to reduce the fluctuation width of the user's motion of twisting the upper body, and when the terminal device 10 accumulates the measurement results, it becomes easy to compare time-series data of the measurement results.

<5 Screen example>
FIG. 10 is a diagram showing an example of notification contents of the terminal device 10 in the phase of measuring the user's body movement.

The screen example of FIG. 10A shows a situation in which the terminal device 10 accepts an operation of logging in for measurement from the user. This corresponds to the processing of step S700 in FIG. For example, the measurement unit 30 is installed in a business office, store space, or the like of a business company, and when a plurality of users use it, user authentication is first performed.

As shown in FIG. 10A, the terminal device 10 displays a notification section 132L on the display 132. FIG. The terminal device 10 displays the operation content for the user to log in on the notification unit 132L. In the illustrated example, assuming that the measurement unit 30 is installed in the office, the notification section 132L indicates that login is possible by reading the user's employee ID card. The terminal device 10 displays in the notification section 132C that another user is about to perform measurement at the same timing. As in the example of FIG. 5, when a plurality of measurement units 30 are installed and a plurality of users perform measurement together, the terminal device 10 displays the progress of another user performing measurement on the notification unit 132C. . In the example of FIG. 10A, the terminal device 10 displays in the notification section 132C that another user who performs the measurement at the same timing has already completed the login process.

The screen example of FIG. 10B shows a situation in which the terminal device 10 measures the user's stepping motion. This corresponds to the processing of steps S801 to S811 in FIG.

As shown in FIG. 10B, the display 132 of the terminal device 10 displays a notification section 132D, a notification section 132E, a guide output section 132F, and a detection result display section 132G.

The terminal device 10 displays the information of the logged-in user (that is, the user who is performing the measurement) on the notification section 132D. Thereby, the user can easily check whether or not the measurement is performed as another user by mistake.

The terminal device 10 displays the history of the user's measurements on the notification unit 132E. In the illustrated example, the number of times the user has performed measurements using the measurement unit 30 is displayed, but in addition to this, the evaluation results of past measurements and the like may also be displayed. Thereby, the user can easily confirm that the measurement is being continued, and can be motivated to continue the measurement.

In the guide output unit 132F, the terminal device 10 notifies the user of the target pattern information according to the height of the user by video and audio. Thereby, the user can easily draw an image of stepping toward the target pattern, and can easily perform the operation for measurement.

The terminal device 10 displays the detection result by the measuring device 31 on the detection result display section 132G. For example, the terminal device 10 displays information such as that the user's feet are in the prescribed positions for measurement (both feet are placed at the home position) on the detection result display section 132G. As a result, the user can easily recognize that the measurement is being performed correctly according to the procedure, and can perform the measurement with a sense of security that measurement errors will not occur.

The screen example of FIG. 10C shows a situation in which the terminal device 10 measures the user's twisting motion.

In the screen example of FIG. 10C, the terminal device 10 detects that the height of the pole 33 is not appropriate, and instructs the user to extend the pole 33 to a predetermined height using the guide output unit 132F. It is displaying a prompt. This corresponds to the processing of step S903 in FIG.

The screen example of FIG. 10D shows a situation in which the terminal device 10 notifies the user of the completion of the measurement, the evaluation result based on the measurement result, and prompting the user to register as a friend with another user.

In the screen example of FIG. 10(D), the terminal device 10 displays in the guide output unit 132F that the measurement of all of the plurality of measurement items has been completed, and the results of evaluating the user's body movements for the measurement items. The terminal device 10 refers to the user's past measurement history, and displays the result of comparison with the past measurement result (such as a change in joint range of motion from the previous time) on the guide output unit 132F.

The terminal device 10 displays, in the notification unit 132M, a display prompting the other user to register as a friend, and a background of proposing the other user as a friend candidate (measurement was taken at the same place, etc.).

<Summary>
(1) As described above, according to the present embodiment, it is not necessary to apply a physical measuring instrument such as a goniometer to the body of the user when performing the ROM check. You can easily measure yourself. Further, by progressing the measurement according to the measurement items, it is possible to evaluate the range of motion of each joint of the user's lower body and upper body.

(2) When measuring the movement of the user's body, the mat 32 and the poles 33 can be used to make the user recognize the target point of the user's body movement. This stabilizes the form in which the user moves the body, so that the user can easily confirm changes in body movement by accumulating and comparing measurement results. For example, by using the mat 32, it is possible to specify (i) how to stride, such as the stride length and the range of motion when the user performs a stepping motion, and instruct the user to perform the motion specified in the measurement. can be encouraged. In the case of continuously measuring the user's bodily motions, it is possible to easily compare the measurement results over time by specifying the motions that the user performs during the measurement. In other words, by defining the motion of the user's body, it is possible to further facilitate the evaluation of how the motion of the user's body has changed. Further, by using the mat 32 , (ii) the standing position of the user with respect to the measuring device 31 can be defined. The terminal device 10 can determine whether the mat 32 is arranged at a predetermined position with respect to the measuring device 31 based on the measurement result of the measuring device 31 . Therefore, when the motion of the user's body is measured using the mat 32 , the user's standing position is defined by each pattern of the mat 32 . This makes it easier for the user's body motion to fall within the measurement range of the measurement device 31 . For example, even if the user sticks out his or her arm straight up or right to the side, by placing the mat 32 in a predetermined position with respect to the measuring device 31 in advance, the movement of the user's body will not affect the movement of the measuring device 31 . Deviating from the measurement range can be suppressed. Further, by using the mat 32, (iii) when making the user perform a specific action, the user can take an appropriate posture before taking measurements. For example, the user's standing position can be defined by the home position, and when the user performs a predetermined action (squatting, raising one leg, etc.), a posture suitable for measurement (predetermined foot width, etc.) Measurement can be started with

These make it possible to easily evaluate the suppleness of each joint of the user's whole body.

(3) Furthermore, in the first embodiment, the terminal device 10 may suggest an exercise suitable for the user according to the measurement result of the user's body motion (ROM check result, etc.). . For example, assume that the terminal device 10 holds an exercise menu (training item information 185, 285) in association with the classification result of the ROM check evaluation results (measurement history information 183, measurement history database 283). That is, it is assumed that the information of the training menu to be proposed is stored in advance in association with the measurement results and evaluation results of the user's body movements. The terminal device 10 refers to the measurement history database 283, specifies an exercise menu to be proposed to the user based on the measurement results of the user's body movement such as ROM check, and the evaluation results, and displays the specified exercise menu on the display 132 or the like. may be presented to the user by displaying the For example, as a result of the ROM check, the range of motion of each joint of the user can be evaluated. In the training item information 285, the server 20 may predetermine the range of motion of the joints required for the user to perform the exercise. For example, as a result of the ROM check, if the range of motion of the user's hip joint is smaller than a certain level (if the hip joint is stiff), the user is not provided with a predetermined training item as an exercise (the terminal device 10 may not provide the training item based on the training item). processing such as displaying an image cannot be executed). Also, based on the result of the ROM check, the user may be provided with an exercise menu according to the evaluation result of each joint of the user's body.

In addition, the terminal device 10 measures the movement of the user's body while performing the training indicated by the training item, and determines whether the user can move the body as specified in the training. judge. As a result of the user continuing to exercise, the muscle strength of each part of the body for improving the walking posture is gradually improved, and it is expected that the user will be able to perform the movements required for exercise. Accordingly, the terminal device 10 may determine that the user has completed the exercise menu and present another exercise menu to the user.

Alternatively, an exercise menu may be proposed to the user based on the history of exercise menus executed by other users that are similar to the evaluation results of the user's physical motion measurement results.

(4) In the description of the first embodiment above, an example of measuring and evaluating the movement of the user's body by installing the measuring device 31, the mat 32, and the pole 33 in the real world has been described.

Alternatively, an object corresponding to the mat 32 and an object corresponding to the pole 33 may be arranged in the virtual space to measure the movement of the user's body. For example, the user views the display 132 or wears a head mounted display (HMD). The terminal device 10 displays, on the display 132 or HMD, a field-of-view image corresponding to the setting of the virtual camera in the virtual space. By attaching a motion sensor to each part of the user's body (head, shoulder, elbow, hand, waist, knee, foot, etc.), the terminal device 10 tracks the movement of the user's body. The user moves the user's body while visually recognizing each object installed in the virtual space. As a result, the movement of the user's body can be measured and evaluated without installing the mat 32 and the poles 33 in the real world.

In addition to setting the mat 32 and the poles 33 in the real world, an image corresponding to the mat 32 may be projected onto the floor surface or the like, and an image (such as a stereoscopic image) corresponding to the pole 33 may be displayed in the real world. It may be displayed in For example, in the real world, light may be emitted from a laser, a projector, or the like to display the virtual mat 32 and the poles 33 so that the user can visually recognize them.

<Second Embodiment>
A measurement system 1 according to the second embodiment will be described. In the second embodiment, the user's walking motion is measured to evaluate the walking posture, thereby assisting the user in acquiring a walking style that is beautiful and facilitates the exercise of bodily functions. will be explained.

FIG. 11 shows a situation in which the user's walking motion is measured by the measurement system 1 according to the second embodiment.

<1 Overview>
The measuring device 31A measures the movement of the first user User_A walking from a predetermined position to the measuring device 31A in the detection range 38A. On the display 132A, the terminal device 10A notifies the first user User_A of measuring the state of walking and how to move the body during walking. As illustrated, the terminal device 10A notifies the user to walk so as to put the weight on the front leg when walking.

In the example of FIG. 11, coordinate axes are defined as follows. That is, with the position of the first user User_A as a reference, the walking direction is defined as "forward", the direction opposite to the walking direction is defined as "backward", and the axis determined by the walking direction is defined as the "front-rear axis". On the other hand, with the position of the first user User_A as a reference, the direction perpendicular to the walking direction on the surface where the walking foot touches the ground is defined as the "left-right axis", the direction of the user's right half of the body is defined as the "right direction", and the user's The direction of the left half of the body is defined as "left direction". Let the axis in the body direction from the feet to the top of the head of the first user User_A be the “vertical axis”.

On the display 132A, the terminal device 10A notifies the first user User_A to walk toward the measuring device 31A starting from a position a predetermined distance away from the position of the measuring device 31A. Here, the starting point where the first user User_A starts the walking motion is, for example, a position several meters away from the measuring device 31A, and the walking cycle of the first user User_A (the motion of stepping on the right foot and the motion of stepping on the left foot). ) can be repeated multiple times. The measurement device 31A can measure the walking cycle of the first user User_A multiple times. When the user's walking motion is measured by the measuring device 31A, the terminal device 10A notifies the first user User_A to move to the starting point if the first user User_A is not located at the starting point. may be performed by the display 132A or the like. In response to the presence of the first user User_A at the starting point, the terminal device 10A notifies the first user to start measuring the walking motion (or notifies the first user to start the walking motion). may

FIG. 12 shows each part of the user's body identified by the terminal device 10 based on the measurement result of the measuring device 31. As shown in FIG. The terminal device 10 identifies the joints of the extremities, the head, the shoulders, the trunk, the pelvis, the feet, the heels, the toes of the feet, the ribs, the anterior superior iliac spine, etc. do. The terminal device 10 identifies the thoracic spine (the center of the trunk), the center of the pelvis, and the like. For example, the terminal device 10 prompts the user to take a predetermined pose (such as a pose in which both arms are extended sideways to form a T shape, a pose in which both arms are attached to the body, etc.). Based on the measurement result of 31, the position of each joint of the user in the measurement result of the measuring device 31 may be identified.

<2 Data structure>
FIG. 13 is a diagram showing data structures of the measurement result database 287 and the evaluation result database 288 stored in the server 20 according to the second embodiment.

The measurement result database 287 is a database that holds the measurement results of each user's body motion (for example, walking motion, trunk balancing motion, etc.) measured by the measuring device 31 .

Each record of the measurement result database 287 includes the item “measurement data ID”, the item “date and time”, the item “user ID”, the item “elapsed time from measurement start”, and the item “measurement data”.

The item “measurement data ID” indicates information for identifying each piece of measurement data measured by the measuring device 31 . In the second embodiment, the terminal device 10 starts and ends the measurement of the user's bodily motion (for example, the measurement of the walking cycle) to associate information identifying each measurement data with the measurement data. and store it in the server 20.

The item “date and time” indicates the date and time when the measurement device 31 measured the motion of the user's body.

The item “user ID” is information for identifying each user whose body motion is measured by the measuring device 31 .

The item “elapsed time from start of measurement” indicates the elapsed time from the start of measurement of the user's body motion by the measuring device 31 .

The item "measurement data" indicates the position at the timing shown in the item "elapsed time from start of measurement" for each part of the user's body (head, shoulders, . . . ) as described with reference to FIG. For example, in the item "measurement data", in a coordinate system based on the position of the measuring device 31, the measurement result of measuring the distance from the position of the measuring device 31 to each part of the user's body may be held.

The evaluation result database 288 is a database that holds evaluation results obtained by evaluating the user's bodily motion (for example, walking motion) based on the measurement results of the measuring device 31 (measurement result database 287).

Each record of the evaluation result database 288 includes the item “evaluation result ID”, the item “measurement data ID”, the item “measuring object”, and the item “evaluation result”.

The item “evaluation result ID” is information for identifying each evaluation result obtained by evaluating the measurement result of the user's body motion by the measuring device 31 .

The item "measurement data ID" is information for identifying each measurement result (measurement result database 287) of the measurement device 31 to be evaluated.

The item "measuring object" indicates the content to be measured as the user's body motion. Objects to be measured include, for example, the posture of the user during walking, the balance of the user's trunk, and the like.

The item “evaluation result” indicates the evaluation result of evaluating the user's body movement based on the measurement result of the measuring device 31 . For example, when evaluating the user's posture during walking, the objects to be evaluated are "the user's stepping forward", "whether the user's weight can be transferred to the stepped front leg", and "whether the user's weight can be transferred to the front leg". Are you ready?” Details will be described later. Also, when evaluating the balance of the user's trunk, the subject of evaluation includes "whether the positions of the shoulders, hips, and feet are out of alignment". Details will be described later.

<3 Operation>
Hereinafter, a process in which the terminal device 10 measures the movement of the user's body and evaluates based on the measurement results in the second embodiment will be described.

FIG. 14 is a flow chart showing processing for measuring and evaluating the user's walking motion by the terminal device 10 in the second embodiment.

In step S<b>1403 , the terminal device 10 detects that the user is at a predetermined position within the measurement range of the measurement device 31 based on the measurement result of the measurement device 31 .

In step S1405, the terminal device 10 outputs guidance (audio, video) to the user to perform a walking motion. The terminal device 10 detects each joint of the user based on the measurement result of the measuring device 31, and records the position of each joint during the user's walking motion.

In step S<b>1407 , the terminal device 10 classifies the user's walking cycle into a plurality of phases based on the measurement result of the measurement device 31 . The terminal device 10 determines an evaluation value obtained by evaluating the walking motion according to the evaluation items for each aspect of the user's walking motion.

In step S1409, the terminal device 10 outputs to the display 132 or the like the measurement result of the measuring device 31 and the evaluation result of evaluating the walking motion together with the ideal walking posture model. The terminal device 10 transmits measurement results and evaluation results to the server 20 .

In step S1453, the server 20 updates the measurement result database 287 and the evaluation result database 288 based on the measurement result and evaluation result of the user's walking motion.

<4 Output example of measurement results and evaluation results>
FIG. 15 is a diagram showing an output example when the terminal device 10 outputs the measurement result of measuring the user's walking motion and the evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. is. In the illustrated example, the terminal device 10 displays these measurement results and evaluation results on the display 132. However, the terminal device 10 prints the measurement results and evaluation results on a paper medium or the like using a printer or the like. It is also possible to output the result.

FIG. 15A shows a situation in which the terminal device 10 prompts the measurement device 31 to walk a certain number of steps (for example, about several meters) in order to measure the user's walking motion. This corresponds to the processing of steps S1403 and S1405 in FIG.

In the display 132, the display unit 132P1 of the terminal device 10 displays to the user a "notification prompting the user to walk so as to put his or her weight on the front leg to be stepped" when performing a walking motion. The terminal device 10 displays, on the display units 132P2 and 132P3, an image showing the specified position of each part of the user's body (each part of the body illustrated in FIG. 12) as the measurement result of the measuring device 31. FIG. The terminal device 10 connects each part of the user's body in a linear fashion, thereby displaying the measurement result of the user's body so that the user can easily visually recognize it. The example of FIG. 15 shows an example in which the parts of the right half of the user's body are connected by dotted lines, and the parts of the left half of the body are connected by solid lines. Thereby, the user can easily recognize whether the measurement result of the measuring device 31 is the movement of the right half of the user's body or the movement of the left half of the user's body.

The terminal device 10 uses the display unit 132P2 to display, at any time, the result of specifying the position of each part of the user when viewed from the front of the user (the front-rear axis shown in FIG. 11) based on the measurement result of the measurement device 31. .

The terminal device 10 uses the display unit 132P3 to display the result of specifying the position of each part of the user when viewed from the side of the user (left-right axis shown in FIG. 11) at any time based on the measurement result of the measurement device 31. . That is, the terminal device 10 displays the measurement results of the measuring device 31 on the display units 132P2 and 132P3 in real time. This allows the user to obtain real-time feedback of the measurement result of the user's own body motion.

The terminal device 10 notifies the user to perform a walking motion to approach the measuring device 31 . Here, the terminal device 10 may display the measurement results on the display units 132P2 and 132P3 as if the movement of the user's body was measured at a certain distance from the user's body. For example, the terminal device 10 generates a three-dimensional virtual section, and based on the measurement result of the measuring device 31 (that is, the distance from the position of the measuring device 31 to the position of each part of the first user User_A), the three-dimensional An object corresponding to the position of each part of the user's body is placed in the virtual space. For example, the terminal device 10 identifies the coordinates corresponding to the measuring device 31 in the three-dimensional virtual space, and determines the position of each part based on the measurement result of the measuring device 31 with respect to the coordinates corresponding to the measuring device 31. An object corresponding to each part is placed at the corresponding coordinates. The terminal device 10 holds information on the position of the object corresponding to each part in the three-dimensional virtual space in association with the elapsed time that accompanies the measurement by the measuring device 31 . That is, by reproducing the trajectory of the movement of the object corresponding to the position of each part of the user in the three-dimensional virtual space, the walking motion of the user can be reproduced as the trajectory of the position of each part. The terminal device 10 coordinates the object group corresponding to the position of each part of the user with a certain positional relationship (for example, a position at a certain distance on the front-rear axis from the "shoulder center", or a chest position). A virtual camera may be set at a position at a certain distance on the left-right axis from the waist, and the virtual camera may be moved in conjunction with the movement of the object corresponding to the position of each part.

The terminal device 10 notifies the information for appropriately performing the measurement by the measuring device 31 on the display section 132P4. This corresponds to step S1403 in FIG. For example, when the user's walking motion is measured by the measuring device 31, the terminal device 10 determines whether or not the user starts walking at a predetermined position, and prompts the user to start walking from the predetermined position. give notice. As illustrated, when the position where the user starts walking is too close to the measuring device 31, the terminal device 10 notifies the user to start walking further from behind.

FIG. 15(B) shows a phase in which the terminal device 10 outputs measurement results and evaluation results for a phase among multiple phases of the user's walking motion. This corresponds to the processing of step S1409 in FIG.

In FIGS. 15B and 15C, the measurement results of the user's body are displayed with the vertical axis as the reference. Accordingly, it is possible to present to the user how the user's body is rotated with respect to the vertical axis.

The terminal device 10 outputs, on the display 132, evaluation results of shoulder rotation balance and waist rotation balance during walking as evaluation results of the beauty of the user's walking posture. The terminal device 10 divides the walking motion into a plurality of phases, and outputs evaluation results such as shoulder rotation balance and hip rotation balance in each phase. In this way, by presenting the user with the measurement results and evaluation results for a plurality of aspects of the walking motion, suggestions for making the user's walking motion beautiful can be provided.

The terminal device 10 displays the rotation inclination of the shoulder in each of the left rotation and the right rotation in the walking cycle (one cycle is the stepping of the right foot and then the left foot) on the display unit 132P5.

The terminal device 10 becomes the center of the "line connecting both shoulders" determined by the positions of the right and left shoulders, the "line connecting both hip joints" determined by the positions of both hip joints, and the positions of the right and left shoulders on the display unit 132P6. The “center position of the shoulders” and the “center position of the waist” determined by the positions of the hip joints (for example, the positions of the right and left pelvises) are displayed. The terminal device 10 calculates the degree of left shoulder rotation ( How far the shoulders rotate) is evaluated. For example, the terminal device 10 draws a “line connecting both shoulders” based on the amount of rotation of the shoulders by left rotation in one cycle with the position of the waist as a reference. The terminal device 10 displays the score (score “4.8 points”) as the evaluation result on the display unit 132P6.

The terminal device 10 displays the “line connecting both shoulders”, “line connecting both hip joints”, “center position of shoulder”, and “center position of waist” on the display unit 132P7. Based on the position of the right shoulder (or both shoulders) when the user steps out the right foot and the positions of the hip joints, the terminal device 10 calculates the degree of left rotation of the shoulders with respect to the vertical axis in one cycle of the walking motion. Evaluate. The terminal device 10 draws a “line connecting both shoulders” based on the amount of rotation of the shoulders by right rotation in one cycle with the position of the waist as a reference. The terminal device 10 displays the score (score “4.5 points”) as the evaluation result on the display unit 132P7, which is lower than the score for the left rotation.

The terminal device 10 displays, on the display units 132P6 and 132P7, the "center position of the shoulder" and the "center position of the waist" in the motion of stepping the left foot and the motion of stepping the right foot in one cycle of the walking motion. This allows the user to recognize how the upper body is tilted with respect to the position of the waist during walking. For example, when the user walks, the user can be made aware of the extent to which the upper body moves along the left-right axis and the extent to which the upper body moves along the front-back axis. In the illustrated example, in the display section 132P6, the degree of overlap between the "center position of the shoulder" and the "center position of the waist" during left rotation is large, and the upper body is not shaken, whereas in the display section 132P7, During clockwise rotation, the overlap between the "shoulder center position" and the "waist center position" is relatively small, and the upper body shakes more laterally than during left rotation. Therefore, the score, which is the evaluation result, is lower for right rotation than for left rotation.

FIG. 15(C) shows, as a phase of the user's walking motion, the terminal device 10 outputting the measurement result and the evaluation result when the forefoot is on the ground during the walking motion. This corresponds to step S1409 in FIG.

The terminal device 10 displays, on the display unit 132P8, the measurement result and the evaluation result of the inclination of rotation of the waist when the front foot of the user touches the ground during walking.

The terminal device 10 displays, on the display unit 132P9, "a line connecting both hip joints" or the like when the left foot is placed on the ground as the front foot. As shown, the "line connecting both hip joints" is slanted, indicating that the user is able to rotate the hips when the left foot is on the ground.

The terminal device 10 displays, on the display unit 132P10, "a line connecting both hip joints" when the right foot is on the ground as the front foot. As shown, the "line connecting both hip joints" is slanted, indicating that the user can rotate the hips when the right foot is on the ground.

FIG. 16 is a diagram showing an output example when the terminal device 10 outputs the measurement result of measuring the user's walking motion and the evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. is. This corresponds to the processing of steps S1407 and S1409 shown in FIG.

In the example of FIG. 16 , the terminal device 10 displays the measurement results and evaluation results of one cycle of the user's walking motion in a manner that can be compared with a reference example (model case, model) for making the walking posture look beautiful. The output example shown in FIG. 16 may also be output by the terminal device 10 by printing on a paper medium using a printer or the like.

On the display 132, the terminal device 10 divides one cycle of the walking motion into a plurality of steps, and for each step, displays a model for making the walking posture look beautiful, the measurement result of the user's walking motion, and the evaluation result for the measurement result. to display.

In the example of FIG. 16, one cycle of the walking motion consists of the first step "stepping forward", the second step "trying to put the weight on the stepped front foot (trying to get on the front foot)", and the second step It is divided into 3 steps, "Put your weight on the stepped front leg (get over the front leg)".

Here, in the first step, the second step, and the third step, the items evaluated by the terminal device 10 include items common to each step and items not common to each step.
(1) Evaluation items for the first step • Inclination of the upper body during walking: Whether the parts of the upper body (head, shoulders, etc.) are inclined forward or backward with respect to the position of the waist. For example, is the angle at which the line determined by the position of the pelvis, the position of the chest and the waist, and the position of the center of the shoulder tilt with respect to the vertical axis within a certain range?
・ Fore-and-aft spread of the feet when trying to step forward: The degree of the fore-and-aft spread of both feet. For example, is the distance between the ankle of the front leg and the ankle of the back leg that has stepped forward a certain distance or more?
・ Hip position when trying to step forward: Is the hip left behind when trying to step forward? For example, is the position of the pelvis forward or backward with respect to the shoulders, chest and waist?
(2) Evaluation items for the 2nd step ・Inclination of the upper body at the timing when the front foot is on the ground: Whether the parts of the upper body (head, shoulders, etc.) are tilted forward or backward with respect to the position of the waist ・Bending of the knee when the front foot touches the ground: Whether each part of the front foot (ankle, knee, pelvis) is bent or bent greatly. For example, is the angle formed by the line connecting the ankle and knee and the line connecting the knee and pelvis within a certain angle? Inclination of the upper body at the timing of putting the weight on: Whether the parts of the upper body (head, shoulders, etc.) are tilted forward or backward with respect to the position of the waist. With the front foot that landed on the ground as the pivot foot, each part of the pivot foot (ankle, knee, pelvis) is stretched or bent when putting weight on the pivot foot. For example, does the angle formed by the line connecting the ankle and the knee and the line connecting the knee and the pelvis fall within a certain angle? Display the exemplar in the first step. The terminal device 10 displays the measurement result of the user's body in the first step on the display unit 132P21. The terminal device 10 displays the evaluation result of the user's body movement in the first step on the display unit 132P31.

The display units 132P11, 132P12, and 132P13 of the terminal device 10 display the position of each part (joint) of the body when each of these evaluation items is good for a beautiful walking posture. For example, in the display unit 132P11, when the upper body is not tilted, the front legs and the back legs are spread apart to a certain extent (the stride is not too short), and the hip position is appropriate (not leaning back). The position of each part of the body is indicated.

In the illustrated example, the display unit 132P31 displays, as the evaluation results, the evaluation result RF_ic for the right foot when the right foot is stepped and the evaluation result LF_ic for the left foot when the left foot is stepped for the evaluation item "inclination of upper body". It is displayed superimposed on a predetermined bar. The predetermined bar is drawn according to the range that can be taken as the evaluation result. In the predetermined bar, the terminal device 10 distinguishes between areas with good evaluation results and areas with poor evaluation results, and in the illustrated example, the areas with good evaluation results are highlighted. doing. The terminal device 10 sets the evaluation result RF_ic for the right foot and the evaluation result LF_ic for the left foot in the range in which the evaluation result is neither excessively tilted forward nor excessively backward in the evaluation item “upper body tilt”. are displayed in association with each other.

In the display unit 132P31, the terminal device 10 associates the evaluation result of the right leg and the evaluation result of the left leg with a predetermined bar for the evaluation item "front and back spread of the legs" and the evaluation item "waist position" as the evaluation results. are displayed.

The terminal device 10 displays the model in the second step on the display unit 132P12. The terminal device 10 displays the measurement result of the user's body in the second step on the display unit 132P22. The terminal device 10 displays the evaluation result of the user's body movement in the second step on the display unit 132P32.

The terminal device 10 displays the model in the third step on the display unit 132P13. The terminal device 10 displays the measurement result of the user's body in the third step on the display unit 132P23. The terminal device 10 displays the evaluation result of the user's body movement in the third step on the display unit 132P33.

FIG. 17 is a diagram showing an example in which the terminal device 10 outputs an evaluation result of the load on the knees in the user's walking motion. This corresponds to step S1409 in FIG.

FIG. 17A shows an example of outputting measurement results and evaluation results of hip joint adduction and abduction in a user's walking motion. The terminal device 10 displays the measurement results of the user's walking motion on the display 132 based on the longitudinal axis (assuming that the user is viewed from the front), so that the user can easily check the bending of the knees during the walking motion. make it

The terminal device 10 displays the measurement results during the walking motion on the display unit 132P41. The terminal device 10 displays the measurement result when walking with the right foot as the pivot on the display unit 132P42. The terminal device 10 displays the measurement result when walking with the left foot as the pivot on the display unit 132P43.

On the display unit 132P44, the terminal device 10 displays a favorable measurement result pattern (“both the pivot leg and the free leg are straight”) and a measurement result pattern that may hurt the knee (“knee "Possibility of hurting the outside" and "Possibility of hurting the inside of the knee") are displayed. In other words, even if the hip joint is excessively internally rotated or externally rotated during walking, the knee may be injured during the walking.

As described above, by referring to the output result of the terminal device 10, the user can be encouraged to walk without hurting the knee.

FIG. 17B is a diagram showing an example of outputting an evaluation result as to whether the knee is too far ahead of the ankle in the user's walking motion. The terminal device 10 displays the measurement result of the user's walking motion on the display 132 based on the left-right axis (assuming that the user is viewed from the side), and checks whether the knee moves too far ahead during the walking motion. can be easily checked by the user.

The terminal device 10 displays the measurement results during walking on the display unit 132P51. The terminal device 10 displays the measurement result when walking with the right foot as the pivot on the display unit 132P52. The terminal device 10 displays the measurement result when walking with the right foot as the pivot on the display unit 132P53. The terminal device 10 displays on the display unit 132P51 the result of the evaluation of the user's walking posture, indicating that "the load on the front of the knee is small".

On the display unit 132P54, the terminal device 10 displays a favorable measurement result pattern (“good”) and a measurement result pattern that may hurt the knee (“the knee is too far forward”) for the user’s reference. ) and about. That is, when the user attempts to step forward with the front foot, if the knee is too far ahead of the ankle, the knee may be injured. The terminal device 10 determines whether or not the angle at which the line connecting the ankle position and the knee position and the line connecting the knee position and the pelvis position intersect is equal to or greater than a predetermined angle (knee to ankle forward). out) to assess whether the knee is too far ahead when the user attempts to step forward.

As described above, the walking motion is evaluated for a plurality of phases of the walking cycle of the user, such as trying to step forward with the pivot leg, landing the pivot leg, and trying to put the weight on the pivot leg. Thereby, the user is prompted to make the walking motion more beautiful by referring to the evaluation result of each phase. In addition, by allowing the user to refer to the measurement result of the user's body movement by comparing the model body movement with the user's body movement, the user is urged to approach a beautiful walking posture.

<Third Embodiment>
A measurement system 1 according to the third embodiment will be described. In the third embodiment, processing for displaying the measurement result of the user's trunk balance and enabling the user to obtain a state in which the trunk is in balance will be described.

<Action>
Hereinafter, in the third embodiment, a process in which the terminal device 10 measures the movement of the user's body and evaluates based on the measurement results will be described.

FIG. 18 is a flow chart showing processing for measuring and evaluating the user's trunk balance by the terminal device 10 in the third embodiment.

In step S<b>1803 , the terminal device 10 outputs guidance (audio, video) to measure the user's trunk balance (shoulders, hips, legs) based on the measurement result of the measurement device 31 . The terminal device 10 displays the measurement results by displaying the positions of the user's shoulders, hips, and feet as circles based on the vertical axis. The terminal device 10 outputs a guide prompting the user to balance the trunk according to how the shoulders, hips, and feet of the user overlap.

In step S1805, the terminal device 10 evaluates the trunk balance of the user based on the measurement result of the measurement device 31 and outputs an evaluation value. The terminal device 10 transmits the measurement result of the measurement device 31 and the evaluation result to the server 20 .

In step S1853, the server 20 updates the measurement result database 287 and the evaluation result database 288 based on the user's trunk balance measurement result and evaluation result.

<Example of output>
FIG. 19 is a diagram showing an example in which the terminal device 10 outputs a measurement result of measuring the trunk balance of the user and an evaluation result of the trunk balance in the third embodiment.

FIG. 19A shows a screen example of a phase in which the terminal device 10 outputs measurement results in real time in order to measure the user's trunk balance using the measurement device 31 . This corresponds to the processing of step S1803 in FIG. In the example of FIG. 19A, the display unit 132P61 displays circular graphics (perfect circle, ellipse, etc.) corresponding to the shoulders, waist, and feet of the user's body.

The terminal device 10 displays the measurement results of each part of the user's body on the display unit 132P62 based on the front-rear axis.

The terminal device 10 displays the positions of the shoulders, waist, and feet of the user's body on the display unit 132P63 based on the vertical axis based on the measurement result of the measurement device 31 . The terminal device 10 displays the measurement result of the measuring device 31 on the display section 132P63 at any time. Therefore, when the user's body shakes, the circular figures corresponding to the shoulders, hips, and legs do not overlap, causing deviation. The screen example of FIG. 19A shows that each part of the user's body is not arranged along the vertical axis and is shifted. The user is prompted to move the body so as to balance the trunk while watching the display on the display section 132P63 of the display 132 .

The terminal device 10 notifies the user on the display unit 132P64 to prompt the user to balance the trunk. Based on the measurement result of the measuring device 31, the terminal device 10 identifies the movement of the user's body required to balance each part of the user's body. In the illustrated example, the position of the head and feet is on the back side with respect to the position of the waist of the user. In other words, the waist of the user is in a state of protruding forward. The terminal device 10 displays on the display unit 132P64 a display prompting the user to adjust the inclination of the body in the front-rear direction. This allows the user to easily recognize the movement of the body required to bring the trunk into a balanced state.

FIG. 19B shows a display example of measurement results when the trunk of the user is well-balanced. In the display section 132P63, the user's shoulders, waist, and feet are aligned along the vertical axis, and the trunk is well balanced. The terminal device 10 notifies the user that the trunk is in a balanced state on the display unit 132P64.

FIG. 19(C) shows a display example when the trunk balance measurement is finished and the evaluation result is output. This corresponds to the processing of step S1805 in FIG.

The terminal device 10 displays on the display unit 132P65 that the trunk balance measurement has been completed, and outputs an evaluation result based on the measurement result.

The terminal device 10 determines a training menu for balancing the trunk as a training menu to be proposed to the user based on the evaluation result of the trunk balance. In the illustrated example, the terminal device 10 displays, on the display unit 132P65, an evaluation result of the user's trunk balance and training menu candidates to be proposed to the user according to the evaluation result. The terminal device 10 receives an operation for starting the training menu proposed to the user on the display unit 132P66. The terminal device 10 reads out the corresponding training menu in response to the user's input operation to the display unit 132P66.

The terminal device 10 receives designation of a training menu from the user on the display unit 132P67. In other words, the display unit 132P67 accepts an input operation for starting other training regardless of the training menu that the terminal device 10 proposes to the user based on the trunk balance evaluation result.

The items described in each of the above embodiments may be combined.

<Fourth Embodiment>
In the description of the fourth embodiment, the measurement results of the user's body movements are ROM check (first embodiment), walking posture check (second embodiment), trunk balance measurement. A technique for proposing a training menu suitable for the user according to (third embodiment) and the like will be described. By measuring and evaluating the movement of the user's body as described above, the terminal device 10 can identify which part of the user's body should be trained. , suggest a suitable training menu.

With this configuration, when new training menu content is added, the content of the training menu can be customized according to which part of the body the training menu is suitable for, or whether the content is suitable for the user's body. By associating whether or not, it is possible to provide the user with content corresponding to the measurement result of the user's body movement, making it easier to add content.

(1) As described above, the terminal device 10 may propose an exercise menu to the user based on the measurement results and evaluation results in each step of the user's walking cycle. For example, the terminal device 10 compares a model body movement with the user's body movement (example in FIG. 16), and identifies muscle parts that the user should improve. For example, the server 20 selects an exercise menu that is expected to allow the user to take a large step forward when the distance between the front and back of the foot is small in the evaluation result of the first step (movement of stepping forward) as the walking motion of the user. It is specified by referring to the training item information 285 . In other words, the server 20 may associate and hold a proposed exercise menu in advance according to the evaluation results of the evaluation items of each step of the walking motion and the evaluation items of the balance of the trunk.

(2) The terminal device 10 may propose a training menu to the user so that the left-right difference in the user's body motion can be eliminated. As described above, the terminal device 10 evaluates the left-right balance of the user's body motion by performing the ROM check and the walking posture check. The terminal device 10, for example, evaluates the left-right balance of the rotational motion with respect to the vertical axis based on the evaluation result of the shoulder rotation and waist rotation during walking motion of the user. The terminal device 10 identifies the muscle part that causes the user's body motion to be left-right symmetrical. For example, the terminal device 10 holds in advance information on muscle parts that are factors that cause left-right differences in the amount of shoulder rotation. As described above, the terminal device 10 preliminarily associates and retains the evaluation target during walking and the information on the muscle part that causes the left-right difference, thereby eliminating the left-right balance difference. You can identify the parts of the muscles that you should train for.

(3) The terminal device 10 predicts possible future states of the user's body based on the evaluation results of the user's body movements, and proposes a training menu to the user based on the prediction results. good too.

For example, the terminal device 10 identifies (i) possible physical concerns of the user, such as “neck/shoulder stiffness”, “sagging upper arms”, “rolled shoulders, forward head posture”, “lower back pain”, “ Each item such as "leg swelling" and "arm stiffness" is associated with (ii) information on the body part of the user and stored as predetermined information. In addition, the terminal device 10 stores information on a training menu to be proposed as training in association with each item of the user's possible physical concerns.

The terminal device 10 performs a ROM check and a walking motion check to specify a muscle part that is desired to be improved based on these measurement results and evaluation results. For example, in the ROM check, the terminal device 10 stores in advance which muscle part is associated when the twist of the body is not good. For example, in the ROM check, the terminal device 10 holds muscle information in association with evaluation items for the range of motion of shoulders, hips, and the like.

The terminal device 10 identifies possible future troubles by referring to the predetermined information, such as by identifying muscle parts that move hard. The terminal device 10 determines a training menu to be proposed to the user according to the result of the identification so as to cope with possible troubles that may occur in the user's body in the future.

As described above, the information of each part of the muscle is associated with the evaluation items of the balance during walking motion and the torsion of the body, and the information of each part of the muscle and each item of trouble that may occur in the user's body are associated. associated with the information of Thereby, the terminal device 10 can propose a training menu customized for each user to the user based on the evaluation result of the user's body motion.

<Data structure in the fourth embodiment>
FIG. 20 is a diagram showing the data structure of various data stored by the server 20 according to the fourth embodiment.

In the measurement item information 282, the server 20 holds the item "probable future trouble" and the item "related muscle part" for each measurement item.

The item "probable future troubles" is information that identifies possible troubles that may occur in the user's body in the future, depending on the results of evaluating the measurement results for each measurement item (for example, tags such as "stiff shoulders" and "stiff neck"). ).

The item "related muscle part" is information on a body part (muscle part) that is desired to be trained for each measurement item, depending on the evaluation results of the measurement results. In the item "relevant muscle part", the information of the body part may be stored step by step according to the degree of the evaluation result.

In the training item information 285, the server 20 holds the item "content", the item "corresponding trouble", and the item "related muscle part" for each training item.

The item "content" indicates content (video, motion data that serves as a reference for the user's body movement, audio data, etc.) presented to the user when training is performed.

The item "corresponding trouble" is information (for example, tags such as "stiff shoulder" and "stiff neck") specifying a trouble that may occur in the user's body in the future depending on the evaluation result for each training item.

The item "related muscle part" is information on a body part (muscle part) that is desired to be trained for each training item depending on the evaluation result. In the item "relevant muscle part", the information of the body part may be stored step by step according to the degree of the evaluation result.

The server 20 retains the item “possible future troubles” in the evaluation result database 288 . The item "probable future troubles" is information (for example, tags such as "stiff shoulder" and "stiff neck") that specify troubles that may occur to the user's body in the future depending on the evaluation results for each evaluation result.

As described above, the server 20 updates the evaluation result database 288 according to the evaluation result of the user's body movement. For example, according to the measurement results of the ROM check (measurement item information 282), it is possible to specify information about possible future troubles in the user's body. The server 20 refers to the evaluation result database 288 to identify information about possible future worries of the user, and refers to the training item information 285 to extract training items to be proposed to the user. When extracting a plurality of training items to be proposed to the user, the server 20 sets priorities based on a predetermined rule. For example, the server 20 may give priority to a training item capable of resolving the concerns associated with the unsatisfactory evaluation result of the user's body movement, depending on the extent to which the evaluation result is unsatisfactory.

<Screen example in the fourth embodiment>
FIG. 21 is a screen example of a phase in which the terminal device 10 presents possible troubles to the user's body based on the measurement results and evaluation results of the user's body movements, and proposes an exercise menu corresponding to the troubles. be.

FIG. 21(A) displays the result of the terminal device 10 identifying possible troubles of the user's body based on the measurement result and the evaluation result of the user's body motion.

The terminal device 10 prompts the user to take daily measurements via the terminal device 10 on the display unit 132P71, and displays possible future conditions of the user's body on the display unit 132P72. In the illustrated example, the operation reception unit 132P73 displays that the user may suffer from "stiff neck/shoulder", and responds to the user's input operation to the operation reception unit 132P73. Display the training menu corresponding to In addition, the terminal device 10 displays in the operation reception unit 132P74 that the user may be troubled by “sagging of the upper arm”, and responds to the user's input operation to the operation reception unit 132P74, and displays “sagging of the upper arm”. display a training menu corresponding to the troubles of In addition, the terminal device 10 responds to the user's input operation to the operation reception unit 132P75 while displaying that the trouble of "rolling shoulders inward, posture with the head forward" may occur in the operation reception unit 132P75. Then, a training menu corresponding to the problem of "inwardly rolled shoulders, forward head posture" is displayed.

On the display unit 132P76 of the terminal device 10, for users who are short on time, among the training menu candidates to be proposed to the user, the number of training menu candidates for starting a training menu that is smaller than the number displayed on the display unit 132P76. Display the operation reception section 132P77. Based on a predetermined rule, the terminal device 10 sets priorities, etc., to training menu candidates to be proposed to the user, and displays them on the display unit 132P76.

The terminal device 10 receives an operation for editing a training menu to be executed, such as "favorite", from the user on the display unit 132P78.

The terminal device 10 presents a questionnaire to the user by accepting an input operation to the operation accepting unit 132P91, and accepts an answer to the questionnaire. The terminal device 10 may identify the user's worries based on the input result of the questionnaire, and propose a training menu corresponding to the identified worries.

FIG. 21B shows a phase of proposing a recommended training menu to the user. The terminal device 10 displays to the user, on the display unit 132P81, information about body parts (“Do you suffer from stiff neck or stiff shoulders?”, etc.) regarding the troubles identified based on the measurement results and evaluation results. . The terminal device 10 displays, on the display unit 132P81, the cause of the user's concern about the body (such as "the position of the shoulder tends to protrude forward") based on the measurement result and the evaluation result. On the display unit 132P81, the terminal device 10 displays a guideline for resolving the user's physical troubles ("It is expected to improve the blood flow in the upper body," "Stretch the back of the thigh, ” etc.) and suggesting an exercise corresponding to the guideline.

For example, in response to an input operation to the operation reception unit 132P73, the terminal device 10 displays the operation reception units 132P83 to 132P87 for starting training using the recommended training menu, as shown in the display unit 132P82. The terminal device 10 may reproduce a moving image for starting training in response to the input operation to these operation reception units 132P83 to 132P87.

In response to the input operation to the operation reception unit 132P77, the terminal device 10, as shown in the display unit 132P88, narrows down the number of training menus to be proposed to the user who is short on time, and selects the operation reception unit. 132P83 to 132P85 are displayed.

Thereby, it is possible to encourage the user to continue training according to the user's condition at the time.

<Operation example>
The operation of each device in the above fourth embodiment will be described.

FIG. 22 shows the processing flow of each device in the fourth embodiment.

In step S2203, the terminal device 10 evaluates at least one of the user's joint range of motion, walking posture, and trunk balance based on the measurement result of the user's body movement, and outputs the evaluation result. The terminal device 10 transmits measurement results and evaluation results to the server 20 .

In step S2253, the server 20 updates the database (evaluation result database 288, etc.) based on various user measurement results such as measurement results and evaluation results received from the terminal device 10. FIG.

In step S2255, the server 20 identifies possible future physical troubles of the user based on the evaluation result database 288 and the like. Based on the training item information 285, the server 20 selects a training menu related to the body part corresponding to the identified trouble from among a plurality of training menus, and responds to the terminal device 10 of the user with the selected training menu.

In step S2205, the terminal device 10 displays to the user a training menu corresponding to possible future physical troubles.

<Modification>
In the fourth embodiment, a question may be presented to the user, and the priority of the training menu to be proposed to the user may be set according to the answer (questionnaire result) received from the user.

This not only proposes a training menu based on the measurement results of the user's body, but also presents questions about the user's body and proposes a training menu based on the results of receiving answers to the questions from the user. be able to. That is, the server 20 updates the priority of the training menu proposed to the user based on the results of the answers to the questions. The server 20 presents the user with a training menu having a high priority for the user, based not only on the physical evaluation results but also on the results of the user's answers to the questions. Thereby, the server 20 can propose a training menu more suitable for the user's physical condition.

For example, the terminal device 10 accepts a questionnaire about the user's body, thereby determining whether the muscles of the user's body that have not yet been measured by the measurement unit 30 in the measurement system 1 (evaluation based on the measurement results has not been performed). can receive information on the part of the from the user. The terminal device 10 can also identify possible future concerns regarding muscle parts that have not yet been evaluated by the measurement system 1, based on the results of the questionnaire input by the user. The terminal device 10 proposes to the user how to appropriately deal with future concerns that may arise in the user's body, based on the concerns identified based on the measurement results of body movements and the results of questionnaire responses from the user. Decide on a training menu.

<Data structure in modified example>
FIG. 23 is a diagram showing the data structure of various data stored by the server 20 in the modification of the fourth embodiment.

In the user questionnaire information 289, the server 20 divides each questionnaire result into an item "answer option", an item "current trouble", an item "possible future trouble", and an item "related muscle part". Hold.

The item “answer options” includes information on options for the user to answer questions (questionnaire) presented to the user of the terminal device 10 . For example, if the content of the question is that the user is feeling unwell, the user is presented with options for possible body parts. In this case, it is possible to identify the user's concern about the body part according to the result of the user's designation for each option.

The item "current troubles" is information specifying the physical troubles that the user is likely to have (for example, "back pain", "knee pain”).

The item "probable future troubles" is information that identifies possible troubles that may occur to the user's body in the future (for example, "stiff shoulder", "stiff neck", etc.) depending on the evaluation results of evaluating the contents of the questionnaire responses for each questionnaire result. tag).

The item "related muscle part" is information on a body part (muscle part) that is desirable to be trained for each questionnaire result, depending on the evaluation result of evaluating the contents of the questionnaire answers. In the item "related muscle part", the information of the body part may be stored step by step according to the degree of the content of the answer.

The server 20 holds the item “current worries” in the evaluation result database 288 . The item "current trouble" is information specifying the physical trouble that the user is likely to have (for example, "back pain", "knee pain", etc.) depending on the evaluation result of evaluating the contents of the questionnaire responses for each of the evaluation results. pain”).

<Operation in modified example>
As described above, the server 20 updates the evaluation result database 288 according to the user's body movement evaluation results and the user questionnaire information 289 . For example, according to the measurement results of the ROM check (measurement item information 282), it is possible to specify information about possible future troubles in the user's body. For example, according to the result of a questionnaire about the body from the user (user questionnaire information 289), it is possible to identify the physical troubles that the user currently has or will have in the near future. The server 20 refers to the evaluation result database 290 to identify information about the user's possible future worries, and refers to the training item information 285 to extract training items to be proposed to the user.

When extracting a plurality of training items to be proposed to the user, the server 20 sets a priority based on a predetermined rule according to the extent to which the evaluation result of the user's body movement is not good. For example, the server 20 may give priority to a training item capable of resolving the concerns associated with the unsatisfactory evaluation result of the user's body movement, depending on the extent to which the evaluation result is unsatisfactory. The server 20 corrects and sets the priority of training items to be proposed to the user based on the results of the questionnaire. For example, the server 20 selects training items that can solve the user's physical troubles identified from the questionnaire results, depending on the extent to which the physical evaluation results related to the contents of the questionnaire answers are less favorable than the evaluation results of the ROM check. may be given priority.

FIG. 24 is a diagram showing the flow of operations in the modification of the fourth embodiment.

As shown in FIG. 24, the terminal device 10 presents a question (questionnaire) to the user of the terminal device 10 at any timing such as step S2202 or step S2204.

The processing of step S2202 will be described. The terminal device 10 displays the question on the display 132, for example, in response to the user completing the process of logging into the measurement system 1 in step S700. The terminal device 10 receives an input of an answer to the question from the user. The terminal device 10 transmits to the server 20 the response result for which the input was accepted.

In step S<b>2252 , the server 20 updates the user questionnaire information 289 based on the information received from the terminal device 10 .

In step S2253, the server 20 stores various databases such as the measurement result database 287 in response to receiving from the terminal device 10 the measurement result by the measurement unit 30 and the like in step S2203 and the evaluation result based on the measurement result. Update.

The processing of step S2204 will be described. For example, the terminal device 10 presents a question to the user (displays the question on the display 132, etc.) in response to completing the measurement of the user's body movement in step S2203. The terminal device 10 transmits to the server 20 the response result for which the input was accepted.

In step S<b>2254 , the server 20 updates the user questionnaire information 289 based on the information received from the terminal device 10 .

In step S2256, the server 20 identifies the user's current troubles and possible future physical troubles of the user based on the user questionnaire information 289, the evaluation result database 288, and the like. Based on the training item information 285, the server 20 selects a training menu corresponding to the identified trouble from among a plurality of training menus, and responds to the user's terminal device with the selected training menu. For example, the user's worries are specified based on the results of the user's answers to the question sentences (user questionnaire information 289), and the training menu to be proposed to the user is specified according to the specified worries (training item information 285). . Moreover, the server 20 identifies a training menu to be proposed to the user based on the evaluation result database 288 as described above. The server 20 responds to the terminal device 10 regarding one or more training menus identified in this manner. At this time, the server 20 may set priorities for the specified training menus based on a predetermined rule, and respond to the terminal device 10 with the training menus according to the priorities.

<Screen example 1 in the fourth embodiment>
FIG. 25 shows a situation in which the terminal device 10 presents possible troubles of the user's body based on the measurement results and evaluation results of the user's body movements, and proposes an exercise menu corresponding to the troubles. It is an example of a screen of a stage of answering a questionnaire. For example, the terminal device 10 accesses the server 20 or the like to display the screen (display unit 132P71) shown in FIG.

The terminal device 10 displays the screen at the timing (timing at which information for displaying the screen is received from the server 20 and displayed by a browser or the like), or when a predetermined time has elapsed since the screen was displayed. In response to the satisfaction of predetermined conditions such as timing and that the user has performed a predetermined operation on the screen, the display unit 132PA2 shown in FIG. 25A is displayed to prompt the user to answer the questionnaire.

FIG. 25(A) shows a screen (FIG. 21(A)) on which the terminal device 10 displays the results of identifying possible troubles of the user's body based on the measurement results and evaluation results of the user's body movements. The display unit 132PA2 for the user to answer the questionnaire is displayed so as to be overlaid on the display unit 132P71) described in .

The terminal device 10 displays, on the display section 132PA2, a question sentence for the user ("What kind of trouble do you have with your body?") on the display section 132PA3.
Server 20 determines the content of the question to be presented to the user on display unit 132PA3. For example, the server 20 holds a plurality of question sentences in advance, and holds information about body parts of the user and information of candidate answers to the question sentences in association with each question sentence. The server 20 refers to the measurement result database 287 and the evaluation result database 288, determines a question to be presented to the user based on the history of measurements performed by the user, and causes the terminal device 10 to display the determined question. You can do it. For example, based on the user's history of measurements with the measurement system 1 (such as the time intervals between measurements), the server 20 determines if a certain period of time has passed since the user last measured the physical condition of the body. It is also possible to present the user with a question (such as "What kind of trouble do you have in your body?") as to whether or not the user is aware that each part is ill.

Further, for example, the server 20 may determine body parts to be included in a question to be presented to the user based on the measurement results of the user's body parts. For example, if there is a part of the user's body that has not been measured for at least a certain period of time (for example, there is no history of measuring the waist, or a certain period of time has passed since the waist was measured) (i.e., "How is your waist?"), and the relevant body part may be included in the question.

In the example of FIG. 25(A), the terminal device 10 displays candidate answers to the question on the display unit 132PA4, and accepts selection from the user. In the illustrated example, on the display unit 132PA4, as candidates for the answer to the question sentence, "waist trouble" for designating a waist-related disorder, "knee trouble" for designating a knee-related disorder, and other physical conditions are displayed. "***" is displayed for displaying answer candidates for the body part. The terminal device 10 accepts the designation of "low back pain" from the user, and by highlighting the candidate, the user can easily recognize that the user has designated it.

The terminal device 10 displays a question about the degree of anxieties (the degree of disorder of a part of the body) on the display unit 132PA5.

The terminal device 10 displays, on the display unit 132PA6, answer candidates for the question displayed on the display unit 132PA5, and accepts selection from the user. In the illustrated example, the terminal device 10 displays, on the display unit 132PA6, “mild” indicating a light level of concern, “medium” indicating a medium level of concern, and “heavy” indicating a high level of concern, as options. ing. The terminal device 10 accepts the specification of the option “moderate” from the user and highlights the specified option.

The terminal device 10 receives an operation from the user on the display unit 132PA1 to indicate that the answer to the question has been completed. Thereby, the terminal device 10 transmits the user's answer result to the server 20 . The server 20 updates the user questionnaire information 289 by receiving the user's answers. For example, when the user specifies information about a body part such as "waist trouble" as an answer result, the server 20 updates the user questionnaire information 289 based on the user's trouble "waist trouble".

FIG. 25B is a screen example of a situation in which a training menu is proposed to the user based on the result of receiving the user's answer to the question, the measurement result of each part of the user's body, and the evaluation result. .

Based on the results of the user's response to the question (user questionnaire information 289), the server 20 collects information about the user's troubles about the body part ("You seem to have a moderate burden on your knees," etc.). information including the body part and the degree of worry, etc.) is displayed on the display unit 132P81. The terminal device 10 displays the cause of the user's concern about the body ("Are you in an unreasonable posture for a long time?", etc.) on the display unit 132P81. The terminal device 10 displays, on the display unit 132P81, a guideline for relieving the user's physical troubles (eg, "relax the muscles around the waist") and a suggestion of an exercise corresponding to the guideline.

<Other Modifications>
(1) Asking User Questions about Exercise Experiences In the above description of the embodiments, an example in which the server 20 presents a question sentence to the user and proposes a training menu based on the result of the answer to the question. explained. The server 20 may ask the user about their experience of exercising as the content of the question presented to the user.

For example, the server 20 may present the user with a question about his/her experience of playing sports in the past and habits of exercising. The server 20 receives an answer to the question from the user (such as the sports the user has participated in in the past, whether or not the user has a habit of exercising, etc.). Based on the user's answer, the server 20 asks the user if the user has relatively little experience of exercising (for example, the length of time the user has played sports is less than a certain period of time, the habit of exercising is less than a predetermined frequency, etc.). On the other hand, among the training items included in the training item information 285, a training item with a relatively small load on the body or a training with a relatively low degree of difficulty may be proposed. For example, for each training item in the training item information 285, the degree of difficulty (training that can be performed by a person who is not accustomed to exercise without problems is lower, etc.), the load on the body (for example, a specific part (upper arm, etc.) ), the load on the body is different for each training method). Thereby, a training menu can be presented based on the user's exercise experience.

According to the above example, by proposing a training menu that is relatively easy to achieve, a user who is not accustomed to exercise can easily feel a sense of accomplishment by completing the training menu. This can motivate the user to continue training.

(2) Evaluate whether the user is accustomed to exercise based on the measurement results of the user's body movement. may be evaluated, and a training menu to be presented to the user may be determined based on the evaluation results.

For example, the server 20 associates the measurement results of the user's body movements, such as the range of motion of the joints, the walking posture, and the balance of the trunk, with the patterns of the measurement results when the user is not accustomed to exercise, and the patterns of the measurement results when the user is accustomed to the exercise. and the pattern of the measurement result of . The server 20 evaluates the degree to which the user is accustomed to exercise based on the measurement result of the user's body movement. For example, the server 20 outputs an evaluation value indicating whether or not the user is accustomed to exercising by comparing the above pattern with the measurement result of the user's body movement and evaluating the degree of agreement. For example, the server 20 presents a model body movement to the user and evaluates that the user is accustomed to exercising when the user can move the body to the same degree as the body movement. At this time, if there is a discrepancy between the movement of the model's body and the measurement result of the user's body movement, a voice may be output to the user so as to correct the discrepancy. For example, if the position of the hand is different from the modeled movement, the voice output prompts the hand to follow the modeled movement (such as "move the hand position in a predetermined direction"). good too. Also, when the user's body movement and the model movement match (that is, when the user can perform the same movement as the model movement), the user's body movement is appropriate. output by voice, etc.

For example, when measuring the movement of the user's body, the server 20 determines whether the user is accustomed to exercise based on whether the user can move the body according to the video or audio guidance presented to the user. It may be output. For example, the server 20 may evaluate that the user is used to exercising when the user is able to move his/her body in accordance with the voice output.

The server 20 determines a training menu to be proposed to the user based on the evaluation value.

As described above, the server 20 holds the history of the measurement results by the measurement unit 30 in the measurement history database 283, and provides the user with the time and evaluation result of evaluating whether the user is accustomed to exercise, A history of training menus proposed to the user may be displayed. Thereby, the user can confirm that the user is gradually accustomed to the exercise and that the training menu is changing, and can realize a sense of self-efficacy.

(3) Asking User a Question About Continuing Training In addition, the server 20 may present a question about continuing training to the user. For example, questions may be presented regarding the user's personality, whether or not he or she was able to continue training in the past, and the length of time until training is stopped.

The server 20 determines the training menu to be presented to the user based on the results of the answers to the questions. For example, as described in the above fourth embodiment and the like, it is assumed that a problem that may occur in the user's body is identified and a training menu related to body parts associated with the problem is presented to the user. If the user gives a negative answer as to whether or not the training can be continued (there is a tendency that the training cannot be continued, etc.), the server 20 changes the training menu proposed to the user according to the task. You can do it. For example, when there are a plurality of training menus related to body parts associated with the task, another training menu is proposed to the user. This is expected to make it easier for the user to work on the training without getting bored, so that the user can continue the training.

Based on the measurement history database 283 and the like, the server 20 identifies intervals between timings at which the user has performed training. That is, the frequency with which the user exercises is specified. Server 20 may present questions to the user regarding continuing training based on the length of the interval. For example, when the frequency of training is decreasing, a different training menu can be proposed so that the user does not get bored by accepting questions about continuing training from the user. Also, the server 20 may change the training menu proposed to the user based on the length of the interval. As a result, the user can continue training without getting bored, and the user can continue training.

<Fifth Embodiment>
<Overview>
In the fifth embodiment, a technique will be described in which the user can more easily check whether or not the user is making ideal movements by moving his/her body.

(1) By visually reducing the amount of information about the body movement when the user moves the body, the body movement can be easily confirmed.
Specifically, when the user performs a specific exercise (for example, exercise for evaluating trunk muscle movement and joint range of motion such as side elevation, or exercise for strength training), the user's The motion of the body is sensed as point cloud data, and based on the point cloud data, an avatar corresponding to the user and auxiliary lines corresponding to the motion of the user's body are drawn and presented to the user. Thereby, the user can more easily check whether each part of the body can be moved as intended.

(2) Make it easier for trainers to guide trainees online using video conferencing tools and the like.
In the fifth embodiment, based on the above technology, a trainee (first user) who receives instruction on how to move the body and a trainer (second user) who instructs the trainee on how to move the body, Describe the aspects of teaching online.

Specifically, by enabling the trainer to check the avatar that expresses the trainee's body movement as point cloud data on the trainer's terminal, the trainer can view not only the image of the trainee's body, but also the image of the trainee's body. To make it possible to check the movement of an avatar with less visual information than images. This makes it easier for trainers to give trainees online advice on how to move their bodies.

That is, in the fifth embodiment, (1) the first processor of the terminal of the trainee (first user) senses the movement of the trainee (first user) as point cloud data using the measurement unit 30 or the like. (2) The first processor transmits information based on the point cloud data of the trainee (first user) (for example, an image of the trainee's avatar drawn based on the point cloud data, or the point cloud data itself, etc.) to the trainer (first user) A series of techniques for performing various processes such as transmitting to terminals of two users) will be described.

Here, the body movement of the trainee (first user) is sensed as point cloud data, and the first processor transmits information based on the point cloud data of the trainee (first user) to the terminal of the trainer (second user). However, devices that execute various processes include terminal devices such as computers, smartphones, and tablet devices possessed by users.

<Data structure in the fifth embodiment>
FIG. 26 is a diagram showing data structures of a trainee information database 2601, a trainer information database 2602, and a training history database 2603 stored by the server 20 in the fifth embodiment.
Although not shown, the server 20 also detects the types of body movements performed by the user (movements for evaluating the range of motion of joints such as side elevations, movements for strength training such as push-ups, Predetermined body movements such as yoga and other types of movement), information on the type of auxiliary lines when displaying the user's body movements as an avatar of a collection of point cloud data, and the auxiliary lines (position associated with the part of the user's body represented by the avatar) at which the .

The trainee information database 2601 is a database for managing information on users who become trainees. The trainer information database 2602 is a database for managing information on users who become trainers. The training history database 2603 is a database for managing the history of the trainee's training under the guidance of the trainer.

As shown in FIG. 26, the trainee information database 2601 includes an item "trainee ID", an item "name", an item "age", an item "sex", an item "weight", and an item "body fat percentage". , the item "purpose", the item "ideal figure", the item "screen display", the item "physical fitness level", the item "remarks", and the like.

The item "trainee ID" is information for identifying each user who receives training.

The item "name" is information indicating the name of each user who receives training.

The item "age" is information indicating the age of each user who receives training.

The item “gender” is information indicating the gender of each user who receives training.

The item “weight” is information indicating the weight of each user who undergoes training.

The item "percentage of body fat" is information indicating the percentage of body fat of each user who undergoes training.

The item "purpose" is information indicating the purpose of each user who receives training. Specifically, the user identified by the name "Mori" with the user ID "U001" has the purpose of training to "shape up", and the user identified by the name "Sato" with the user ID "U002" "Diet" is the purpose of training. Here, in a certain aspect, a user (trainer) who is in a position to instruct a user (trainee) undergoing training may present a training menu to the user (trainee) according to the purpose. In a certain aspect, terminal device 10 or server 20 may specify a training menu to be presented to the user based on a learned model or the like, which will be described later.

The item "ideal figure" is information indicating the ideal figure of each user who undergoes training. Specifically, the user identified by the name "Mori" of the user ID "U001" has an ideal figure of "not specified", and the user identified by the name "Sato" of the user ID "U002" "〇〇" is specified as the ideal figure. Here, this ideal figure may be specified by a real entertainer or the like, or a fictional figure (for example, if the body shape of a cartoon character is ideal, the user can specify the character). may be In a certain aspect, server 20 may specify and present a training menu based on the contents of the "ideal figure" and "purpose".

The item "screen display" is information indicating a display mode when the user is photographed by a camera or the like. As a display mode, when the user is photographed with an RGB camera or the like, the photographed image generated by the photographing is displayed as it is, and a sensor capable of measuring distance (depth camera, LiDAR (light detection and ranging), etc.) ) measures the movement of the user's body as point cloud data and displays it as an avatar based on the point cloud data. Specifically, the user identified by the name "Mori" of the user ID "U001" is displayed as "avatar" on the screen, and when the moving image of the user captured by a camera or the like is displayed on the screen, It indicates that the user himself/herself is displayed as an avatar (for example, an avatar using a character) based on the captured moving image instead of the captured moving image as it is.

As a result, it is possible to respond to the requests of the user undergoing training, for example, because it is the first training, the user does not want others to see his/her face and body, and wants to protect privacy, so that the user can receive the training with peace of mind. can be done.

The item “physical fitness level” is information indicating the physical fitness level obtained by evaluating the physical exercise ability of each user who undergoes training in stages. Specifically, when sensing the movement of the user's body undergoing training, at the same time, based on the information on the movement of the user's body, the physical fitness level is used as a criterion for judging how much exercise the user is likely to be able to do. are identified. The user identified by the user ID "U001" and the name "Mori" has a physical strength level of "5", and the user identified by the user ID "U002" and the name "Sato" has a physical strength level of "2". be. At this time, as a method of specifying the physical fitness level, the physical fitness level of the user may be specified based on a learned model or the like when the terminal device 10 or the server 20 senses the movement of the user's body. A user (trainer) who is in a position to instruct the user (trainee) who receives the training may specify the physical fitness level of the trainee based on the sensing result.

As a result, the user can receive a training menu suitable for his or her physical fitness level, so that the user can continue training without difficulty. In addition, in the case of group training, the server 20 can identify users with the same physical strength level and match users (trainers, trainees) so that the identified users can train together. For example, when users (trainees) who have different physical strength levels perform training, there may be differences in the trainer's instruction content and the trainee's training content, which may lead to a decrease in motivation. You can prevent the physical strength level so that.

The item "remarks" is information indicating circumstances that should be specially considered for each user who undergoes training. Circumstances that should be taken into special consideration include items that should be considered based on safety during training (safety based on a medical point of view, safety of the training content itself, etc.). For example, if the user has high blood pressure and the doctor has stopped excessive training, the relevant matter is described in the remarks. The server 20 may specify the optimal training menu for the user who receives the training based on the contents described in the item "Remarks". In addition, by acquiring the information of the item "remarks" from the server 20 and presenting it to the trainee in the terminal device 10B of the trainer, the trainer can provide information that can be used as a reference when determining the training menu for the trainee. good too.

As a result, even if the user has a chronic disease, the server 20 or the like identifies the training menu to be presented to the user based on the item "remarks", and the trainer selects a training menu based on the item "remarks". As a result, the user can continue training without difficulty.

The trainer information database 2602 includes an item “trainer ID”, an item “name”, an item “gender”, an item “years of experience”, an item “strength area”, an item “recommended physical fitness level”, and an item “guidance possible types” and items such as “remarks”.

The item “trainer ID” is information for identifying each user who is in a position to instruct the user who is training.

The item "name" is information indicating the name of each user who is in a position to guide the user who is undergoing training.

The item “gender” is information indicating the gender of each user who is in a position to guide the user who is undergoing training.

The item "years of experience" is information indicating the number of years of experience in training guidance of each user who is in a position to guide the user who is to undergo training. Specifically, the user identified by the name “Tanaka” with the trainer ID “T001” has “5 years” of experience, and the user with the trainer ID “T003” with the name “Kato” is shown. indicates that the user identified by has "10 years" of experience.

As a result, for example, in a certain situation, when a user (trainee) who receives training designates a user (trainer) who is in a position to give guidance, by designating a user who is in a position to give guidance with reference to the number of years of experience, You can train with confidence. As a user (trainer) who is in a position to give guidance, by being designated as a user who is in a position to give guidance based on the information, it is possible to improve his/her name recognition.

The item "area of specialty" is information indicating the area of specialty in training guidance of each user (trainer) who is in a position to instruct a user (trainee) who is to undergo training. Specifically, the user identified by the name “Tanaka” of the trainer ID “T001” is good at “yoga” and “exercise”, and the user identified by the name “Saito” of the trainer ID “T002”. indicates that "muscle training" and "exercise" are areas of expertise.

As a result, the server 20 or the like can be a user (trainer) who is in a position to instruct the user in accordance with information such as the item "purpose" and the item "ideal figure" of the above-described trainee information database 2601 for the user undergoing training. may be specified. As a result, it is possible to match a trainer that matches the purpose of the trainee's training. For example, while a trainee aims at exercise, the server 20 or the like can assign a trainer who is good at training that meets the purpose as a leader.

The item "recommended physical fitness level" indicates the physical fitness level of the trainee, which is defined for each user (trainer) who is in a position to instruct the user (trainee) who receives training and is recommended when receiving training from the user (trainee). . Specifically, for guidance from a user (trainer) identified by the name "Tanaka" of the trainer ID "T001", the recommended physical strength level is set to "3 or higher", and the user ID "U001" Since the physical strength level of the user (trainee) identified by the name "Mori" is "5", it can be seen that the physical strength level for receiving guidance from the user has been reached. Since the user (trainee) identified by the name "Sato" in the above has a physical fitness level of "2", it can be seen that the physical fitness level has not reached the standard for receiving guidance from the user.

As a result, the user who undergoes training can receive training suitable for his/her physical fitness level. In addition, since users who are in a position to instruct users who receive training can also instruct training to users who are above a predetermined level, it is possible to suppress the occurrence of variations in training menus and proficiency levels among users. It can also be expected to maintain the motivation of users who are undergoing training. The server 20 or the like may match the trainer and the trainee based on the item "recommended physical fitness level" of the trainer and the item "physical fitness level" of the trainee.

The item "guidable type" indicates the type of training that can be instructed by a user (trainer) who is in a position to instruct a user undergoing training. Specifically, the user identified by the name “Tanaka” with the trainer ID “T001” has the coachable types “individual” and “group”, and is identified by the name “Saito” with the trainer ID “T002”. The user indicates that the coachable type is "individual" only.

As a result, it can be seen whether the user (trainer) in the position of instructing the user undergoing training is suitable for individual instruction or for group instruction. In matching with , it is possible to distribute training by one-to-one individual guidance and training by one-to-many group guidance. Also, the terminal device 10 of the trainee may present the trainer's "trainable type" to the trainee.

The item "remarks" is information such as other special notes of each user (trainer) who is in a position to instruct the user who is to undergo training. Remarks may include information that objectively evaluates the performance of training by a trainer. Specifically, it can be seen that the user identified by the name "Tanaka" of the trainer ID "T001" is "** grand prix winner".

As a result, in a certain situation, when a user (trainee) undergoing training designates a user (trainer) in a position to instruct the user undergoing training, information in the item "remarks" is provided to the trainee on the terminal device 10 of the trainee. By presenting, the trainee can designate a trainer with this information also taken into consideration. A user (trainer) who is in a position to instruct a user undergoing training can also expect to increase the chances of being designated as a trainer by recording the information.

The training history database 2603 includes an item “training ID”, an item “date and time”, an item “meeting ID”, an item “trainee ID”, an item “trainer ID”, an item “sensing result”, an item “guidance content”, item “trainer evaluation”, item “matching ID”, and the like.

The item “training ID” is information for identifying training by trainers and trainees.

The item "date and time" indicates the date and time when the training was performed.

The item "meeting ID" is information for identifying each communication session in the case of conducting online training guidance through communication between the terminal device 10B of the trainer and the terminal device 10A of the trainee. For example, information identifying a communication session includes information such as a URL for an online meeting.

The item "trainee ID" is information for identifying the trainer who has performed the training.

The item “trainer ID” is information for identifying a trainee who received training guidance from the trainer.

The item "sensing result" is the result (point cloud data) of the terminal device 10A measuring the movement of the trainer's body with the measurement unit 30 during training, or the point cloud data is placed in the virtual space and set to the virtual camera. (Avatar expressing the trainee's body movement as a set of point cloud data in the video and how the trainee moves (the type of body movement presented to the user, such as side lifting)) (including auxiliary lines drawn according to ).

The item "guidance content" indicates the content of guidance given by the trainer to the trainee regarding how to move the trainee's body. For example, when the trainer instructs the trainee how to move the body by voice (including advice to guide the user to the desired posture in how to move the body. (Trainer's advice to take a desired posture when arms, legs, etc. are bent), the voice including the guidance content, or the analysis result of the voice is stored in the item "guidance content".

The item "trainer evaluation" indicates the evaluation received by the trainer from the trainee. For example, after conducting online training, the trainer's evaluation is accepted from the trainee by presenting the trainee with a screen for inputting the trainer's evaluation on the trainee's terminal device 10A.

The item “matching ID” is information that identifies each matching when the server 20 matches the trainer and the trainee. If the matching is established, the trainer and the trainee may start training, or even if the matching is established, the training may not be carried out.

<Example of operation in the fifth embodiment>
Hereinafter, in the fifth embodiment, the first processor of the terminal of the first user (trainee) in the system 1 senses the body movement of the first user as point cloud data, and the first processor senses the first A series of processes when information based on the user's point cloud data is transmitted to the terminal of the second user (trainer) and various processes are executed will be described.

(Sensing the first user's movement and sending it to the second user's terminal)
FIG. 27 is a flowchart showing a series of processes for sensing the movement of the first user's body as point cloud data and transmitting information based on the first user's point cloud data obtained by sensing to the second user's terminal. is.

In the following description, regarding the terminal device 10 shown in FIG. 1, the terminal device 10 of the first user is referred to as "terminal device 10A", and the configuration of terminal device 10A is referred to as "processor 19A" and the like. Similarly, the terminal device 10 of the second user is described as "terminal device 10B", and the configuration of terminal device 10B is described as "processor 19B" and the like.

Further, in the example of FIG. 27 , by establishing communication between the terminal device 10A of the first user and the terminal device 10B of the second user, the video captured by the camera 160 of each terminal and the sound collected by the microphone 141 are collected. It is assumed that the two terminals transmit and receive voice to and from each other.

For example, the terminal device 10A and the terminal device 10B are running an application for conducting an online video meeting, and identification information for participating in the meeting (for example, meeting ID issued by the application, URL etc.).

The terminal device 10A acquires the movement of the first user's body as point cloud data using the measurement unit 30 or the like. The terminal device 10A arranges the point cloud data in the virtual space, and sets the virtual camera in the virtual space to the position of the front of the first user's body (front-rear axis in the description of FIG. 11) and the side of the first user's body. (the left and right axis in the description of FIG. 11), rendering is performed, and the rendering result is displayed. The rendering result includes an avatar as a set of point cloud data when the first user is viewed from the front and an avatar as a set of point cloud data when the first user is viewed from the side. The terminal device 10A draws an auxiliary line corresponding to how the user moves the body, in association with the display area of the rendering result (the avatar viewed from the front and the avatar viewed from the side).

The terminal device 10A displays an image drawn with the avatar and auxiliary lines on the screen, and screen-shares the image with the terminal device 10B by an application for online video meeting. As a result, the image of the first user captured by camera 160A, the sound picked up by microphone 141, and the image in which the avatar and auxiliary lines are drawn are transmitted from terminal device 10A to terminal device 10B. The second user of the terminal device 10B confirms the movement of the body of the first user by visually recognizing the video captured by the camera 160A and the image shared on the screen (the image in which the avatar and auxiliary lines are drawn). Then, the advice to the first user is notified to the first user by voice or the like.

In step S2701, the processor 19A of the terminal device 10A of the first user senses the movement of the first user's body as point cloud data using the measurement unit 30 or the like.

Specifically, the processor 19A displays to the first user a menu of body movements performed by the first user so that the first user can perform specific movements (e.g., side movements) for body movement sensing. Encourage them to do movements such as side lifts, movements that can be used as training items (squats, etc.). The processor 19A of the terminal device 10A of the first user captures the specific movement using the measurement unit 30 or the like, thereby sensing the body movement of the first user as point cloud data.

In step S2703, the processor 19A generates an avatar representing the first user (trainee) and body movements performed by the first user as information based on the first user's point cloud data obtained by sensing by the measurement unit 30 or the like. (type of exercise menu presented to the first user).

In step S2705, the processor 19A displays the image generated in step S2703 on the display 132A of the terminal device 10A. The processor 19A processes the displayed image (image including the avatar and the auxiliary lines) and the separately captured image generated by the camera 160A that captures the first user (not the set of point cloud data, but the image captured by the RGB camera). captured image) to the terminal device 10B of the second user.

In step S2751, the processor 19B of the terminal device 10B of the second user receives from the terminal device 10A of the first user the image (avatar and auxiliary line ). The processor 19B receives the captured image of the first user (the captured image that is not the set of point cloud data) generated in step S2705.

In step S2753, the processor 19B generates the first avatar corresponding to the movement of the first user's body as a set of point cloud data based on the video (including the avatar and auxiliary lines) received in step S2751 and the captured image. is displayed on the display 132B or the like. The processor 19B displays the captured image of the first user (not the set of point cloud data) on the display 132B or the like. As a result, the first avatar and the photographed image of the first user are presented to the second user as the information on the body movement of the first user.

Here, in step S2703, the processor 19A identifies various types of information regarding the first user's exercise (which part of the body is easy to move, hard to move, etc.), physical fitness level, etc. based on the trainee information database 2601 of the server 20. It is also possible to At this time, the processor 19A may store the specified information in the storage unit 180 of the terminal device 10A of the first user, or may transmit the information to the server 20 and store it in the trainee information database of the storage unit 202 of the server 20. 2601 may be stored.

Further, the processor 19A refers to the trainer information database 2602 based on the result of sensing the body movement of the first user by the measurement unit 30 or the like, and identifies the second user (trainer) to whom the information is to be transmitted. You may

As a result, a user who is a trainee can receive training from an optimal instructor based on the information about the body movement at the start of training.

Here, in step S2751, data sensed by the measurement unit 30 or the like is received from the terminal device 10A of the first user, and rendering is performed in the terminal device 10B based on the point cloud data based on the received information. may be That is, the terminal device 10B, not the terminal device 10A, may render the first avatar representing the body movement of the first user based on the point cloud data measured by the terminal device 10A. The terminal device 10B generates an image including the first avatar, and displays the generated image on the display 132B or the like. At this time, the motion of the body of the first user may be drawn from a plurality of viewpoints, and an image in which the first avatars drawn for each viewpoint are arranged may be presented to the second user.

In one aspect, the terminal device 10B draws an auxiliary line at a position based on the body part for evaluating whether or not the body movement performed by the first user is ideal, and displays the video with the drawn auxiliary line as a second user. may be presented to the user. For example, when the trainer evaluates whether the user's body movement is ideal based on whether the user's back is straight and whether the knee is at a predetermined angle to the ground in the exercise that the user should perform , the terminal device 10B displays an auxiliary line corresponding to the shape of the stretched back muscle near the part of the back muscle of the avatar. When the terminal device 10B evaluates whether the knees are at a predetermined angle with respect to the ground, the terminal device 10B places the knees near the avatar's knees (or in the region where the knees are displayed) at the predetermined angle. display the auxiliary lines near the avatar's knees.

In addition, the display mode of the body surface position other than the joint positions is changed based on the part of the body that is evaluated as to whether or not the body movement performed by the first user is ideal, and the display mode of the body surface is changed. The video may be presented to the second user.

The surface of the body that is not the joint position is, for example, a surface that can be recognized as the external appearance of the body, such as the surface of the abdomen. For example, in the terminal device 10B, the displayed color may be changed depending on whether the movement of the first user's body is too large or too small compared to the ideal movement. For example, when the abdomen protrudes too far forward than the ideal movement, the abdomen is displayed in a warm color, and when the movement of the abdomen is smaller than the ideal movement, the abdomen is displayed in a cold color. , to change the colors displayed. Accordingly, the first user or the second user can easily confirm how to move the user's body in an ideal manner. In addition, this allows the user to train while grasping how much the movement of his/her body deviates from the ideal movement. In addition, by changing the display mode of the body surface, it is possible to grasp the movements of the muscles on the body surface instead of the joints.

Here, in the above example, regarding the video including the first user's first avatar displayed on the display 132B or the like, the first user's captured image captured when generating the first avatar and the first avatar's described as presenting both to the second user. In addition, the terminal device 10B on the trainer side may refer to the trainee information database 2601 and display a screen according to the designation of the item "screen display" on the display 132 or the like.

For example, in the terminal device 10B, the movement of the body of the first user is displayed as the first avatar (that is, the collection of point cloud data), while the captured image of the first user (not the collection of point cloud data) may be hidden. Further, when a plurality of trainees receive training from a trainer, the photographed images of the trainees (those that are not aggregates of point cloud data) can be displayed on the terminal device 10 of the trainer, and simultaneously with the trainer. It may be hidden on the terminal devices 10 of other trainees who are taking the training. As a result, both the point cloud data and the photographed image of the trainee's sensing results can be disclosed to the trainer, while the avatar (collection of point cloud data) is shared among the trainees, but the photographed image is not shared. can also

This allows users to train while protecting their privacy.

Also, in one aspect, at least one of processor 19A, processor 19B, or processor 29 of server 20 defines a virtual space and displays an image including the first avatar associated with the first user in the virtual space. and a second object for displaying a video including a second avatar associated with a third user who trains with the first user, and processor 19A or processor 19B performs virtual space An image including the first object and the second object may be displayed on display 132A, display 132B, or the like.

As a result, the user can experience training in the same virtual space, such as training performed by a plurality of people, and can obtain a sense of unity and a sense of accomplishment in the training, which can be expected to maintain and improve motivation.

Next, with reference to FIG. 28, the flow of processing when the trainer instructs the trainees online will be described.

In the following description, while the trainer instructs the trainee by voice, by controlling the application running on the trainee's terminal device 10 with the trainer's voice, the trainer can easily obtain information suitable for the instruction. An example of how to Specifically, on the trainee's terminal, the trainer can control the drawing of the avatar based on the point cloud data, the drawing of the auxiliary lines, and the determination of the type of body movement to be imposed on the trainee by voice input. make it

FIG. 28 is a flow chart showing the flow of processing for receiving an input of the trainer's voice from the trainer's terminal and transmitting it to the trainee's terminal.

In step S2851, the processor 19B of the terminal device 10B of the second user accepts the voice input of the second user through the microphone 141B or the like, and transmits the voice of the second user from the terminal device 10B of the second user to the voice of the first user. It transmits to the terminal device 10A.

In step S2801, the processor 19A of the terminal device 10A of the first user outputs the voice of the second user received from the terminal device 10B to the first user through the speaker 142A or the like.

In step S2803, the processor 19A analyzes the second user's voice received from the terminal device 10B by voice recognition. The processor 19A identifies the content of the second user's voice based on the result of the voice recognition analysis. The processor 19A identifies the content of the instruction to the application for drawing the avatar based on the point cloud data of the first user, based on the identified content of the voice of the second user. The processor 19A controls the application based on the specified instruction to the application.

Specifically, the content of the voice of the second user specified by the processor 19A is as follows:
(i) There is content for making the first user's body movement ideal. The processor 19A, for example, through voice recognition, recognizes words related to the user's body parts (knees, elbows, arms, shoulders, etc.) in the voice of the second user. In addition, the processor 19A selects, from the voice of the second user, a word indicating how to move the body part of the first user in an ideal motion (for example, "bring your knees close to your body", "legs level”, “stretch your back”, etc.).

(ii) The content of the voice specified by the processor 19A includes content indicating the type of exercise (training menu, etc.) that the first user should undertake for training of the first user.

If the second user's voice includes the type of exercise, the processor 19A sets the type of exercise to be imposed on the first user to be the exercise specified by the voice of the second user, and The auxiliary line drawn in association with the corresponding avatar is made to correspond to the designated exercise type. This makes it easier for the second user to evaluate the body movements of the first user.

(iii) The content of the voice specified by the processor 19A includes content that causes an application that draws the first user's avatar and auxiliary lines, which operates on the terminal device 10A, to perform a specific process. For example, in the application, commands indicated by the voice of the second user include the following.
・ Avatar drawing mode (including the position where the virtual camera is arranged with respect to the point cloud data representing the body movement of the first user. In other words, the avatar corresponding to the first user is drawn from the front or from the side) or draw from other directions, etc.)
・ How to draw the auxiliary line (as the auxiliary line, draw the auxiliary line parallel to the user's body movement, draw the auxiliary line along the shape of the user's body, whether to draw the auxiliary line etc.)
・ How to draw the avatar when there is a difference compared to the ideal movement (including drawing the parts of
・ Instructions to suspend the application and other contents

When the processor 19A recognizes that the speaker is a trainer by speech recognition (for example, if the speaker is a trainer registered in the trainer information database 2602), the processor 19A recognizes the first user based on the speech of the speaker. may control an application that draws the avatar of the This can prevent a user other than the trainer from unintentionally controlling the application. In addition, when communicating with another terminal device 10 such as the terminal device 10B by using a web conference tool or the like, the processor 19A, based on the information of the speaker specified by the web conference tool or the like, It is good also as controlling the application which draws an avatar. For example, in a web conferencing tool, etc., information (such as an icon) that identifies each user participating in the meeting is displayed, and the information that identifies each user is displayed separately for the user who is speaking. There is The processor 19A may perform the above processing assuming that the speaker is the trainer when the icon or the like corresponding to the trainer is displayed in a web conference tool or the like to be distinguished from the icons or the like of other users. good.

For example, when a plurality of trainees receive guidance from a trainer, depending on the trainee's voice, the application may perform processing related to training (processing to change the type of exercise to be performed by the trainee, processing to draw auxiliary lines with the avatar along with the change). change processing, etc.), and the trainer can be made to perform the processing. As a result, even if each user speaks during training, the training can be performed as intended by the trainer.

As described above, as a result of the second user giving voice guidance to the first user, the display mode of the video including the first avatar is changed, and based on the video including the changed avatar, the second user can One user can be instructed.

(Sensing the movement of the first user and identifying the instruction content based on the learned model)
FIG. 29 is a diagram showing the motion of the body of the first user sensed as point cloud data, and based on the information based on the point cloud data of the first user obtained by sensing and the learned model, the It is a flowchart which shows a series of processes which identify the content of instruction. As a result, even without a trainer, the trainee can check how the trainee moves his/her own body using the learned model.

In step S2901, the processor 19A of the terminal device 10A of the first user senses the movement of the first user's body as point cloud data using the measurement unit 30 or the like.

In step S2902, the processor 19A identifies guidance content for the first user based on the information based on the first user's point cloud data obtained by sensing and the learned model.

Here, a method for creating a learned model will be illustrated.
(i) For example, after acquiring information on point cloud data of one or more users, various information (eg, age, weight, purpose, etc.) and information of the user (trainer) who is in a position to instruct the user undergoing training Then, based on the content of the training actually received by the trainee and information such as the history of changes in the information of various trainees due to training, the information of the point cloud data of one or more users and the training content are associated. Create a trained model.

In other words, the trained model can be said to output how much the trainee changes as the trainee is trained by learning the information of the trainee and the content of instruction by the trainer as time-series data.

In addition, the trained model is associated with the information of the user who acquired the information based on the point cloud data, which is the learning data of the trained model (that is, the information of the trainee), or the information of the trainer who instructed the trainee. may have been Accordingly, when a trainee or a trainer designates a trained model to be used by the trainee, the trainee's information or the trainer's information associated with the trained model can be referred to. Alternatively, the processor 19A or the like may specify a trained model to be proposed to the user who is a trainee based on user information associated with the trained model.

(ii) Alternatively, in the trained model, the trainer attaches tags (body parts Generate the time-series data as learning data by commenting on how to move ideally, that you should breathe, that your heart rate is too high and you should take a break, etc.) You can do it.

For example, in one aspect, based on the result of measuring the body movement of the trainee and the learned model, a training menu is presented that matches the physical constitution of the user who is the trainee, the timing at which to breathe during training, and the heart rate. The timing to get up, the timing to drink water, the timing to take a break, etc. can be identified and presented to the user.

For example, in a certain aspect, a trainee wears a wearable device (wristwatch type, eyeglass type, ring type, etc.) and uses the above learned model while measuring the heart rate, breathing timing, etc. of the user who is a trainee. , the learned model may output the content to be instructed to the trainee.

In another aspect, the user's exercise intensity (exercise intensity) including calorie consumption is specified from information on point cloud data acquired from the user, and advice such as rest and hydration is presented to the user. be able to.
(iii) As a method for generating a trained model, as described in the first embodiment, the measured data (measurement history database 283) accumulated as a result of the ROM check is used as teacher data to generate a trained model. may be For example, a learned model may be generated based on the measurement data, the training menu performed by the user, and the results of responses to a questionnaire or the like input by the user.
As a result, the type of exercise proposed to the user, the content of advice to the user, and the like may be output to the learned model generated based on the measurement data of the ROM check.

As a method for generating a trained model, as described in the second embodiment, measured data accumulated as a result of evaluating the user's walking posture (measurement result database 287) is used as teacher data to generate a trained model. may be generated. For example, a learned model may be generated based on the measurement data, the training menu performed by the user, and the results of responses to a questionnaire or the like input by the user.
As a result, the type of exercise proposed to the user, the content of the advice to the user, and the like may be output by the trained model generated based on the measurement data for walking posture evaluation.

For example, as a method of generating a trained model, a trained model may be generated based on measurement data accumulated as a result of estimating a user's posture (joint positions, etc.). Specifically, as a method of generating a learned model in the server 20 or the like, a learned model is generated based on the result of estimating the user's joint positions, the training menu performed by the user, and the content of advice from the trainer. It is also possible to generate and store the generated learned model.

As described above, the server 20 may specify the instruction content for the user based on the information based on the estimation result of the user's posture and the learned model, and present the specified instruction content to the user.

Further, the server 20 or the like may store the learned model in association with the information of the user who acquired the learning data (result of estimating the user's posture) of the learned model.

Also, the server 20 and the like may identify the user's exercise intensity in the same manner as described above, based on the information based on the estimation result of the user's posture and the learned model. The server 20 may identify the guidance content for the user based on the result of identifying the exercise intensity of the user, and present the identified guidance content to the user.

In step S2903, the processor 19A presents the guidance content specified by the learned model to the first user by displaying it on the display 132A or the like.

<Screen example in the fifth embodiment>
30 to 32 are diagrams showing examples of screens displayed on the terminal device 10. FIG.

FIG. 30 shows that the terminal device 10A of the first user senses the movement of the body of the first user as point cloud data, and transmits information obtained by sensing based on the point cloud data of the first user to the terminal device 10A of the second user. It is an example of the screen when sending to.

In FIG. 30, the camera 2901 of the measurement unit 30 shows a device for sensing the body movements of the first user.

Specifically, the camera 2901 captures an image of the first user through a capturing unit provided in the camera 2901 . At this time, the camera 2901 is configured to be able to measure distance and direction, and is configured by, for example, an RGB-D camera and LIDAR (Laser Imaging Detection and Ranging) technology.

The previous screen 2902 is a screen that displays an image including the first user's avatar (that is, an image drawn based on point cloud data) when the first user performed training of the same kind before. be. The terminal device 10A acquires point cloud data representing the body movements of the first user when the first user received training in the past, for example, by referring to the training history database 2603 of the server 20 .

In the illustrated example, on the previous screen 2902, the first user's avatar is shown as an aggregate of figures such as circles, but it may be displayed as an aggregate of point cloud data measured by the measurement unit 30. .

To allow a first user and a second user to easily confirm how much the body movement of the first user has changed due to training by displaying an avatar based on point cloud data of the first user's past training. can be done. For example, when the first user holds the knees and performs a motion of curling the body, the terminal device 10A displays the previous screen 2902, which is the previous training screen. The first user or the second user can compare the previous screen 2902 with the display contents of the current screen 2903, which will be described later. Thereby, the first user can confirm the difference in the movement of his/her own body.

Further, the terminal device 10A may always display the previous screen 2902, or may display or turn off the display according to an instruction from the first user or the second user. At this time, the terminal device 10A may display, as an avatar representing the first user, images of the first user drawn from a plurality of viewpoints such as the front, the side, and the top. In addition, an auxiliary line may be drawn at a position based on a body part for evaluating whether or not the body movement performed by the first user is ideal. In the illustrated example, on the past screen 2902, an auxiliary line is drawn in association with the area where the first user's avatar is displayed.

The current time screen 2903 is a screen that displays an image of the first user's avatar based on the sensing result of the measurement unit 30 while the first user is training.

In the illustrated example, a first user 2094 is shown, showing a state in which the first user is actually training on a mat or the like. The terminal device 10A captures the movement of the first user with the camera 2901, and acquires the point cloud data information of the first user.

A first user image 2905 indicates a captured image of the first user captured by the camera 160 or the like. In the first user image 2905, an image captured by a 2D camera or the like is displayed instead of rendering the first user as an aggregate of point cloud data. That is, in the first user image 2905, a photographed image equivalent to that which can be visually recognized by the human eye is displayed.

Thereby, the terminal device 10A allows the first user to visually compare the actual image of the first user himself and the image based on the information of the point cloud data. In addition, the terminal device 10 does not display the photographed image of the first user himself/herself in the first user image 2905 according to the designation of the first user, and the second user or the like can identify the first user. icons, character strings, etc. may be displayed.

A second user image 2906 shows an image of a second user who is in a position to guide a user undergoing training. The terminal device 10A displays the second user's captured image in the second user image 2906 by receiving the second user's captured image from the second user's terminal device 10B. Thereby, the first user and the second user can perform training while looking at each other's images and communicating with each other through a terminal-to-terminal video call or the like through the network 80 or the like.

FIG. 31 shows the overall configuration when sensing the movement of the first user's body as point cloud data and transmitting information based on the first user's point cloud data obtained by sensing to the terminal of the second user. It is an example of the screen of the 1st user and the 2nd user including.

In FIG. 31, a camera 3001 is a device for sensing the motion of the first user's body as point cloud data, similar to the camera 2901 in FIG.

This time images 3002 and 3006 are screens that display images of the first user as an aggregate of point cloud data when the first user is training, similarly to the current time screen 2903 in FIG. At this time, the current image 3002 is displayed on the display 132A or the like of the first user's terminal device 10A, and the current time image 3006 is displayed on the display 132B or the like of the second user's terminal device 10B.

A first user 3003 is a diagram showing a state in which the first user is actually training on a mat or the like, similar to the first user 2904 in FIG.

First user images 3004 and 3007 show actual captured images, not videos containing avatars, when the first user is captured by camera 2901, similarly to first user image 2905 in FIG. At this time, the first user image 3004 is displayed on the display 132A or the like of the first user's terminal device 10A, and the first user image 3007 is displayed on the display 132B or the like of the second user's terminal device 10B. .

Second user images 3005 and 3008 show images of a second user who is in a position to guide a user undergoing training, similar to second user image 2906 in FIG. At this time, the second user image 3005 is displayed on the display 132A or the like of the first user's terminal device 10A, and the second user image 3008 is displayed on the display 132B or the like of the second user's terminal device 10B. . Further, in a certain aspect, as in the screen example, an input operation by voice from the second user causes the second user's voice (such as "Let's sit down more!") through the speaker 142A or the like of the terminal device 10A. ) may be output.

The second user 3009 visually recognizes the screen displayed on the display 132B or the like by the terminal device 10B, and gives guidance based on the image of the first user. At this time, the second user can give guidance to the first user by voice. The second user confirms both or one of an image based on the point cloud data and an image equivalent to that seen with the naked eye as an image of the first user, and responds to the movement of the body of the first user. can give advice. As a collection of point cloud data, the amount of information for the second user is reduced from the image of the first user shown by the actual photographed image, so that the second user can more easily identify the movement of the first user's body. and facilitate evaluation and advice.

FIG. 32 is an example of a screen when the body movements of a plurality of users are sensed as point cloud data, and information based on the point cloud data of each user obtained by sensing is transmitted to a defined virtual space. .

In FIG. 32, virtual space 3101 represents a space for one or more trained users to place objects for displaying videos, including avatars associated with them.

A trainer image 3102 shows an image of a second user who is in a position to guide a user undergoing training. The terminal device 10B of the second user takes a picture of the second user and transmits the taken image to the computer managing the virtual space as needed.

A user image 3103 is a screen that displays an image of each user when one or more users are training. At this time, the image to be displayed may be only a video containing an avatar as a collection of point cloud data associated with the user, or an actual photographed image of each user, which is not a video containing an avatar, may be displayed together. good too.

As a result, each user can place an image including an avatar corresponding to the movement of the user's own body in the virtual space, and each user can visually recognize the image of the avatar of each user placed in the virtual space. By doing so, it is possible to get the feeling that a plurality of users are training at the same place. As a result, a sense of unity and a sense of accomplishment in training can be obtained, and motivation to continue training can be provided to each user. In response to matching each trainee who receives training from the trainer, the server 20 enables communication between each trainee's terminal and the trainer's terminal, and builds a virtual space in either the server 20 or each user's terminal. It is also possible to That is, the server 20 may designate a computer that serves as a host when each user communicates.

<6 Modifications>
A modification of this embodiment will be described. That is, the following aspects may be adopted.

(1) In the examples of FIGS. 31 and 32 above, an avatar based on the user's sensing results when the user underwent training in the past and an avatar based on the current sensing results of training from a trainer. An example of displaying them side by side was explained.

Alternatively, avatars corresponding to trainees may be displayed side by side based on the time-series data of the trainee's point cloud data sensed along the time series during training with the trainer.

In other words, the avatar based on the point cloud data sensed at the timing of starting training or within a certain period of time after the start, and the result of real-time sensing based on the trainer's guidance or a certain amount of time after the trainer's guidance started The terminal device 10A may display the avatars based on the sensing results at the timings set side by side.

Also, in the course of one training session, the part of the trainee's body that has moved closer to the ideal one is emphasized compared to other parts and displayed on the avatar. It is also possible to For example, the processor 19A may evaluate how closely the body movements of the trainee are to the auxiliary lines drawn together with the avatar, and draw the screen according to the evaluation result. For example, if the ideal motion is to stretch the knee in a specific direction, the processor 19A draws an auxiliary line along the specific direction. The processor 19A changes the knee part of the avatar or the drawing mode of the auxiliary line associated with the knee part as the movement of the knee approaches the direction along the auxiliary line in the trainee's body movement. good too.

As a result, the trainee can easily check how much the trainee has changed before and after a single training session or during the course of training, so the trainee himself can check how much he has changed (such as whether he is getting closer to his goal). easier to do. This can further improve the trainee's motivation to continue training.

When training online, the visual information is reduced compared to training face-to-face instead of online (visible to the naked eye in face-to-face vs. two-dimensional display image for online). (visually recognized as such) may reduce the quality of the communication experience. Moreover, when training is conducted online, it is very difficult for the trainer to touch the body of the trainee. Therefore, by providing training result feedback to the trainee as described above, it is possible to provide the trainee with motivation to continue training online.

As described above, a learned model may be generated based on the sensing result when the user's body movement is changed to an ideal one by one training session.

(2) In the above embodiments, the avatar is displayed based on the result of sensing the movement of the user's body, and the avatar corresponding to the trainer is displayed to the trainee in online training between the trainer and the trainee.

In addition, the processor 19A may perform training while confirming how the user himself or herself moves as the avatar described above, regardless of the communication between the trainer and the trainee. For example, in the terminal device 10A, the user sets how to move the body, draws an auxiliary line or the like according to the set way to move the body, and notifies the user of the information on how to move the body ( The user's body movement may also be sensed by the measurement unit 30 or the like.

(2-1) Method for Sensing Movement of User's Body The terminal device 10A presents the result of sensing the movement of the user's body to the user as an avatar or the like. Further, as described in the first and second embodiments, the terminal device 10A identifies the positions of the joints of the user's body based on the sensing results of the user's body, and estimates the skeleton (bone). (Fig. 12).

The terminal device 10A may estimate the posture based on the position of each joint of the user and the result of estimating the skeleton in the following manner.
- The terminal device 10A estimates the posture of the user shown in the moving image based on the photographed result (two-dimensional moving image) of the user, who is the subject, without using the point cloud data.
That is, the terminal device 10A shoots the user with a camera that does not have a configuration for optically measuring the distance to the subject, such as LiDAR, and based on the shooting result, the user's posture in the shot image is: It is also possible to estimate joint positions, skeletons, and the like.
For example, by using a captured image as an input and preparing a trained model that outputs the positions of joints and postures such as the skeleton of a person who is the subject in the captured image, the terminal device 10A can obtain the user's captured result, The user's pose can be estimated based on the trained model.
Such a trained model is generated by preparing learning data in which information such as joint positions and bone structure in a photographed image is associated with a person who is a subject in the photographed image, and causing a computer to perform the learning. can.
- As described in other embodiments, the terminal device 10A senses the movement of the user's body as point cloud data based on the camera or the measurement unit 30 mounted on the terminal device 10A or the like, and the sensing result is Based on this, the positions of the joints, the skeleton, etc. are estimated.

(2-2) Drawing of Auxiliary Lines Based on Joint Position and Skeleton Estimation Result The terminal device 10A draws an auxiliary line for the user as a guideline for moving the body, based on the result of estimating the user's posture. It is also possible to The terminal device 10A may draw the auxiliary line based on the estimation result of the user's posture as follows.
The type of body movement presented to the user by the terminal device 10A or the like (for example, exercise for evaluating movement of trunk muscles and range of motion of joints such as side elevation, or exercise for strength training) motion, etc.), indirect positions, and skeleton estimation results, the positions for drawing auxiliary lines are determined.
For example, some types of body movements include those in which the user moves the body by a certain amount (for example, an action of stepping a front foot to evaluate the joint range of motion). While displaying the avatar to the user, the terminal device 10A draws an auxiliary line as a guideline for the amount of body movement based on the results of estimating the positions of specific joints of the user.
For example, when the user steps forward, the terminal device 10A first prompts the user to take a predetermined posture (for example, to stand upright before stepping forward) by voice or video. The terminal device 10A estimates the amount of stride of the user based on the positions of the joints in the posture (for example, shoulder joint, hip joint, knee joint) (for example, the height of the user, the length of the lower limbs of the user, The amount of stride to be taken can be specified), and based on the estimated amount of stride, determine the position where the auxiliary line should be drawn based on the joint position estimation result.
The terminal device 10A associates in advance the position of the joint referred to for drawing the auxiliary line and the positional relationship for drawing the auxiliary line with respect to the position of the joint, according to the type of body movement performed by the user. It is also possible to store the

FIG. 33 is a diagram showing a concept of processing in which the terminal device 10A draws auxiliary lines based on the result of estimating the posture of the user.

In the example (A) of FIG. 33, the terminal device 10A senses the movement of the user's body as point cloud data. The terminal device 10A stores the three-dimensional object data of the avatar AVT_A corresponding to the user based on the sensing result. Based on the 3D object data of the avatar AVT_A, the terminal device 10A estimates the positions of the user's joints (for example, each joint shown in FIG. 12) of the 3D object data. In example (A), when the three-dimensional object data of avatar AVT_A is arranged in the virtual space, terminal device 10A identifies joint position JNT_01A, which is the coordinate resulting from estimating the position of the user's hip joint in the avatar.
In the example (A) of FIG. 33, the terminal device 10A moves the three-dimensional object data of the auxiliary line LN_01A to a position separated by a certain distance DIST_01A from the joint position JNT_01A according to the type of movement of the user's body. Decide to place. The terminal device 10A may set the fixed distance based on the physique of the user (obtained by measuring the user's height, sitting height, shoulder width, etc. based on the result of sensing the movement of the user's body). . Depending on the physique of the user, the amount that the user should move the body can be determined. In example (A), the terminal device 10A arranges the three-dimensional object data of the auxiliary line LN_01A so as to be parallel to the ground when the user moves his/her body.

In the example (B) of FIG. 33 , the terminal device 10A arranges the three-dimensional object data of the auxiliary line LN_02A based on the joint position JNT_01A so that it overlaps with the joint position JNT_01A and is aligned with the ground surface when the user moves the body. are arranged so as to be perpendicular to the

FIG. 34 shows an example screen presented to the user.
As shown in FIG. 34, the terminal device 10A, as a result of sensing the movement of the user's body on the screen 3403, displays the avatar corresponding to the user in a plurality of positions such as the left side 3403A, the front 3403B, and the right side 3403C. Place the result drawn from the viewpoint on the same screen.

For example, the terminal device 10A arranges three-dimensional object data (that can be recognized by the user as an avatar) corresponding to the sensing result of the motion of the user's body (for example, point cloud data) in the virtual space. In the virtual space, the terminal device 10A captures three-dimensional object data corresponding to the sensing result of the user by using a plurality of virtual cameras (for example, a virtual camera corresponding to the right side, a virtual camera corresponding to the front, and a virtual camera corresponding to the left side). corresponding virtual camera). Thereby, the terminal device 10A can present to the user the result of drawing the avatar corresponding to the user from a plurality of viewpoints.

As illustrated, the terminal device 10A may present the avatar to the user, but may not present the result of estimating the skeleton (bone). For example, compared to visually recognizing oneself in a mirror or the like, it is assumed that the user has fewer opportunities to visually perceive the estimation result of the skeleton (bone) itself with the naked eye on a daily basis. Therefore, the terminal device 10A presents the avatar to the user and does not display the skeleton (bone) estimation result, so that the user can see on the screen an avatar similar to his/her own body shape that can be seen on a daily basis with the naked eye. Awareness and avatars projected onto the screen can be immersive as if the user were themselves.

The terminal device 10A arranges the three-dimensional object data of auxiliary lines LN_01A and LN_02A in association with the three-dimensional object data corresponding to the user in the virtual space. Therefore, the terminal device 10A can display the auxiliary line together with the avatar corresponding to the user by performing rendering according to the setting of the virtual camera.

Specifically, the terminal device 10A draws an auxiliary line LN_01A and an auxiliary line LN_02A in association with the avatar AVT_A on the left side 3403A and displays them to the user. The terminal device 10A draws an auxiliary line LN_01B and an auxiliary line LN_02B in association with the avatar AVT_B on the front 3403B and displays them to the user. The terminal device 10A draws an auxiliary line LN_01C and an auxiliary line LN_02C in association with the avatar AVT_C on the right side 3403C and displays them to the user.

(2-3) Method of Evaluating User's Body Movement (i) As described above, the terminal device 10A displays the avatar to the user without displaying the skeleton (bone) estimation result. By making them realize that they are a projection of themselves, they are immersed in moving their bodies and training. On the other hand, the terminal device 10A evaluates the movement of the user's body in order to evaluate whether the user's posture is desirable and whether the movement of the user's body is ideal according to the type of body movement. , an estimation result of the user's skeleton (bone) may be used.

For example, in the example of FIG. 34, when it is desirable that the user's knees are vertical, the terminal device 10A determines (the knees are vertical ), it can be determined whether the user's posture is desirable.
(ii) The terminal device 10A evaluates the movement of the user's body, and corresponds to three-dimensional object data (for example, point cloud data) forming an avatar, which is the result of sensing the movement of the user's body, and auxiliary lines. It may be based on three-dimensional object data to be used.

For example, in the terminal device 10A, the three-dimensional object data forming the user's avatar (for example, the avatar AVT_A on the left side 3403A) collides with the three-dimensional object data corresponding to the auxiliary line (for example, the auxiliary line LN_01A on the left side 3403A). Whether or not the user's body movement is ideal may be evaluated by determining that the user's body movement is ideal, or determining that the user has approached within a certain distance.

When the terminal device 10A senses the movement of the user's body without relying on the point cloud data (for example, when capturing the user as a two-dimensional image), based on the sensing result (two-dimensional captured image data), Three-dimensional object data corresponding to the user's body may be generated. The terminal device 10A evaluates whether the movement of the user's body is ideal or not, based on whether the three-dimensional object data and the three-dimensional object data corresponding to the auxiliary line collide or come close to a certain distance. It is also possible to

When the terminal device 10A senses the movement of the user's body without relying on the point cloud data (when capturing the user as a two-dimensional image), the user's posture (joint joints) is captured in the two-dimensional image as described above. Position, skeleton) may be estimated (using a trained model for estimating posture as described above), and based on the estimated result, auxiliary lines may be drawn as object data in the two-dimensional image. For example, the terminal device 10A, from the position of the user's joints (hip joint, shoulder joint, etc.) estimated in the two-dimensional image, corresponding to the type of exercise performed by the user, in the two-dimensional image, as object data Determines where to draw extension lines. The terminal device 10A determines whether the user's body movement is ideal based on whether the two-dimensional image corresponding to the user and the auxiliary line drawn as the object data collided or approached a certain distance. may be evaluated.

(2-4) Screen example presented to the user to measure specific body movements (a) Standing Alignment
By enabling the user to adjust so that the user's body does not tilt forward, backward, left, or right while the user is in a specific posture, such as a standing position, the user's posture can be maintained well.

FIG. 35 is a diagram showing a screen example for the terminal device 10A to adjust the tilt in the front, back, left, and right while the user is standing.

FIG. 36 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guideline for the movement of the user's body, based on the result of estimating the position of a specific part of the user.

As shown in FIG. 35, the terminal device 10A draws an avatar corresponding to the user from a plurality of viewpoints such as the left side 3501A and the front 3501B as a result of sensing the movement of the user's body on the screen 3500. Place the results on the same screen. The terminal device 10A displays the positions of a plurality of parts of the user's body on the screen 3501C such that the positional relationships on the horizontal plane can be compared.

The terminal device 10A displays an avatar AVT_35A corresponding to the user (corresponding to when the user is viewed from the left side) on the left side 3501A. The terminal device 10A displays the auxiliary line VL_35A as a graphic object together with the avatar AVT_35A.

Here, a process of drawing the auxiliary line VL_35A by the terminal device 10A will be described.
As shown in example (A) of FIG. 36, the terminal device 10A estimates the position JNT_35D of the ankle (heel) as a specific part of the user's body.

The terminal device 10A can estimate the positions of the right ankle, right foot, right toe, left ankle, left foot, and left toe based on point cloud data obtained by measuring the user's body (for example, the example in FIG. 12). For example, based on the estimated result of the position of the right ankle or the left ankle, the terminal device 10A creates an auxiliary line VL_35A as a graphic object so that it passes through the estimated result of the position of the ankle and is perpendicular to the ground. The drawing position may be specified.

As a result, the terminal device 10A instructs the user to update the display of the avatar AVT_35A each time it acquires the sensing result of the movement of the user's body (that is, based on the point cloud data obtained by measuring the user's body in real time). While drawing the avatar AVT_35A), the positional relationship with the auxiliary line VL_35A can be recognized. The user can recognize by comparing the avatar AVT_35A and the auxiliary line VL_35A whether or not the body is out of alignment in the front, back, left, and right directions. For example, when the avatar AVT_35A protrudes significantly from the position of the auxiliary line VL_35A, it can be said that the user's body is largely displaced forward, backward, leftward, or rightward.

Note that the terminal device 10A draws the auxiliary line VL_35A so that the user can compare it with the avatar AVT_35A viewed from the left side as described above. Drawing may be performed based on the estimated results (for example, based on the results of estimating the position of the pelvis).

As shown in the front 3501B of FIG. 35, the terminal device 10A displays an avatar AVT_35B corresponding to the user (corresponding to when the user is viewed from the front). The terminal device 10A displays an auxiliary line LN_35A, an auxiliary line LN_35B, an auxiliary line LN_35C, and an auxiliary line VL_35B as graphic objects in association with the avatar AVT_35B.

Here, the process of drawing the auxiliary lines LN_35A and the like on the front surface 3501B by the terminal device 10A will be described.

As shown in the example (B) of FIG. 36, the terminal device 10A designates a head position JNT_35A (“head” in the example of FIG. 12), a shoulder position JNT_35B (“head” in the example of FIG. 12), and a shoulder position JNT_35B (“head” in the example of FIG. "shoulder center" in the example), waist position JNT_35C ("pelvis center", "right pelvis", "left pelvis", etc. in the example of FIG. 12) are estimated. The terminal device 10A draws the auxiliary line VL_35B so that it passes through at least one of the head position JNT_35A, the shoulder position JNT_35B, and the waist position JNT_35C and becomes a line perpendicular to the ground. Identify. Here, the terminal device 10A sets the auxiliary line VL_35B based on the position of a specific part other than these (for example, it passes through the midpoint between the position of the right heel and the position of the left heel and becomes perpendicular to the ground). line), and the drawing position may be specified.

The terminal device 10A specifies the positions where the auxiliary lines LN_35A and the like are to be drawn based on the estimation results of the positions of these specific parts. For example, the terminal device 10A draws the auxiliary line LN_35A in the horizontal direction (when the direction toward the ground is vertical) from the head position JNT_35A, which is the estimated result of the position of the user's head. It is specified that the positions are separated by a distance (distance DIST_35A in the example of FIG. 36). The terminal device 10A specifies that the auxiliary line LN_35B should be drawn at a position horizontally separated from the shoulder position JNT_35B by a predetermined distance. The terminal device 10A specifies that the auxiliary line LN_35C should be drawn at a position horizontally separated from the waist position JNT_35C by a predetermined distance.

The terminal device 10A draws these auxiliary lines LN_35A, LN_35B, and LN_35C in different modes (for example, different colors, different shapes, etc.). This allows the user to easily recognize which part of the user's body each drawing mode corresponds to. Here, the terminal device 10A draws the auxiliary circles RD_35A, RD_35B, and RD_35C on the screen 3501C according to the respective drawing modes of these auxiliary lines LN_35A, LN_35B, and LNN_35C.

The terminal device 10A draws an auxiliary circle RD_35D, which is a graphic object based on the position of the user's heel, on the screen 3501C. Example (C) of FIG. 36 shows a mode in which auxiliary circles RD_35A, RD_35B, and RD_35C are not drawn on the screen 3501C of FIG. The terminal device 10A draws an auxiliary circle RD_35A corresponding to the position of the user's head, an auxiliary circle RD_35B corresponding to the shoulders, and an auxiliary circle RD_35C corresponding to the waist on the screen 3501C. These auxiliary circles RD_35A, RD_35B, RD_35C and the auxiliary circle RD_35D may have the same shape or similar shapes. The terminal device 10A displays these auxiliary circles RD_35A, RD_35B, and RD_35C at the relative positions of the head, shoulders, and hips to be compared with the positions of the reference specific parts on the horizontal plane. As a result, the user can recognize whether the body is tilted forward, backward, left, or right, and can use the plurality of graphic objects (auxiliary circles RD_35A, RD_35B, and RD_35C) corresponding to the plurality of body parts as reference assistance circles. You will be prompted to match the circle RD_35D.

As described above, the terminal device 10A sequentially senses the movement of the user's body as point cloud data or the like, and based on the results of sensing the movement of the user's body, detects at least three specific parts of the user (the above 35 and 36, the positions of the head, shoulders, hips, and heels are estimated). The terminal device 10A uses any one of at least three specific parts as a reference (in the example of FIGS. 35 and 36, the position of the heel is used as a reference, but it may be any part along the body axis such as the waist). ), determine the position to draw a graphic object indicating the positional relationship of other specific parts on the horizontal plane, and display the graphic object at the determined position.

The terminal device 10A displays, in the object 3501D in FIG. 35, an indication of the remaining time of the period in which the user performs an action to adjust the balance between the front, rear, left, and right in the standing position. As a result, the user is encouraged to adjust the front-back, left-right balance as much as possible within the remaining time, and the user's posture can be further improved.

(b) Lateral lift Evaluate the good posture of the user by measuring the motion of lifting both arms while the user is in a standing position without extending the arms out in front of the sides of the body. can do. In the side lifting motion, the timing at which the user raises both arms in a standing position will be described as the peak time of the motion.

FIG. 37 is a diagram showing an example of a screen on which the terminal device 10A causes the user to perform a side-lifting motion.

FIG. 38 is a diagram showing a process of specifying a position for drawing a graphic object that serves as a guideline for the movement of the user's body, based on the result of estimating the position of a specific part of the user.
FIG. 39 is a diagram showing a process in which the user is moving his/her body.
FIG. 40 is a diagram showing a state in which the user raises both arms at the peak timing of the lateral lifting motion.
FIG. 41 is a diagram showing an example in which the user saves the sensing result at the peak timing of the side lifting motion and displays it as a comparable avatar.

As shown in FIG. 37, the user assumes an initial posture by standing with both arms on the sides of the body before starting the side lift motion. The terminal device 10A may notify the user by voice, image, or the like to take the initial posture.

The terminal device 10A draws a graphic object based on the user's initial posture and the result of estimating the position of the specific part. In the illustrated example, the terminal device 10A displays an avatar AVT_37A corresponding to the user on the left side 3701A of the screen 3700 (corresponding to when viewed from the user's left side). The terminal device 10A draws the following graphic objects on the left side 3701A.

- Auxiliary line VL_37A, which is a graphic object for making it easier for the user to perceive the front-rear deviation. It is fixedly placed without moving while the user is moving his or her body.
• An auxiliary line VL_37D indicating the position of the surface of the user's head in the initial posture. While the user is doing the side lifting motion, the user's body moves back and forth by making the user perceive the positional relationship between the avatar AVT_37A, which is the sensing result of the movement of the user's body, and the auxiliary line VL_37D. It is possible to make it even easier to recognize by focusing on the head. It is fixedly placed without moving while the user is moving his or her body.
- Auxiliary line VL_37E indicating the position of the surface of the user's abdomen in the initial posture. While the user is doing the side lifting motion, the user's body moves back and forth by making the user perceive the positional relationship between the avatar AVT_37A, which is the sensing result of the movement of the user's body, and the auxiliary line VL_37E. It is possible to make it even easier to recognize by focusing on the abdomen. It is fixedly placed without moving while the user is moving his or her body.
- An auxiliary line HL_37A indicating the length between the auxiliary line VL_37A and the auxiliary line VL_37D. It is fixedly placed without moving while the user is moving his or her body.
- An auxiliary line HL_37B for facilitating recognition of the position of the user's shoulders in the initial posture. It is fixedly placed without moving while the user is moving his or her body.
- An auxiliary line HL_37C indicating the length between the auxiliary line VL_37A and the auxiliary line VL_37E. It is fixedly placed without moving while the user is moving his or her body.

The terminal device 10A displays auxiliary lines VL_37C, VL_37F, VL_37G, HL_37D, HL_37E, and HL_37F along with the avatar AVT_37C (corresponding to when the user is viewed from the right side) on the right side 3701C.
The auxiliary line VL_37C corresponds to the auxiliary line VL_37A.
The auxiliary line VL_37F corresponds to the auxiliary line VL_37D.
The auxiliary line VL_37G corresponds to the auxiliary line VL_37E.
The auxiliary line HL_37D corresponds to the auxiliary line HL_37A.
The auxiliary line HL_37E corresponds to the auxiliary line HL_37B.
The auxiliary line HL_37F corresponds to the auxiliary line HL_37C.

The terminal device 10A displays an avatar AVT_37B corresponding to the user (corresponding to when the user is viewed from the front) on the front 3701B. The terminal device 10A displays the following graphical objects on the front 3701B.
- An auxiliary line VL_37B, which is a graphic object for making it easier for the user to perceive a left-right shift. It is fixedly placed without moving while the user is moving his or her body.
- In association with the avatar AVT_37B, an auxiliary line LN_37L (corresponding to the user's left arm) and an auxiliary line LN_37R (the corresponding to the right arm). Each of the terminal devices 10A updates the drawing position of the graphic object in conjunction with the sensing result of the movement of the user's body. That is, the terminal device 10A updates the drawing positions of the auxiliary lines LN_37L and LN_37R in conjunction with the movement of the avatar.
A polygon QB_37A (corresponding to the left arm) indicating the end point of the auxiliary line LN_37L, and a polygon QB_37B (corresponding to the right arm) indicating the end point of the auxiliary line LN_37R. If the user bends his/her elbow while performing the lateral behavior, these polygons QB_37B or QB_37A will be drawn at a position deviated from the avatar AVT_37B. Alternatively, by recognizing the positional relationship with QB_37B, bending of the elbow can be easily recognized.
- Polygon QB_37C (corresponding to the left arm), multi Square QB_37D (corresponding to the right arm). As will be described later, when the user raises both arms without shrugging, the terminal device 10A adjusts the right arm so that the polygon QB_37C and the polygon QB_37A overlap with each other. Polygons QB_37C and QB_37D are drawn at positions where polygons QB_37D and QB_37B overlap.

When the terminal device 10A prompts the user to perform the side elevation motion, the icon 3701D indicating the initial posture, the icon 3701E indicating that the side elevation motion is in progress, and the end of the measurement are displayed. An icon 3701F indicating that it has been performed is displayed.

Processing for drawing these graphic objects will be described.
As shown in the example (A) of FIG. 38, the terminal device 10A selects, as specific parts of the user's body, the waist position JNT_38B (such as "pelvic center" in the example of FIG. 12), the position between the waist and the shoulders. JNT_38C (such as “chest and waist” in the example of FIG. 12), shoulder position JNT_37D, and head position JNT_38E are estimated.

The terminal device 10A identifies a position to draw the auxiliary line VL_37A so that it passes through the waist position JNT_38B and the like and becomes perpendicular to the ground.

The terminal device 10A specifies the drawing position of the auxiliary line VL_37D based on the position of the body surface when the auxiliary line VL_37D passes through the head position JNT_38E and extends horizontally from the rear to the front of the user. That is, the terminal device 10A identifies the position where the graphic object is to be drawn based on the result of estimating the position of the body part and the sensing result of the movement of the user's body sensed as point cloud data.

With respect to the auxiliary line VL_37E, the terminal device 10A passes through an intermediate position between the position JNT_38B of the waist and the position JNT_38C between the waist and the shoulder, and extends horizontally from the rear to the front of the user. Identify the position to draw based on the position.
The method of drawing each auxiliary line shown in example (C) may be the same as in example (A).

As shown in example (B), the terminal device 10A draws the auxiliary line VL_37B based on the results of estimating the positions of, for example, the waist and pelvis as parts of the user's body. Here, based on the result of estimating the position of the heel of the user's right foot and the position of the heel of the left foot, the terminal device 10A causes the auxiliary line VL_37B to pass through these intermediate positions and become perpendicular to the ground. It may be drawn as follows.

Based on the result of estimating the user's wrist position JNT_38F (left wrist) and the user's elbow position JNT_38G (left elbow), the terminal device 10A sets the auxiliary line LN_37L from the left wrist as a starting point, and The drawing position is specified so that the distance from the point to the elbow is set as the end point (polygon QB_37A) that is extended from the elbow.

The terminal device 10A determines the auxiliary line LN_37R as the auxiliary line LN_37L based on the results of estimating the user's wrist position JNT_38I (right wrist) and the user's elbow position JNT_38J (right elbow). Specify the length to specify the position to draw.

Based on the shoulder position JNT_38H (left shoulder), the terminal device 10A identifies the coordinates outside and above the user's body by a predetermined distance. The coordinates are set to correspond to the position of the shoulder to which the user would move when the peak of the lateral lift is reached. The terminal device 10A may identify the coordinates based on the size of the user's physique (for example, the user's height, shoulder width, etc.). The terminal device 10A identifies the position to draw the polygon QB_37C based on the identified coordinate values.

The terminal device 10A identifies the position for drawing the polygon QB_37D based on the user's shoulder position JNT_38K (right shoulder) in the same manner as the method for identifying the position for drawing the polygon QB_37C.

As shown in FIG. 39 , when the user raises his/her arm, the terminal device 10A detects the trajectory of the hand movement based on the history of sensing results of the user's body movement and the history of specifying the wrist position. are displayed as a trajectory LN_39A (left hand motion trajectory) and a trajectory LN_39C (right hand motion trajectory).

As shown in FIG. 40, it is assumed that the user reaches the peak by raising both arms. The terminal device 10A detects that the user's lateral lifting motion has reached a peak (for example, detects that the user has raised both arms based on the estimation result of the position of a specific part of the user). ), and draws an avatar corresponding to the sensing result (posture of the user) at that timing. Here, when the polygon QB_37A whose drawing is updated in conjunction with the movement of the user's body and the polygon QB_37C which serves as a guideline for the movement when both arms are fully raised have a specific positional relationship (for example, when they overlap), at least one of these polygons QB_37A or QB_37C may be displayed in a different manner than when they do not overlap. As a result, the terminal device 10A can allow the user to easily perceive that the lateral lifting motion is appropriate (shoulders are not shrugged, etc.). Similarly, the terminal device 10A may change the display mode when the polygon QB_37B and the polygon QB_37D are in a specific positional relationship, compared to when they are not.

As shown in FIG. 41, the terminal device 10A displays a history avatar AVT_37D, which is the state of the avatar at the peak, so that the avatars AVT_37A and AVT_37C whose drawing is updated in conjunction with the sensing result of the movement of the user's body are aligned. , to display the history avatar AVT_37E.

As shown in FIG. 41 , the avatar in a state in which both arms are fully raised, the trajectory of the process of the user raising both arms (trajectories LN_39A and LN_39C), and the user's Since the avatars AVT_37A and AVT_37C, which are real-time postures, are displayed side by side, it is possible to provide the user with material for considering how to perform the action.

(c) Crouching FIG. 42 is a diagram showing a screen example for the terminal device 10A to cause the user to crouch.
FIG. 43 is a diagram showing a process of specifying a position for drawing a graphic object that serves as a guideline for the movement of the user's body, based on the result of estimating the position of a specific part of the user.
FIG. 44 is a diagram showing a process in which the user is moving his/her body.
FIG. 45 is a diagram showing a fully crouched state, which is the peak timing of the user's crouching motion.
FIG. 46 is a diagram showing an example in which the user saves the sensing result at the peak timing of the crouching motion and displays it as a comparable avatar.

As shown in FIG. 42, in the initial posture before starting the crouching motion, the terminal device 10A draws a graphic object based on the initial posture and the results of estimating the positions of specific parts.

In the illustrated example, the terminal device 10A draws the following auxiliary lines on the left side 4201A of the screen 4200 while drawing the avatar AVT_42A.
- Auxiliary line LN_42A for visualizing the angle of the user's back
- Auxiliary line LN_42B for visualizing the user's shin angle

The terminal device 10A draws the following auxiliary lines on the right side 4201C while drawing the avatar AVT_42C.
- Auxiliary line VL_42C for making the user perceive whether the posture is tilted forward or backward. It is displayed in a fixed position while measuring the movement of the user's body.

The terminal device 10A draws the following auxiliary lines while drawing the avatar AVT_42B on the front 4201B.
- Auxiliary line VL_42B for letting the user perceive whether or not the user's posture is tilted to the left or right. It is displayed in a fixed position while measuring the movement of the user's body.
- Auxiliary line LN_37L for making the user perceive whether or not the left leg of the user is open to the left or right
- Auxiliary line LN_37R for making the user perceive whether or not the right leg of the user is open to the left or right

The terminal device 10A displays an object 4201D indicating that the state is before the crouching motion is started, an object 4201E that indicates that the crouching motion is being measured, and an object 4201F that indicates that the measurement is finished.

As shown in the example (A) of FIG. 43 , the terminal device 10A, as a result of estimating the positions of the user's body parts, uses the heel position JNT_43A as the starting point based on the heel position JNT_43A and the knee position JNT_43B. A line segment passing through the knee position JNT_43B is drawn as an auxiliary line LN_42B.

Based on the waist (pelvis, etc.) position JNT_43C and the shoulder position JNT_43D, the terminal device 10A draws a line segment starting from the waist position JNT_43C and passing through the shoulder position JNT_43D as an auxiliary line LN_42A.
Here, the terminal device 10A may determine the lengths of the auxiliary lines LN_42A and LN_42B so that the end points of the auxiliary lines LN_42A and LN_42B have the same height.

As shown in example (C), the terminal device 10A specifies the position to draw the auxiliary line VL_42C so as to be perpendicular to the ground based on the heel position JNT_43J. Here, the terminal device 10A may draw the auxiliary line VL_42C based on the waist position or the like, in addition to the heel position JNT_43J.

As shown in example (B), the terminal device 10A uses the heel position as the starting point based on the heel position JNT_43E (left heel) and the knee position JNT_43F (left knee), and measures the distance from the heel to the knee. is measured, and the position where the auxiliary line LN_37L is to be drawn is specified with a point further extended from the knee by that length as the end point.

The terminal device 10A specifies a position to draw the auxiliary line VL_42B so that it passes through the waist position JNT_43I and is perpendicular to the ground. Note that the terminal device 10A specifies the position to draw the auxiliary line VL_42B so as to pass through the middle position and be perpendicular to the ground based on the positions of the left and right heels (JNT_43E, JNT_43G). You may

As shown in FIG. 44 , when the user starts crouching, the terminal device 10A draws a trajectory LN_42M along which the user squats on the front surface 4201B. For example, the terminal device 10A may draw the trajectory LN_42M by holding the result of estimating the neck position (or head or shoulders) of the user as a history.

In the right side 4201C of the terminal device 10A, the trajectory LN_42 along which the user crouched
N may be displayed. For example, the terminal device 10A may draw the trajectory LN_42N by holding the result of estimating the position of the user's neck or the like as a history.

Thereby, the user compares the trajectory LN_42M and the auxiliary line VL_42B to determine whether the user is squatting without moving left and right, or compares the trajectory LN_42N and the auxiliary line VL_42C to determine whether the user is squatting without moving forward and backward. easily recognizable.

As shown in FIG. 45, a peak in crouching motion is reached as the user squats down. By detecting that the user's crouching motion has reached its peak (for example, based on the result of estimating the position of a specific part, the terminal device 10A compares the posture with the posture when the user's crouching motion reaches its peak. and detects that the peak has been reached), and holds the sensing result (point cloud data, etc.) at the timing of the peak.

Here, the terminal device 10A may determine whether the extension line LN_42B (that is, the angle of the knee) and the extension line LN_42A (that is, the angle of the back) are open in the fully crouched state. In the example of FIG. 45, the auxiliary line LN_42B and the auxiliary line LN_42A are in an open state with reference to the starting point (heel or waist), and it may be determined that the person is not slouching and is in good condition. . In addition, the terminal device 10A is configured so that the end point of the auxiliary line LN_42B (when the heel is the starting point) and the end point of the auxiliary line LN_42A (when the waist is the starting point) are at the same height from the ground when the user is fully crouched. It is also possible to draw like wax. In addition, the terminal device 10A may determine whether the user's motion is good according to whether the distance between the end point of the auxiliary line LN_42A and the end point of the auxiliary line LN_42B is a certain distance or more ( It is judged to be good when the user is accustomed to the place).

As shown in FIG. 46 , when the user stands up after crouching down, the terminal device 10A displays the sensing result of the user crouching down to an avatar AVT_42D on the left side 4201A, an avatar AVT_42E on the front 4201B, and an avatar AVT_42E on the right side 4201C. Displayed as avatar AVT_42F.

The terminal device 10A displays the auxiliary lines LN_42A and LN_42B in the crouched state on the left side 4201A together with the avatar AVT_42D.

The terminal device 10A draws the trajectory LN_42P (for example, the trajectory is drawn based on the position of the neck) at the time of standing up after crouching down (after reaching the peak timing) on the front 4201B.

The terminal device 10A draws the trajectory LN_42Q (for example, the trajectory is drawn based on the position of the neck) at the time of rising after crouching down (after reaching the peak timing) on the right side 4201C.

(d) Motion to lower back while taking a step FIG. 47 is a diagram showing an example of a screen for the terminal device 10A to cause the user to perform a lower back motion while stepping forward.
FIG. 48 is a diagram showing a screen example of a situation that presents to the user that the user's initial posture is appropriate.
FIG. 49 illustrates the phase of starting to measure the user's motion.
FIG. 50 is a diagram showing a situation in which it is determined whether or not the user's motion is appropriate, and the determined result is presented to the user.
FIG. 51 is a diagram showing a situation in which it is determined whether or not the user's motion is appropriate, and the determined result is presented to the user.

As shown in FIG. 47, the terminal device 10A displays an avatar AVT_47B (corresponding to the front) corresponding to the user on the front 4701B of the screen 4700. FIG. The terminal device 10A displays, on the right side 4701C, an avatar AVT_47C (right side) corresponding to the user, and auxiliary lines LN_47A and LN_47B, which serve as guidelines for the position where the user steps both feet.

In the illustrated example, the terminal device 10A displays an arrow-shaped object that prompts the user to lower the hind leg further behind.

In the example of FIG. 47, the terminal device 10A measures the length of the user's leg based on the result of estimating the position of the user's specific part. For example, the terminal device 10A measures the length of the user's legs based on the position of the user's heels and the position of the waist (pelvis). Thereby, the terminal device 10A specifies the distance to spread the legs when the user performs an action. The terminal device 10A may adjust the distance that the user should spread the legs according to the user's physique (for example, leg length). The terminal device 10A determines the interval between the auxiliary line LN_47A and the auxiliary line LN_47B based on the result of measuring the length of the user's leg, and draws it on the right side surface 4701C.

In the illustrated example, the terminal device 10A displays an object 4701D on the screen 4700 for visualizing the rotation of the user's hips and shoulders. The terminal device 10A identifies the rotation of the user's waist based on the result of estimating the positions of the pelvises (positions of "left pelvis" and "right pelvis" in the example of FIG. 12). The terminal device 10A identifies the rotation of the user's shoulders based on the result of estimating the positions of the shoulders (“left shoulder” and “right shoulder” in the example of FIG. 12).

The terminal device 10A displays an auxiliary line LN_47SB indicating waist rotation and an auxiliary line LN_47SH indicating shoulder rotation, for example, with reference to the position of the waist (such as the center of the pelvis) in the object 4701D. This makes it easier for the user to perceive how much the hips and shoulders are rotating in the action of stepping forward and lowering the hips.

As shown in FIG. 48, the terminal device 10A determines whether or not the legs are sufficiently opened before the user measures the motion of opening the legs, lowering the waist, and raising and lowering the legs.

The terminal device 10A may determine that the user's feet are properly opened forward and backward in the following manner.
・ Based on the results of estimating the positions of parts of the user's body, the angle at which the legs are open (such as the angle formed by the line segment from the left knee to the pelvis and the line segment from the right knee to the pelvis) When the distance is greater than or equal to a predetermined value • When the specific position of the foot (such as the heel) stepped forward or backward by the user and the auxiliary lines LN_47A and LN_47B have a predetermined positional relationship. For example, when the result of estimating the position of the heel of the user's front leg and the auxiliary line LN_47A overlap or are within a certain distance. Similarly, when the result of estimating the position of the heel of the user's hind leg and the auxiliary line LN_47B overlap or are within a certain distance.

The terminal device 10A displays an object OBJ_47A and an object OBJ_47B indicating that the front and rear legs are properly opened, respectively. If the legs are not properly opened, the terminal device 10A displays a message to that effect (for example, "NG") in the object OBJ_47A or OBJ_47B.

The terminal device 10A may draw the object OBG_47A or the object OBJ_47B based on the result of estimating the position of a specific part of the user. For example, the terminal device 10A may draw the object OBJ_47A in association with the coordinates resulting from estimating the position of the user's heel (for example, the balloon of the object OBJ_47A is drawn based on the position of the user's heel. )

As described above, the terminal device 10A, in response to detecting that the posture before the user performs an action is appropriate, notifies the user by voice or image for starting the action (for example, " Sit down slowly and get back on your feet.", etc., to remind the user of the specific action content). As a result, the user can confirm that the posture before the action is appropriate, and it becomes easy to start the action of raising and lowering the waist.

As shown in FIG. 49, in a situation where the user raises and lowers his/her waist, the terminal device 10A is fixed (at a fixed position) on the front 4701B together with the avatar AVT_47B corresponding to the user regardless of the user's movement. Draw auxiliary lines VL_47B and VL_47E to be displayed.

The auxiliary line VL_47B is an auxiliary line for allowing the user to perceive whether or not the user is moving left and right while the user is moving up and down. For example, the terminal device 10A draws the auxiliary line VL_47B based on the result of estimating the position of the waist of the user before the user starts the motion of lowering the waist.

The auxiliary line VL_47E is an auxiliary line for allowing the user to perceive whether the user's legs (for example, front legs) are swaying from side to side while the user is raising and lowering the waist. For example, the terminal device 10A draws the auxiliary line VL_47B based on the results of estimating the positions of the heel and knee of the front leg of the user before the user starts the motion of lowering the waist.

The auxiliary line VL_47F is an auxiliary line for allowing the user to perceive whether or not the user's shoulders are swaying from side to side while the user is moving his/her hips up and down. The terminal device 10A draws the auxiliary line VL_47F in conjunction with the motion of the user to sit down.

The terminal device 10A displays an avatar AVT_47C corresponding to the user and auxiliary lines VL_47C and LN_47C that are fixedly displayed regardless of the movement of the user on the right side 4701C.

The auxiliary line VL_47C is an auxiliary line for allowing the user to perceive whether or not the user can keep his or her hips down vertically without swaying forward or backward while the user is raising or lowering his or her hips. For example, the terminal device 10A draws the auxiliary line VL_47C based on the result of estimating the position of the user's pelvis or the like before the user starts the motion of lowering his or her back.

The auxiliary line LN_47C is for allowing the user to perceive whether the user is leaning too much forward or backward (for example, whether the knees are too forward) while the user is raising and lowering the hips. It is an auxiliary line. For example, the terminal device 10A draws the auxiliary line LN_47C based on the result of estimating the position of the user's heel or the like before the user starts the motion of bending down.

The terminal device 10A draws an object OBJ_47E and an object OBJ_47F, which serve as a guideline for the position and speed of lowering the waist, in conjunction with the speed of the user's lowering and lowering the waist. These objects OBJ_47E and OBJ_47F move along the auxiliary line VL_47C in conjunction with the speed at which the user raises and lowers his hips.

The object OBJ_47E is an object for allowing the user to perceive the reference positions of the neck and shoulders while the user raises and lowers the waist.

Object OBJ_47F is an object for allowing the user to perceive a reference position of the waist while the user moves the waist up and down.

As shown in FIG. 50, it is assumed that the user is leaning slightly forward in order to sit down. As illustrated, on the right side 4701C, the terminal device 10A displays the actual position where the user is moving (the estimated result of the position of the neck, shoulders, head, etc.) with respect to the object OBJ_47E, which is the guideline for the ideal movement. ) is drawn.

The terminal device 10A draws an object OBJ_47K indicating the actual position (estimation result of the position of the waist, pelvis, etc.) at which the user is moving the body, with respect to the object OBJ_47F, which serves as a guideline for ideal movement.

In the terminal device 10A, the object OBJ_47E and the object OBJ_47J are in a predetermined positional relationship (overlapping, within a certain distance, etc.) so that the user can easily perceive whether or not the user is vertically moving his or her hips up and down. In some cases, it may be rendered in a different display mode than in other cases. Similarly, when the object OBJ_47F and the object OBJ_47K are in a predetermined positional relationship, the terminal device 10A may draw them in a different display mode than when they are not. This makes it easy for the user to perceive in real time whether the user is moving his/her hips up and down in an ideal motion, and whether the user is moving the body in accordance with the ideal motion guideline.

The terminal device 10A determines whether or not the forefoot is properly moved, and displays the determination result as an object OBJ_47P. For example, the terminal device 10A may determine whether or not the front leg is properly operated as follows.
・Determine whether the user is properly moving the front leg (e.g., whether the user is leaning forward too much) based on the angle of the front leg, such as the heel, knee, and pelvis of the user ・Parts of the user's lower body position estimation result (heel, knee, etc.) and the auxiliary line LN_47C have a predetermined positional relationship. (For example, if the position of the heel or knee exceeds the auxiliary line LN_47C, it is forward leaning. In addition, the line segment (knee angle) connecting the heel position and the knee position is the auxiliary line LN_47C and the predetermined If it is within the angle, you can drop your hips well, etc.)

The terminal device 10A may specify the position to draw the object OBJ_47P based on the result of estimating the position of the user's body part. For example, the terminal device 10A may draw the object OBJ_47P based on the position of the user's knees. As a result, the object OBJ_47P is drawn while moving in real time with the movement of the user's body, so that it is possible to easily perceive whether the user is appropriately moving the body.

The terminal device 10A determines whether or not the hind legs are properly moved, and displays the determination result as an object OBJ_47N. As in the case of the front legs described above, it may be determined whether the rear legs are properly operated.

The terminal device 10A may determine whether or not the front leg is swaying to the left or right in the front 4701B, and display the determination result as an object OBJ_47M. For example, based on the results of estimating the heel position and the knee position, the terminal device 10A, when the angle of the knee is not vertical (for example, when the angle is greater than or equal to a predetermined angle with respect to the auxiliary line VL_47E), It may be determined that the forefoot is swaying from side to side. In the example of FIG. 50, the front leg does not move left and right, and "OK" is displayed in object OBJ_47M.

As shown in FIG. 51, on the right side 4701C, the user has his/her waist raised and lowered vertically, and "OK" is displayed for objects OBJ_47P and OBJ_47N.

(3) Supplementary notes The matters described in the above embodiments are additionally noted below.

(Appendix 1)
A program for operating a computer, wherein the program instructs the processor of the computer to sequentially sense the movement of the user's body, and based on the sensing result, the user's appearance is photographed. A step of presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the user's body surface instead of the image, and based on the sensing result, associating with the avatar to serve as a guideline for the movement of the user's body. A program for executing the steps of identifying a position for drawing a graphic object and displaying the graphic object at the identified position.
(Appendix 2)
1. The program according to appendix 1, wherein, in the step of presenting the avatar, the avatar is presented to the user in a form in which the amount of information is lower than that of the actual image.
(Appendix 3)
3. The program according to appendix 1 or 2, wherein in the step of presenting the avatar, the avatar is presented to the user each time the sensing result is acquired.
(Appendix 4)
In the step of presenting the avatar, based on the sensing result, both the avatar drawn from the first direction and the avatar drawn from the second direction different from the first direction are drawn side by side. 4. The program according to any one of Appendixes 1 to 3, for presenting an avatar to a user by doing.
(Appendix 5)
Supplementary Note 1: In the step of sensing, the movement of the user's body is sensed as point cloud data, and in the step of presenting the avatar, the avatar is presented to the user by drawing the avatar as an aggregate of the point cloud data. 5. The program according to any one of 4 to 4.
(Appendix 6)
The program further instructs the processor to estimate the positions of a plurality of specific parts of the user based on the results of sensing the movement of the user's body, and to draw the avatar while estimating the positions of the specific parts. 6. The program according to any one of appendices 1 to 5, causing the execution of a step of evaluating how to move the user's body based on the estimation result of the position of the specific part without drawing the result.
(Appendix 7)
In the step of presenting the avatar, the drawing of the avatar is updated according to the sensing result of the body movement of the user, while in the step of displaying, the graphic object is displayed at a fixed position regardless of the movement of the user's body. 7. The program according to any one of 1 to 6.
(Appendix 8)
The program further instructs the processor to estimate the position of a specific part of the user based on the result of sensing the movement of the user's body; 8. The program according to appendix 7, causing the execution of the step of drawing the sensed history in chronological order along with the avatar for the position.
(Appendix 9)
7. The method according to appendix 7, wherein in the specifying step, based on the result of sensing the user, the linear graphic object is specified to be drawn parallel to the body axis of the user and at a fixed position regardless of the movement of the user's body. program.
(Appendix 10)
In the specifying step, based on the sensing result of the first posture of the user before the user starts the specific action, the position where the graphical object is to be drawn as a reference when the user performs the specific action is specified. , Supplementary Note 1.
(Appendix 11)
The specific action is a predetermined action for measuring the movement of the user's body. 11. The program according to appendix 10, which identifies a position to draw a graphical object.
(Appendix 12)
11. according to appendix 10, wherein in the specifying step, based on the first posture, a position for drawing a graphical object serving as a reference for at least one of a group consisting of front-rear balance and left-right balance of the user's body is specified program.
(Appendix 13)
11. The method according to appendix 10, wherein in the specifying step, a position for drawing a graphical object that serves as a guideline for a second posture that the user can take while performing a specific action is specified based on the first posture. program.
(Appendix 14)
The specific action is the action performed by the user with both feet spread and stepped on. In the step of specifying, the position for drawing the graphic object as a guideline for the state where the user steps on the front foot is set to the first posture. 14. The program of clause 13, wherein the program is identified based on.
(Appendix 15)
Based on the result of sensing the movement of the user's body, the step of estimating the positions of the lower limbs of the user is further executed, and in the step of specifying, based on the result of the estimation, the position where the user steps with both legs open is determined. 15. The program according to appendix 14, which specifies a position for drawing a graphical object that serves as a guide.
(Appendix 16)
a step of estimating the position of a specific part of the user based on the result of sensing the movement of the user's body; 14. The program according to appendix 13, causing the execution of a step of evaluating whether the posture of 2 is properly performed.
(Appendix 17)
17. The program according to appendix 16, wherein when it is evaluated that the second posture is properly performed in the evaluating step, the user is presented with that fact.
(Appendix 18)
The specific action involves a body movement performed by the user in the second posture, and in the evaluating step, the specific action is performed in response to evaluating that the second posture is properly performed. 17. The program according to appendix 16, causing the measurement to start and presenting to the user to start the measurement.
(Appendix 19)
further executing a step of estimating positions of a plurality of specific parts of the user based on the result of sensing the movement of the user's body; 2. The program according to appendix 1, which identifies a position to draw a linear graphic object for letting the user perceive whether the part is properly moved during the motion of the user's body.
(Appendix 20)
20. The program according to appendix 19, wherein, in the displaying step, a line segment obtained by extending a line extending from a part surrounding a specific part to the specific part is displayed together with the avatar as a graphic object.
(Appendix 21)
21. The program according to appendix 20, wherein in the displaying step, a graphic object having a size larger than the width of the line segment is drawn and displayed at the end point of the line segment.
(Appendix 22)
In the step of displaying, the graphic object corresponding to the end point of the line segment is displayed so as to move in conjunction with the sensing result of the user's body movement, and is fixedly arranged regardless of the user's body movement. 22. The program according to appendix 21, wherein the display mode of the graphic object is changed and displayed based on the positional relationship between the object and the graphic object corresponding to the end point.
(Appendix 23)
In the step of further executing and displaying a step of estimating the positions of a plurality of specific parts of the user based on the result of sensing the movement of the user's body, a graphic object indicating the inclination of the first part of the user; 20. The program according to Supplementary Note 19, wherein a graphical object indicating the inclination of a second portion different from the first portion is drawn and displayed in association with the first portion and the second portion.
(Appendix 24)
further executing a step of estimating the position of a specific part of the user based on the result of sensing the movement of the user's body; 20. The program according to appendix 19, wherein a position for drawing a graphical object is identified based on and.
(Appendix 25)
The specific action is a predetermined action for measuring the movement of the user's body. The program according to appendix 1, wherein the user's sensing result at the timing of is stored in the storage unit, so that the avatars at the predetermined timing are displayed side by side and presented to the user.
(Appendix 26)
In the step of presenting the avatar, the step of estimating the positions of at least three specific parts of the user based on the result of sensing the movement of the user's body, and estimating one of the at least three specific parts As a reference, determining a position for drawing an object indicating the positional relationship of other specific parts on a horizontal plane, and displaying the object at the determined position; 1. The program according to appendix 1, which specifies a position to draw the graphic object in association with the avatar based on the estimation result of .
(Appendix 27)
A program for operating a computer, the program causing a processor of the computer to sequentially sense the movement of a user's body; estimating the positions of three specific parts, and using one of the at least three specific parts as a reference, determining the position of drawing a graphical object indicating the positional relationship of the other specific parts on the horizontal plane. , displaying the graphical object at the determined position, and executing the program.
(Appendix 28)
28. The program according to appendix 27, wherein in the step of displaying the graphic objects, the graphic objects representing the specific parts that are not the reference are displayed as graphic objects including graphics of the same or similar shape.
(Appendix 29)
A method for operating a computer, the method comprising:
A step of sequentially sensing the movement of the user's body, and responding to the user in a manner in which the shape of the user's body surface is perceived instead of a photographed image of the user's appearance based on the sensing results. a step of presenting an avatar to a user; a step of identifying a position for drawing a graphical object that serves as a guideline for movement of the user's body in association with the avatar based on the sensing results; The steps of displaying and the method of performing.
(Appendix 30)
An information processing device, wherein the control unit sequentially senses the movement of the user's body, and based on the sensing result, the user's appearance is captured instead of the actual image. A step of presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the body surface of the user; An information processing apparatus that performs the steps of specifying a drawing position and displaying a graphical object at the specified position.

10 terminal device, 20 server, 30 measurement unit, 31 measurement device, 32 mat, 33 pole, 132 display, 181 user information, 182 measurement item information, 183 measurement history information, 184 friend list, 185 training item information, 281 user information Database, 282 measurement item information, 283 measurement history database, 284 friend list, 285 training item information, 287 measurement result database, 288 evaluation result database, 289 user questionnaire information, 290 evaluation result database, 2601 trainee information database, 2602 trainer information database , 2603 Training History Database

Claims (30)

A program for operating a computer, the program causing a processor of the computer to:
a step of sequentially sensing the movement of the user's body;
a step of presenting an avatar corresponding to the user to the user based on the result of the sensing in such a manner that the user perceives the shape of the body surface of the user rather than a photographed image of the appearance of the user;
identifying a position to draw a graphical object that serves as a guideline for movement of the user's body in association with the avatar based on the sensing result;
displaying the graphic object at the identified position;
The program that causes the to run. 2. The program according to claim 1, wherein, in the step of presenting the avatar, the avatar is presented to the user in a form in which the amount of information is lower than that of the photographed image. 3. The program according to claim 1, wherein in the step of presenting the avatar, the avatar is presented to the user each time the sensing result is obtained. In the step of presenting the avatar,
By arranging both an avatar of the user drawn from a first direction and an avatar of the user drawn from a second direction different from the first direction, based on the sensing result , presenting the avatar to the user. in the sensing step, sensing the movement of the user's body as point cloud data;
5. The program according to any one of claims 1 to 4, wherein in the step of presenting the avatar, the avatar is presented to the user by rendering the avatar as an aggregate of the point cloud data. The program further causes the processor to:
estimating the positions of a plurality of specific parts of the user based on the results of sensing the body movements of the user;
Evaluating how to move the user's body based on the result of estimating the position of the specific part while drawing the avatar without drawing the result of estimating the position of the specific part. 6. The program according to any one of claims 1 to 5, which causes In the step of presenting the avatar, updating the drawing of the avatar according to the result of sensing the movement of the user's body;
7. The program according to any one of claims 1 to 6, wherein, in said displaying step, said graphical object is displayed at a fixed position regardless of said user's body movement. The program further causes the processor to:
estimating the position of a specific part of the user based on the result of sensing the movement of the user's body;
8. The program according to claim 7, causing the execution of a step of drawing the sensed history along with the avatar in chronological order with respect to the position of the first part whose movement of the user's body is to be measured. In the identifying step, based on the result of sensing the user, it is identified that the linear graphic object is drawn parallel to the user's body axis and at a fixed position regardless of the user's body movement. 8. The program according to claim 7. In the identifying step, based on the result of sensing the first posture of the user before the user starts a specific action, the graphic object is drawn as a guideline for when the user performs the specific action. 2. The program according to claim 1, which identifies a position to be set. the specific action is a predetermined action for measuring the movement of the user's body;
11. The method according to claim 10, wherein, in said identifying step, a position to draw said graphic object that serves as a guideline for movement of said user's body at a predetermined timing of said specific action is identified based on said first posture. program. In the identifying step, identifying a position for rendering the graphic object that serves as a guideline for at least one of a group consisting of front-rear balance and left-right balance of the user's body, based on the first posture. Item 11. The program according to item 10. In the identifying step, a position at which the graphical object is to be drawn, which serves as a guideline for a second posture that the user can take while performing the specific action, is specified based on the first posture. 11. The program according to claim 10. The specific motion is a motion performed by the user with both feet spread and stepped on,
14. The program according to claim 13, wherein, in said specifying step, a position for drawing said graphic object that serves as a guideline for a state in which said user has stepped a front foot is specified based on said first posture. Further executing a step of estimating the position of the user's lower limb part based on the result of sensing the user's body movement;
15. The program according to claim 14, wherein, in said specifying step, based on said estimated result, a position to draw said graphic object, which is a guideline for a position where said user steps with both feet open, is specified. estimating the position of a specific part of the user based on the result of sensing the movement of the user's body;
and evaluating whether the user is appropriately performing the second posture based on the positional relationship between the estimated position of the specific part and the position of the graphic object. 13. The program according to 13. 17. The program according to claim 16, wherein when it is evaluated that the second posture is properly performed in the step of evaluating, the fact is presented to the user. the specific action involves a body movement performed by the user in the second posture;
In the evaluating step, in response to evaluating that the second posture is properly performed, measurement of the specific action is started, and the user is presented with the start of the measurement. 17. The program according to Item 16. Further executing a step of estimating the positions of a plurality of specific parts of the user based on the result of sensing the movement of the user's body;
In the identifying step, the linear graphic for making the user perceive whether the specific part is appropriately moved during the movement of the user's body, based on the estimated position of the specific part. 2. The program according to claim 1, which specifies a position to draw an object. 20. The program according to claim 19, wherein, in said displaying step, a line segment obtained by extending a line extending from a part surrounding said specific part to said specific part is displayed together with said avatar as said graphic object. 21. The program according to claim 20, wherein in said displaying step, said graphic object having a size larger than the width of said line segment is drawn and displayed at the end point of said line segment. In the displaying step, the graphic object corresponding to the end point of the line segment is displayed so as to move in conjunction with the sensing result of the body movement of the user;
changing the display mode of the graphical object based on the positional relationship between the graphical object fixedly arranged regardless of the movement of the user's body and the graphical object corresponding to the end point and displaying the graphical object; Item 22. The program according to item 21. Further executing a step of estimating the positions of a plurality of specific parts of the user based on the result of sensing the movement of the user's body;
In the displaying step, the graphical object indicating the inclination of the first part of the user and the graphical object indicating the inclination of the second part different from the first part are displayed on the first part, 20. The program according to claim 19, which draws and displays each in association with the second part. Further executing a step of estimating the position of a specific part of the user based on the result of sensing the movement of the user's body;
20. The program according to claim 19, wherein, in said specifying step, a position for drawing said graphic object is specified based on said estimated position of said specific part and the shape of said user's body surface. the specific action is a predetermined action for measuring the movement of the user's body;
In the presenting step, the avatar is presented to the user each time the sensing result is acquired, and the user's sensing result at a predetermined timing, which is the delimiter of the specific action, is stored in a storage unit. , the program according to claim 1, wherein the avatars at the predetermined timing are arranged and presented to the user. In the step of presenting the avatar,
estimating the positions of at least three specific parts of the user based on the results of sensing the movement of the user's body;
using any one of the at least three specific parts as a reference, determining a position for drawing an object indicating the positional relationship of other specific parts on a horizontal plane, and displaying the object at the determined position. let
2. The program according to claim 1, wherein, in said specifying step, a position to draw said graphic object is specified in association with said avatar based on the estimation results of the positions of said at least three parts. A program for operating a computer, the program causing a processor of the computer to:
a step of sequentially sensing the movement of the user's body;
estimating the positions of at least three specific parts of the user based on the results of sensing the movement of the user's body;
determining a position for drawing a graphical object indicating the positional relationship of the other specific parts on a horizontal plane with reference to one of the at least three specific parts, and displaying the graphical object at the determined position; The program that causes the to run. In the step of displaying the graphic object,
28. The program according to claim 27, wherein the graphic objects representing the specific parts that are not the reference are displayed as graphic objects including graphics of the same or similar shape. A method for operating a computer, said method comprising: a processor of said computer;
a step of sequentially sensing the movement of the user's body;
a step of presenting an avatar corresponding to the user to the user based on the result of the sensing in such a manner that the user perceives the shape of the body surface of the user rather than a photographed image of the appearance of the user;
identifying a position to draw a graphical object that serves as a guideline for movement of the user's body in association with the avatar based on the sensing result;
displaying the graphic object at the identified position;
how to run An information processing device, wherein the control unit comprises:
a step of sequentially sensing the movement of the user's body;
a step of presenting an avatar corresponding to the user to the user based on the result of the sensing in such a manner that the user perceives the shape of the body surface of the user rather than a photographed image of the appearance of the user;
identifying a position to draw a graphical object that serves as a guideline for movement of the user's body in association with the avatar based on the sensing result;
displaying the graphic object at the identified position;
An information processing device that executes

Images