WO2022030619A1

WO2022030619A1 - Guidance support system

Info

Publication number: WO2022030619A1
Application number: PCT/JP2021/029301
Authority: WO
Inventors: 侑也 ▲高▼久
Original assignee: 株式会社Sportip
Priority date: 2020-08-07
Filing date: 2021-08-06
Publication date: 2022-02-10
Also published as: JPWO2022030619A1

Abstract

[Problem] To easily evaluate physical exercise. [Solution] Provided is a guidance support system which supports, by a supporter, guidance given to a user who performs physical exercise, and comprises: a reference value storage unit which stores a reference value related to the position of at least one part of a body or a portion of a tool; a part and portion specifying unit which specifies the part or the portion by analyzing an image of the user's physical exercise; an evaluation unit which determines an evaluation value of the physical exercise by comparing the position of the part or the portion in the image with the reference value; and a supporter terminal having an imaging function for capturing an image of the user's physical exercise and a display function for displaying the evaluation value.

Description

Guidance support system

The present invention relates to a guidance support system.

Physical exercise guidance and coaching are widely performed in aspects such as sports and other sports, muscle strength training, and rehabilitation.

Guidance and coaching are often provided by experienced and knowledgeable supporters (including instructors, coaches, physiotherapists, care workers, etc.), and users who perform physical exercise can hear objective opinions. Although it is useful, it is often done based on the supporter's intuition and experience, and sometimes the supporter cannot verbalize, so the effectiveness of guidance and coaching is low.

Patent Document 1 discloses an invention that supports instruction by comparing the movement of a user and the movement of an instructor by juxtaposing them as a method of instructing sports skills.

Japanese Unexamined Patent Publication No. 2007-313362

However, in the technology of Patent Document 1, although the difference between the user and the instructor becomes clear, the knowledge of how to make the difference close and what kind of training is required depends on the instructor. In addition, since moving images are shot and compared, it is not possible to give guidance immediately after or during physical exercise, and the effect is limited.

The present invention has been made in view of such a background, and can easily detect and evaluate the movement of a body part or a tool part from an image of a user's body movement, further improving skills, training, rehabilitation, etc. The purpose is to provide technology that can support exercise by giving advice for improving the effect of.

The present invention is a guidance support system that supports guidance given by a supporter to a user who performs physical exercise, and is a reference value storage that stores a reference value relating to the position of at least one body part or tool part. The part, the part specifying the part or the part that identifies the part by analyzing the image of the physical exercise of the user, and the position of the part or the part in the image and the reference value are compared with each other. It is characterized by including an evaluation unit for determining an evaluation value, an imaging function for capturing an image of the user's body movement, and a supporter terminal having a display function for displaying the evaluation value.

Other problems disclosed in the present application and solutions thereof will be clarified by the columns and drawings of the embodiments of the invention.

According to the present invention, it is possible to easily evaluate the skills and effects of physical exercise using tools, and provide support that contributes to skill improvement and effect improvement.

It is a figure which shows the whole structure example of the physical exercise support system which concerns on this embodiment. It is a figure which shows the hardware configuration example of the user terminal 10. It is a figure which shows the software configuration example of a user terminal 10. It is a figure which shows the structural example of the physical information which a physical information storage part 130 stores. It is a figure which shows the configuration example of the evaluation request which the evaluation request transmission unit 112 transmits to a server apparatus 20. It is a figure which shows the structural example of the evaluation information which the evaluation information receiving part 113 receives from a server apparatus 20. It is a figure which shows the configuration example of the improvement measure request which the improvement measure request transmission part 116 transmits to the server apparatus 20. It is a figure which shows the configuration example of the reference value request which the reference value request transmission unit 119 transmits to the server apparatus 20. It is a figure which shows the structure example of improvement measure information. It is a figure which shows the hardware configuration example of the server apparatus 20. It is a figure which shows the software configuration example of a server apparatus 20. It is a figure which shows the structural example of the image information stored in the image data storage unit 231. It is a figure which shows the structural example of the reference information which a reference information storage part 232 stores. It is a figure which shows the structural example of the evaluation condition information stored in the evaluation condition information storage unit 233. It is a figure which shows the structural example of the improvement measure information stored in the improvement measure information storage unit 234. It is a figure which shows the structural example of the group analysis information stored in the group analysis information storage part 235. It is a figure which shows the flow of the process executed in the instruction support server of this embodiment. It is a figure which shows an example of the evaluation screen of a moving image in a mode of weightlifting. It is another figure which shows an example of the evaluation screen of a moving image in a mode of weightlifting. It is another figure which shows an example of the evaluation screen of a moving image in a mode of weightlifting. It is another figure which shows an example of the evaluation screen of a moving image in a mode of weightlifting. It is another figure which shows an example of the evaluation screen of a moving image in a mode of weightlifting. It is another figure which shows an example of the evaluation screen of the moving image in the squat mode, which is presented to a user terminal 10. It is a figure which shows an example of the evaluation screen of the moving image in group training presented to a supporter terminal 40. It is another figure which shows an example of the evaluation screen of the moving image in group training presented to a supporter terminal 40. It is another figure which shows an example of the evaluation screen of the moving image in group training presented to a supporter terminal 40. It is another figure which shows an example of the image which a supporter views through a supporter terminal 40.

The contents of the embodiments of the present invention will be listed and described. The guidance support server according to the embodiment of the present invention has the following configuration.

[Item 1]
It is a guidance support system that supports the guidance given by the supporter to the user who performs physical exercise.
A reference value storage unit that stores a reference value related to the position of at least one body part or tool part, and a reference value storage unit.
A part specifying part that identifies the part or the part by analyzing an image of the user's physical exercise,
An evaluation unit that determines the evaluation value of the physical exercise by comparing the position of the portion or the portion in the image and the reference value.
Equipped with
An imaging function that captures an image of the user's physical exercise,
A supporter terminal having a display function for displaying the evaluation value, and
Guidance support system characterized by being equipped with.
[Item 2]
The supporter terminal must be a wearable computer.
The guidance support system according to claim 1, wherein the guidance support system is characterized by the above.
[Item 3]
There are multiple reference values,
A reference value determining unit that presents one or more reference values to the user based on at least one of the user's purpose, physical information, and the evaluation value.
The guidance support system according to claim 1 or 2, wherein the system is provided with.
[Item 4]
The reference value selection information transmission unit selected by the user from the reference value candidates, and
The guidance support system according to claim 1 to 3, further comprising.
[Item 5]
It is a guidance support method that supports the guidance given by the supporter to the user who performs physical exercise.
A reference value storage step that stores a reference value relating to the position of at least one body part or tool part,
A site part identification step for identifying the part or the part by analyzing an image of the user's physical exercise,
An evaluation step of comparing the position of the part or the part in the image and the reference value to determine the evaluation value of the physical exercise, and the evaluation step.
Equipped with
An imaging function that captures an image of the user's physical exercise,
A supporter terminal having a display function for displaying the evaluation value, and
Guidance support method characterized by providing.

The guidance support server according to the embodiment of the present invention evaluates the movement of the body and the movement of the tool in the physical exercise performed by the user (including the one performed by using the tool), and then the supporter gives guidance. It is an attempt to support that.

Physical exercises performed by the user include those that do not use tools such as gymnastics, fitness, walking, running, yoga, self-weight training, and rehabilitation, and those that use tools such as soccer, volleyball, and basketball. Competitions that use balls and other tools such as baseball, tennis, table tennis, and golf, competitions that use tools other than balls such as kendo and fencing, physical exercises that use tools such as juggling, barbells, etc. It includes, but is not limited to, physical exercise using training equipment and machines, various rehabilitation using tools, movements of the care recipient using tools such as canes, and playing using musical instruments.

The instruction support server of the present embodiment is, for example, an image obtained by capturing a state in which a user is exercising a body (a still image or a moving image may be used, but in the present embodiment, the moving image is used. From), identify the body and parts of the body, tools and parts of the tool, and based on the absolute position of the part or part and the relative positional relationship of multiple different parts, parts, etc. Evaluate the movement of the tool. The movement of the tool may be evaluated based on the relative positional relationship between the plurality of different parts of the body and the part of the tool.

FIG. 1 is a diagram showing an overall configuration example of the guidance support server according to this embodiment. As shown in the figure, the instruction support server of the present embodiment includes a user terminal 10 and a server device 20. Further, the image pickup terminal 50 may be included. The user terminal 10, the server device 20, and the image pickup terminal 50 are connected to each other so as to be able to communicate with each other via the communication network 30. The communication network 30 is, for example, the Internet or a LAN (Local Area Network), and is constructed by a public telephone line network, a dedicated telephone line network, a mobile telephone line network, an Ethernet (registered trademark), a wireless communication path, or the like.

== User terminal 10 ==
The user terminal 10 is a computer operated by a user who performs physical exercise or a supporter thereof. The user terminal 10 is, for example, a smartphone, a tablet computer, a personal computer, a wearable computer, or the like. The user terminal 10 includes an image pickup device such as a camera, which can image the user's body during exercise. In the present embodiment, it is assumed that the moving image of the body of the user during exercise is transmitted from the user terminal 10 to the server device 20. Further, the user terminal 10 has an image pickup function that captures an image close to the user's field of view, and has an output function for visually recognizing characters, images, videos, etc. by a virtual image projection method, a retinal projection method, or any other method. , A hat-type computer, an HMD (head-mounted display), or the like, or a separate terminal may be responsible for the image pickup function and the output function. The user terminal 10 may also serve as the image pickup terminal 50 and the output terminal 60, the image pickup function of the user terminal 10 may also serve as the image pickup terminal 50, and the output function of the supporter terminal 40 is the output terminal 60. May also serve as. Further, although only one user terminal 10 is shown in FIG. 1, it goes without saying that there may be a plurality of user terminals 10.

== Server device 20 ==
The server device 20 is a computer that evaluates the movements of the body and tools. The server device 20 is, for example, a workstation, a personal computer, a virtual computer logically realized by cloud computing, or the like. The server device 20 receives the moving image taken by the user terminal 10, analyzes the received moving image, and evaluates the physical exercise. The server device 20 also makes a proposal for improving physical exercise. Details of the evaluation of physical exercise and the proposal of improvement measures will be described later.

== Supporter terminal 40 ==
The supporter terminal 40 is a computer operated by a person (supporter) who plays a role of providing guidance, teaching, explanation, support, etc. to a user who performs physical exercise, such as a trainer, a physiotherapist, and a caregiver. The supporter terminal 40 is, for example, a smartphone, a tablet computer, a personal computer, a wearable computer, or the like. In addition, the supporter terminal 40 has an image pickup function that captures images close to the field of view of the supporter, and has an output function for visually recognizing characters, images, videos, etc. by a virtual image projection method, a retinal projection method, or other methods. A computer such as a lens type, a hat type, or an HMD (head-mounted display) may be used, and another terminal may be responsible for the image pickup function and the output function. The supporter terminal 40 may also serve as the image pickup terminal 50 and the output terminal 60, the image pickup function of the supporter terminal 40 may also serve as the image pickup terminal 50, and the output function of the supporter terminal 40 is output. It may also serve as a terminal 60. Further, although only one supporter terminal 40 is shown in FIG. 1, it goes without saying that there may be a plurality of supporter terminals 40.

== Imaging terminal 50 ==
The image pickup terminal 50 is a computer that captures an image of the user, a camera having a communication function, or the like, which is installed or attached to a place where the user exercises. The image pickup terminal 50 is, for example, a smartphone, a tablet computer, a personal computer, a wearable computer, or the like. The image pickup terminal 50 includes an image pickup device such as a camera, which can take an image of the user's body during exercise. In the present embodiment, it is assumed that the moving image of the body of the user who is exercising is transmitted from the image pickup terminal 50 to the server device 20. The data stored in the image pickup terminal 50 and captured by the user may be directly input to the server device 20 by the user, its supporter, or a business operator using the guidance support server, or the communication network. It may be input via 30. Further, although only one image pickup terminal 50 is shown in FIG. 1, it goes without saying that there may be a plurality of image pickup terminals 50. Further, the image pickup terminal 50 may be installed at a plurality of places to image the subject at the same time or in order. Further, the image pickup terminal 50 may also serve as the user terminal 10 and the output terminal 60, or may be responsible for the image pickup function of the user terminal 10 and the supporter terminal 40.

== Output terminal 60 ==
The output terminal 60 shall output information under the control of the server device 20 via the communication network 30. The output terminal 60 may be, for example, a smartphone, a tablet computer, a personal computer, a wearable computer, or the like, or by connecting a computer to an output device such as a display and communicating between the computer and the server device 20. , Information may be output to the output terminal 60 in response to an instruction from the server device 20. Further, the output terminal 60 includes an audio output device, and a user can be attached to an audio output device such as headphones, earphones, or a neck speaker to output information as audio. In addition, virtual image projection method, retinal projection method, eyeglass type, contact lens type, hat type, other types of output terminals that make characters, images, images, etc. visible by other methods, HMD (head mounted display), and brain waves. Information is sent to the user or supporter via an interface such as BMI (Brain Machine Interface) that inputs characters, images, videos, etc. by direct stimulation to the brain without using a sensory device. It can also be output. Further, although only one output terminal 60 is shown in FIG. 1, it goes without saying that there may be a plurality of output terminals 60, and video output, audio output, and other outputs may be performed by one output terminal. It may be done with another output terminal. Further, the output terminal 60 may also serve as the user terminal 10 and the supporter terminal 40, or may have an output function of the user terminal 10 and the supporter terminal 40.

FIG. 2 is a diagram showing a hardware configuration example of the user terminal 10. The user terminal 10 includes a CPU 101, a memory 102, a storage device 103, a communication interface 104, a touch panel display 105, and a camera 106. The storage device 103 stores various data and programs, such as a hard disk drive, a solid state drive, and a flash memory. The communication interface 104 is an interface for connecting to the communication network 30, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, and a wireless communication device for performing wireless communication. , USB (Universal Serial Bus) connector for serial communication, RS232C connector, etc. The touch panel display 105 is a device that inputs / outputs data. The user terminal 10 may further include an input device such as a keyboard, a mouse, a button, and a microphone, and an output device such as a speaker and a printer.

FIG. 3 is a diagram showing a software configuration example of the user terminal 10. The user terminal 10 includes an imaging unit 111, an evaluation request transmission unit 112, an evaluation information receiving unit 113, an evaluation display unit 114, a checkpoint display unit 115, an improvement measure request transmission unit 116, an improvement measure information receiving unit 117, and an improvement measure information display. Each functional unit of the unit 118, the reference value request transmission unit 119, the reference value selection information transmission unit 120, and each storage unit of the physical information storage unit 130, the image storage unit 131, the evaluation information storage unit 132, and the improvement measure storage unit 133. To prepare for.

Each of the above functional units is realized by the CPU 101 included in the user terminal 10 reading a program stored in the storage device 103 into the memory 102 and executing the program, and each of the above storage units is the memory 102 included in the user terminal 10. And implemented as part of the storage area provided by the storage device 103.

The image pickup unit 111 captures an image during physical exercise performed by the user, including a moving image. By controlling the camera 106, the image pickup unit 111 can acquire a moving image of body movement. The user or the user's supporter installs the user terminal 10 on a flat place, a wall, or the like, directs the optical axis of the camera 106 to the place where the user exercises, and instructs the start of video shooting, and responds accordingly. The imaging unit 111 may operate the camera 106 to acquire a moving image. The image pickup unit 111 stores the acquired moving image in the image storage unit 131.

The image storage unit 131 stores the image captured by the image pickup unit 111. In the present embodiment, the image is a moving image, but the image is not limited thereto. The image storage unit 131 can store a moving image as a file, for example.

The physical information storage unit 130 stores information (hereinafter referred to as physical information) related to the user's body, physical ability, factors affecting the training effect, and the like. FIG. 4 is a diagram showing a configuration example of physical information stored by the physical information storage unit 130. As shown in the figure, physical information includes height, weight, gender, dominant hand, arm length, foot length, hand size, finger length, grip strength, muscle strength, flexibility, and shoulder strength. It includes genomes, epigenomes, gene polymorphisms, intestinal flora, diet, etc. In addition, the type and number of physical exercises performed may be stored in the image analysis performed by the image analysis unit 212.

The evaluation request transmission unit 112 transmits a request for evaluating physical exercise using a tool (hereinafter referred to as an evaluation request) to the server device 20 based on the image captured by the image pickup unit 111.

FIG. 5 is a diagram showing a configuration example of an evaluation request transmitted by the evaluation request transmission unit 112 to the server device 20. As shown in the figure, the evaluation request includes a user ID, a mode, physical information and image data. The user ID is information that identifies the user. The mode is information indicating the physical exercise performed by the user. The modes can be, for example, "walking", "yoga: cat pose", "bench press", "tennis serve", "walking rehabilitation" and the like. The mode shall be selected from predetermined options. The physical information is physical information stored in the physical information storage unit 130. The image data is moving image data acquired by the imaging unit 111.

The evaluation information receiving unit 113 receives information related to the evaluation of physical exercise (hereinafter referred to as evaluation information), which is responded to by the server device 20 in response to the evaluation request. The evaluation information receiving unit 113 registers the received evaluation information in the evaluation information storage unit 132.

FIG. 6 is a diagram showing a configuration example of evaluation information received from the server device 20 by the evaluation information receiving unit 113. As shown in the figure, the evaluation information includes a mode, a user ID, a tool position information, a body part position information, a posture information, a movement information, and a checkpoint information.

The user ID and mode are the user ID and mode included in the evaluation request. The captured image indicates that the body is photographed by the user performing the exercise indicated by the mode.

The tool position information is for each part of the tool (for example, the entire bat used for baseball, both ends, the grip, the place where the ball meets, the center of gravity, and any other part. Also, the entire barbell, shaft, and plate (weight part). , The gripped part, the center of the shaft, the center of gravity, and any other part. Including the entire tool.) Indicates the position in the image. The tool position information includes each part of the tool and the position of the part in association with the time point on the time axis of the moving image. Based on the tool position information, the movement of the tool and the relationship with the body part can be displayed. That is, for example, a figure (for example, a circle) indicating a portion can be superimposed and displayed on the image at the position indicated by the tool position information. It should be noted that the positions of a plurality of parts may be included at one time point. Further, the part connecting the two parts (for example, the midpoint between the parts where the barbell is held by the right hand and the left hand) does not have to include the tool position information. In this case, by connecting a pair of marks (such as a circle) indicating two predetermined parts with a line, the part connecting these two parts can be expressed. The tool partial position information may be included for each frame constituting the video, for each key frame (including the frame for the checkpoint described later), or for any number. It may be included for each frame, or it may be included for random time points. When the position information is not included in My Frame, the figure can be displayed based on the position information at the nearest past time point.

Body position information indicates the position of each part of the body (for example, head, shoulders, elbows, hips, knees, ankles, etc.) in the image. The body position information includes a part and a position of the part in association with a time point on the time axis of the moving image. The state of the skeleton (bone) of the body can be displayed based on the body position information. That is, for example, a figure (for example, a circle) indicating a part can be superimposed and displayed on an image at a position indicated by body position information. It should be noted that the positions of a plurality of parts may be included at one time point. The position information may not be included in the part connecting the two parts (for example, the forearm connecting the wrist and the elbow or the thigh connecting the waist and the knee). In this case, by connecting a pair of marks (such as a circle) indicating two predetermined parts with a line, the part connecting these two parts can be expressed. The position information may be included for each frame constituting the video, for each key frame (including the frame for the checkpoint described later), or for each frame of any number. It may be included in, or it may be included at random time points. If My Frame does not contain location information, you can display bones based on the location information at the nearest past point in time.

Tool orientation information is information related to the orientation of the tool used by the user and the direction in which the part of the tool is facing. The tool orientation information includes a part of the tool to be evaluated, a tool movement value, an evaluation rank, an evaluation comment, and the like in association with a time point on the time axis of the moving image. The tool orientation value is a value indicating the orientation of the tool. The tool orientation value is, for example, the distance from the ground to a certain part of the tool, the distance between two parts of the tool, the angle of the part (to the first part of the tool and, for example, the part where the user holds the tool). The angle between the straight line of the tool, the second part of the tool, and the straight line to the part where the user holds the tool, for example), the movement of a certain part of the tool, and the like. The evaluation rank is a value representing the evaluation value by rank. The evaluation rank is expressed by, for example, 1 to 5 in 5 stages, ABC, or the like. The evaluation comment is a comment related to the evaluation regarding the posture. For example, if the mode is "upright row" and the distance between the right and left ends of the barbell is different from the ground, it may contain an evaluation comment such as "different forces are applied to the left and right".

Tool movement information is information related to the movement of the tool used by the user. The tool movement information includes a part of the tool to be evaluated, a list of the tool orientation values, an evaluation rank, and an evaluation comment in association with a period on the time axis of the moving image. The list of tool orientation values is a time-series tool orientation value within a period. The evaluation comment is a comment related to the evaluation regarding the movement of the tool. For example, if the mode is "upright row" and the barbell does not move up and down sufficiently, it may include a rating comment such as "the barbell has not been lifted sufficiently".

Posture information is information related to the posture of the user's body. The posture information includes a part to be evaluated, a posture value, an evaluation rank, and an evaluation comment in association with a time point on the time axis of the moving image. The posture value is a value representing the posture. Posture values are, for example, the distance from the ground to the part, the distance between the two parts, the angle of the indirect part (the straight line from the first end part to the indirect part, and the second end part to the indirect part). The angle created by the straight line to). The evaluation rank is a value representing the evaluation value by rank. The evaluation rank is expressed by, for example, 1 to 5 in 5 stages, ABC, or the like. The evaluation comment is a comment related to the evaluation regarding the posture. For example, if the mode is "lifting" and the flexion is not sufficient, an evaluation comment such as "not sitting down" may be included.

Body movement information is information related to the user's body movement. The body movement information includes a part to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment in association with a period on the time axis of the moving image. The list of posture values is a time-series posture value within a period. The evaluation comment is a comment related to the evaluation of movement. For example, if the mode is "lifting" and the knee extension is not smooth, it may include a rating comment such as "knee movement is not smooth".

The relationship information indicates information on one or more positional relationships between the tool and the body. The relationship information includes information on the relationship between information such as the position, orientation, and movement of the tool portion and information such as body parts, posture, and movement, in association with the time point on the time axis of the moving image. For example, it is possible to display the positional relationship between the tool position information and the body position information. That is, for example, a figure indicating a part (for example, a circle) may be superimposed on an image at a position indicated by tool position information, and a figure indicating a part (for example, a circle) may be displayed at a position indicated by body position information. can. It should be noted that the positions of a plurality of parts and parts may be included at one time point. It should be noted that the position information may not be included in the portion connecting the portion and the portion (for example, the tip portion of the bat and the center point of the portion where the user holds the bat). In this case, by connecting a pair of predetermined two parts and a figure (circle or the like) indicating the part with a line, a part connecting these two parts and the part can be expressed. The relationship information may be included for each frame constituting the video, for each key frame, for each checkpoint (details will be described later), or for each frame of any number. It may be included or it may be included for random time points. If my frame does not contain location information, it is possible to display a figure showing a part or part and a line connecting them based on the location information at the nearest past time point.

Checkpoint information is information indicating points (hereinafter referred to as checkpoints) for checking the orientation of the tool, the posture of the body, etc. in the movement of the tool used by the user or a series of movements of the user's body. Checkpoints include, for example, when the mode is "weightlifting", where the barbell reaches the highest position, when it reaches the lowest position, and when it is lifted. When the mode is "pitching", the place where the foot is raised, the place where the raised foot is lowered to move the weight, the place where the ball is released, and the like. The checkpoint information stores information indicating a checkpoint (hereinafter referred to as a checkpoint ID) in association with a time point on the time axis of the moving image. That is, it is possible to specify a frame (still image) in the moving image in which the checkpoint indicated by the checkpoint ID is displayed.

The evaluation display unit 114 displays evaluation information. For example, the evaluation display unit 114 superimposes the tool position, the tool orientation, the tool movement information, the body position, the posture, the body movement information, and the like included in the evaluation information on the moving image, and the tool part and the body. By displaying a figure representing a part (for example, the end of a tool, a circle representing the center of gravity of the body, a line connecting them, etc.), the movement of the tool part or body part can be superimposed on the video. .. Further, the evaluation display unit 114 may display, for example, a graph of changes over time such as the position of the tool part, the orientation of the tool, the movement of the tool, and the position, posture, and movement of the body part. can.

Further, the evaluation display unit 114 can display the evaluation rank and the evaluation comment together with the display of the moving image based on the tool orientation information and the tool movement information included in the evaluation information. For example, the evaluation display unit 114 reaches the tool orientation information when the playback time of the moving image is near the front and back of the time point included in the tool orientation information (for example, it can be an arbitrary length such as around 5 seconds). You can view the rating ranks and rating comments contained in. Further, the evaluation display unit 114 can display the evaluation rank and the evaluation comment included in the tool movement information when the reproduction time of the moving image comes within the period included in the tool movement information. Further, the evaluation display unit 114 can display the posture value included in the posture information. Further, the evaluation display unit 114 can display a graph of the time-series conversion of the tool orientation values based on the list of the tool orientation values included in the tool movement information.

Further, the evaluation display unit 114 can display the evaluation rank and the evaluation comment together with the display of the moving image based on the posture information and the movement information included in the evaluation information. For example, the evaluation display unit 114 is included in the posture information when the playback time of the moving image is near the front and back of the time when it is included in the posture information (for example, it can be an arbitrary length such as around 5 seconds). It is possible to display the evaluation rank and evaluation comment that have been given. Further, the evaluation display unit 114 can display the evaluation rank and the evaluation comment included in the motion information when the reproduction time of the moving image comes within the period included in the motion information. Further, the evaluation display unit 114 can display the posture value included in the posture information. Further, the evaluation display unit 114 can display a graph of the time-series conversion of the posture values based on the list of posture values included in the movement information. The evaluation rank and the evaluation comment may be displayed together with the evaluation rank and the evaluation comment displayed according to the display of the moving image based on the tool orientation information and the tool movement information described in the previous paragraph. ..

The checkpoint display unit 115 can extract and display a checkpoint image from a moving image. The checkpoint display unit 115 can read a frame corresponding to a time point included in the checkpoint information from the image data of the moving image stored in the image storage unit 131 and display it as a still image. Further, the checkpoint display unit 115 may, for example, extract and display only the body part from the read frame.

The improvement measure request transmission unit 116 transmits a request for acquiring improvement measures regarding how to handle the tool and physical exercise (hereinafter referred to as an improvement measure request) to the server device 20. FIG. 7 is a diagram showing a configuration example of an improvement measure request. As shown in the figure, the improvement measure request includes a user ID, a mode, a purpose, and the like. The purpose is for the user to make improvements. The purpose can be, for example, "to increase the speed of the ball", "to increase muscle strength", "to stabilize the lower body", and the like. The purpose shall also be selected from the predetermined options.

The improvement measure information receiving unit 117 receives information on the improvement measure (hereinafter referred to as improvement measure information) transmitted from the server device 20 in response to the improvement measure request. The improvement measure information receiving unit 117 stores the received improvement measure information in the improvement measure storage unit 133. FIG. 8 shows a configuration example of the improvement measure information. As shown in the figure, the improvement measure information includes the purpose, advice, and standard information. In the present embodiment, the advice is assumed to be a character string representing the improvement measure, but may be content that presents the improvement measure by an image, a moving image, or the like. Reference information includes suitable tool orientation and movement (position and orientation of each part, movement, speed, angle, etc.), body posture (position and orientation of each part, movement, speed, angle, etc.), and tool and body. (Position and orientation of each part and each part, movement, speed, angle, etc. and their relationship, etc.). The improvement measure information receiving unit 117 may receive the improvement measure information transmitted by the improvement measure information transmitting unit 216 based on the evaluation result and the reference value even if there is no improvement measure request.

The improvement measure information display unit 118 displays the improvement measure. The improvement measure information display unit 118 displays the advice included in the improvement measure information. Further, when the improvement measure information includes a suitable position or angle of the portion, each may be superimposed on the image information and displayed.

The reference value request transmission unit 119 transmits a request (hereinafter referred to as a reference value request) for acquiring a reference value or a candidate thereof targeted by the user to the server device 20. FIG. 8 is a diagram showing a configuration example of an improvement measure request. As shown in the figure, the improvement measure request includes a user ID, a mode, a purpose, the image information, the physical information, and the evaluation information. The purpose is for the user to make improvements. The purpose can be, for example, "to increase the speed of the ball", "to increase muscle strength", "to stabilize the lower body", and the like. The purpose shall also be selected from the predetermined options.

The reference value selection information transmission unit 120 transmits information selected from one or more of the reference values presented by the reference information determination unit 220 to the server device 20. The reference value selection information transmission unit 120 may transmit to the server device 20 an option requesting that another option be presented.

FIG. 10 is a diagram showing a hardware configuration example of the server device 20. The server device 20 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, such as a hard disk drive, a solid state drive, and a flash memory. The communication interface 204 is an interface for connecting to the communication network 30, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, and a wireless communication device for performing wireless communication. , USB (Universal Serial Bus) connector for serial communication, RS232C connector, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like for inputting data. The output device 206 is, for example, a display, a printer, a speaker, or the like that outputs data.

FIG. 11 is a diagram showing a software configuration example of the server device 20. As shown in the figure, the server device 20 includes an evaluation request receiving unit 211, an image analysis unit 212, an evaluation unit 213, an evaluation information transmitting unit 214, an improvement measure request receiving unit 215, an improvement measure information transmitting unit 216, and a group analysis unit. 217, group analysis information presentation unit 218, reference information request reception unit 219, reference information determination unit 220, image data storage unit 231, reference information storage unit 232, evaluation condition information storage unit 233, improvement measure information storage. Each storage unit includes a unit 234 and a group analysis information storage unit 235.

Each of the above functional units is realized by the CPU 201 included in the server device 20 reading a program stored in the storage device 203 into the memory 202 and executing the program, and each of the above storage units is the memory 202 included in the server device 20. And implemented as part of the storage area provided by the storage device 203.

The evaluation request receiving unit 211 receives the evaluation request transmitted from the user terminal 10. The evaluation request receiving unit 211 registers information including the image data included in the received evaluation request (hereinafter, referred to as image information) in the image data storage unit 231. FIG. 12 is a diagram showing a configuration example of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data in association with a user ID indicating a photographed user. The image data was included in the evaluation request.

The reference information storage unit 232 is related to the physical exercise using the tool, such as the tool position, the tool movement (direction and movement of the tool), the posture (position and angle of the part, etc.), and the relationship derived from the relationship between the tool and the body. Stores information including reference values (hereinafter referred to as reference information). FIG. 13 is a diagram showing a configuration example of reference information stored in the reference information storage unit 232. As shown in the figure, the reference information includes information on the absolute position of the tool part and how the tool part moved when performing physical exercise using the tool (movement speed and movement distance). , Direction of movement, etc.), reference information about the absolute position of the part or relative position to other parts or other reference objects (hereinafter referred to as position reference information), and for three parts including the joint part, two parts It includes, but is not limited to, reference information on the angle formed by a straight line connecting each and the joint part (hereinafter referred to as angle reference information), and information on the relationship between the tool part and the body part. In addition, reference values are prepared for each physical exercise (mode), and for each physical exercise (mode), each purpose, each feature of the physical information, each feature of the evaluation information, and a specific individual (constant). There are multiple standard values such as those that standardize the physical exercise of athletes, professional athletes, stepped athletes, experienced athletes, etc., but not limited to this). You may.

The tool position reference information includes the tool part and the reference position of the part in association with the mode and the checkpoint ID. There may be multiple parts. Regarding the position of the portion, the vertical position may be, for example, the height from the ground or the distance from any of the toes. Further, for example, when the mode is "weightlifting", it may be a distance from a body part or a line connecting parts such as a distance between a line connecting both shoulders and a shaft. The horizontal position of the site may be a distance from a predetermined reference object (for example, a mound plate, a mark on the floor, etc.) or a reference site such as a shoulder, chest, or foot. It is assumed that the position reference information is registered in advance.

The tool movement reference information includes the reference value of information such as the movement speed and distance of the tool part, the direction of movement at a certain point in time, and the trajectory of movement in a certain period in association with the mode and the checkpoint ID. And are included.

The position reference information includes a part and a reference position of the part in association with the mode and the checkpoint ID. There may be multiple parts. Regarding the position, the vertical position may be, for example, the height from the ground or the distance from any of the toes. The horizontal position may be a distance from a predetermined reference object (for example, a mound plate, a mark on the floor, etc.) or a reference site such as a shoulder, chest, or foot. It is assumed that the position reference information is registered in advance.

The angle reference information includes two parts (part 1 and part 2), one joint part, a straight line connecting the part 1 and the joint part, and the part 2 and the joint in association with the mode and the checkpoint ID. Includes a reference value for the angle between the straight line connecting the site.

The relationship standard information includes information related to the standard represented by the relationship between the tool part and the body part in association with the mode and the checkpoint ID. The relational reference information includes information obtained from the moving speed, the moving distance, the angle, and the like in one or more parts and parts in association with the mode and the checkpoint ID. For example, when the mode is batting, the relational reference information includes the moving speed of the tip of the bat at the time of meeting the ball and the angle between the bat and the dominant arm holding the bat.

The evaluation condition information storage unit 233 stores information for performing evaluation (hereinafter referred to as evaluation condition information). FIG. 14 is a diagram showing a configuration example of evaluation condition information stored in the evaluation condition information storage unit 233. The evaluation condition information includes categories, conditions, evaluation ranks, and comments. A category is a category of evaluation. The category can be, for example, "muscle strength", "ball speed", "control" and the like. The conditions are conditions for the position, orientation or movement of each part of the tool in the image (change in position in time series), and the position or movement of each part of the body (change in position in time series). For example, when analyzing the movement of weightlifting, the checkpoint at the moment of lifting the barbell, the conditions for the angle of the elbow, the speed of extending the arm, the movement of the shaft during the period of lifting and lowering the barbell, the speed of raising and lowering, etc. The condition for can be set in the evaluation condition information. Further, when analyzing the pitching form, it is possible to set conditions for the angle of the elbow, the line speed of the arm, and the like for the checkpoint for releasing the ball in the evaluation condition information. The evaluation rank is an evaluation value when the above conditions are satisfied. The comment is an explanation of the posture and movement of the body when the above conditions are satisfied.

The image analysis unit 212 (part / part identification unit) analyzes the image data. The image analysis unit 212 analyzes the image data, extracts the feature amount of each part of the tool and each part of the body, and specifies the position of each part and each part in the image. Further, the image analysis unit 212 analyzes the image data, extracts the feature amount of each part of the tool, and specifies the direction in which each part is facing. As the image analysis method by the image analysis unit 212, a general method is adopted, and detailed description thereof will be omitted here. The image analysis unit 212 may analyze the image data frame by frame or key frame, analyze the image data at each checkpoint, or analyze at random timing. May be good.

The image analysis unit 212 also compares the position of each part extracted from the image data with the position reference information stored in the reference information storage unit 232 for each checkpoint ID, and sets the nearest point as the checkpoint. Specify as a point in time.

The evaluation unit 213 evaluates the movement of the tool used by the user based on the image data. In the present embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including the condition that the position and the movement of each part of the tool specified from the image data are satisfied, and the condition is satisfied. If there is evaluation condition information, the evaluation rank and comments included in it are acquired. The evaluation unit 213 may evaluate the movement of the tool and count the number of physical exercises.

The evaluation unit 213 evaluates the movement of the user's body based on the image data. In the present embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including the condition that the position of each part and the movement of the part specified from the image data are satisfied, and the evaluation condition that the condition is satisfied is searched. If there is information, get the rating rank and comments included in it. The evaluation unit 213 may evaluate the movement of the body and count the number of physical exercises.

The evaluation unit 213 evaluates the tool used by the user and the movement of the body based on the image data. In the present embodiment, the evaluation unit 213 evaluates the evaluation condition information including the position of each part of the tool and each part of the body specified from the image data, and the condition that the movement or relationship between the part and the part is satisfied. It searches from the storage unit 233, and if there is evaluation condition information that satisfies the condition, the evaluation rank and the comment included in the evaluation condition information are acquired. The evaluation unit 213 may evaluate the tool and the movement of the body and count the number of physical exercises.

The evaluation information transmission unit 214 transmits the evaluation information to the user terminal 10. The evaluation information transmission unit 214 generates tool position information including a time point on the time axis of the moving image specified by the image analysis unit 212 and the position of each part of the tool. For the evaluation rank and comment acquired by the evaluation unit 213, if the position of the tool part satisfies the condition, the posture information including the time point, the part and the tool orientation value, and the evaluation rank and the comment is generated, and the movement of the part is generated. If (change in position in time series) is satisfied, tool movement information including a list of time points, parts and tool orientation values, and evaluation ranks and comments is generated. Further, the evaluation information transmission unit 214 generates checkpoint information including a time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmission unit 214 creates evaluation information including the generated tool position information, tool orientation information, tool movement information, and checkpoint information, and transmits the evaluation information to the user terminal 10. The evaluation unit 213 and the evaluation information transmission unit 214 may correspond to the comment output unit of the present invention.

The evaluation information transmission unit 214 transmits the evaluation information to the user terminal 10. The evaluation information transmission unit 214 generates position information including a time point on the time axis of the moving image specified by the image analysis unit 212 and the position of each part. For the evaluation rank and comment acquired by the evaluation unit 213, if the position of the part satisfies the condition, the posture information including the time point, the part and the posture value, and the evaluation rank and the comment is generated, and the movement of the part (time series). If the condition (change in position in) is satisfied, motion information including a list of time points, parts and posture values, and evaluation ranks and comments is generated. Further, the evaluation information transmission unit 214 generates checkpoint information including a time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmission unit 214 creates evaluation information including the generated position information, posture information, motion information, and checkpoint information, and transmits the evaluation information to the user terminal 10. The evaluation unit 213 and the evaluation information transmission unit 214 may correspond to the comment output unit of the present invention.

The improvement measure information storage unit 234 stores information related to the improvement measure (hereinafter referred to as improvement measure information). FIG. 15 is a diagram showing a configuration example of improvement measure information stored in the improvement measure information storage unit 234. As shown in the figure, the improvement measure information includes advice in association with the purpose, category, and condition. The condition may be a condition for the tool itself (weight of the barbell, etc.), how to use the tool, a physical condition (flexibility, etc.), a condition for the position and orientation of the part of the tool, a condition for movement, and the body. It may be a condition for the position and movement of the part of the body.

The improvement measure request receiving unit 215 receives the improvement measure request transmitted from the user terminal 10.

The improvement measure information transmission unit 216 includes the physical information of the user included in the evaluation request and each part specified by the image analysis unit 212 among the improvement measure information corresponding to the mode and the purpose included in the improvement measure request. Search for items that satisfy the conditions such as the position, orientation, and movement of each part. The improvement measure information transmission unit 216 acquires the advice of the searched improvement measure information, creates the improvement measure information for which the purpose and the advice are set, and responds the created improvement measure information to the user terminal 10. The improvement measure information transmission unit 216 also includes the position, direction, speed, angle, etc. of each part and each part included in the reference information in the improvement measure information and transmits the information. The improvement measure information transmission unit 216 may search for improvement measures based on the evaluation information and the reference value without the improvement measure request, and the improvement measure information transmission unit 216 transmits the improvement measures to the user terminal 10. You may.

FIG. 17 is a diagram showing an example of the flow of processing executed in the guidance support server of the present embodiment.

In the user terminal 10, the imaging unit 111 receives the input of the mode, images the body during the exercise of the user, and acquires the moving image data (S321). The evaluation request transmission unit 112 transmits an evaluation request including a user ID indicating a user, a received mode, physical information, and moving image data to the server device 20 (S322).

When the evaluation request receiving unit 211 receives the evaluation request in the server device 20, the image analysis unit 212 analyzes the moving image data to extract the feature amount (S323) and specifies the position of each part and each part (S324). Here, the image analysis unit 212 may specify the position on the image, or may use the body information to determine the actual size position (height from the ground, distance from the reference point such as the center of gravity of the body, etc.). You may try to specify. The evaluation unit 213 acquires an evaluation rank and a comment from the evaluation condition information in which the position or part of each part, the position or part of each part, or the movement of the part (change in time series of position) satisfies the condition (S325). The evaluation information transmission unit 214 creates evaluation information and transmits it to the user terminal 10 (S326).

In the user terminal 10, the evaluation display unit 114 displays the position, posture, movement, position, orientation, movement, etc. of the body on the moving image data based on the received evaluation information (S327). Further, in the user terminal 10, the evaluation display unit 114 may display the position (bone) of each unit indicating the posture of the body, and may display the evaluation rank and the comment (S327). Here, the evaluation display unit 114 may display a graph of the position, orientation, movement, etc. of the portion, and the time-series change of the position, movement, etc. of the portion. Further, the checkpoint display unit 115 may extract the checkpoint image from the moving image and display it. The improvement measure request transmission unit 116 transmits the improvement measure request to the server device 20 in response to an instruction from the user (S328).

In the server device 20, when the improvement measure request receiving unit 215 receives the improvement measure request transmitted from the user terminal 10, the improvement measure information transmitting unit 216 searches for the improvement measure information that satisfies the condition, and the searched improvement measures. The advice included in the information is acquired (S329), improvement measure information including the acquired advice is created, and the information is transmitted to the user terminal 10 (S330).

When the improvement measure information receiving unit 117 receives the improvement measure information in the user terminal 10, the improvement measure information display unit 118 displays the advice included in the received improvement measure information and superimposes the suitable usage of the tool on the video data. It can be displayed (S331).

When the improvement measure information receiving unit 117 receives the improvement measure information in the user terminal 10, the improvement measure information display unit 118 displays the advice included in the received improvement measure information and displays a suitable body posture in the form of a bone. It can be superimposed on the moving image data and displayed (S331).

As described above, according to the guidance support server of the present embodiment, it is possible to easily evaluate the physical exercise. In particular, regarding physical exercise related to sports, since it is possible to evaluate the positional relationship and movement of each part of the tool and each part of the body, it is easy to lead to concrete improvement efforts, and improvement in results is expected. In addition, since the guidance support server of the present embodiment also provides comments and advice, the user can easily grasp the current situation and improvement measures.

FIG. 18 is a diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. The figure illustrates the case where a moving image is captured in the weightlifting mode. As shown in the figure, the screen 41 displays a mark 411 indicating the position of the barbell shaft. The movement of the barbell shaft is indicated by line 412.

FIG. 19 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. The figure illustrates the case where a moving image is captured in the weightlifting mode. As a result of evaluation by the evaluation unit 213 in the weight lifting mode, for example, the inclination of the shaft of the barbell, the moving distance of the shaft, the moving speed (line 421), and the like are displayed. In the figure, the reference value (line 422) is displayed. The actual measurement result (line 423) may be displayed, and the difference from the reference value may be displayed by a numerical value or the like, or may be shown by a graph or the like. The user can consider the movement and posture to be corrected by referring to this.

FIG. 20 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. The figure illustrates the case where a moving image is captured in the weightlifting mode. As a result of the evaluation performed by the evaluation unit 213 in the weight lifting mode, for example, the evaluation result (line 432) is displayed together with the reference value (line 431) such as the angle of the joint of the body at the lowest point of the barbell. It is a figure. Further, the difference from the reference value may be displayed by a numerical value or the like, or may be shown by a graph or the like. Further, the evaluation result based on the relationship between the body and the tool may be displayed. The user can consider the movement and posture to be corrected by referring to this.

FIG. 21 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. The figure illustrates the case where a moving image is captured in the weightlifting mode. As a result of the evaluation performed by the evaluation unit 213 in the weight lifting mode, the line 441 displays a bone that is displayed by connecting a predetermined position of a tool part or a body part specified from the image with a line. The bone may be superimposed on the captured image and displayed. Further, the line 442 indicates the acceleration of each part of the body. Further, a count result such as how many times the weight is lifted may be displayed. Further, as shown by line 443, the evaluation result and the next training result according to the purpose and the like may be shown.

FIG. 22 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. The figure illustrates the case where a moving image is captured in the weightlifting mode. As a result of the evaluation performed by the evaluation unit 213 in the weight lifting mode, the evaluation (line 441) obtained by comparison with various reference values is displayed.

The group analysis unit 217 is a scene where multiple users use this system at the same time and a small number of supporters provide guidance remotely (for example, training guidance using online meeting tools, online communication tools, etc., remote rehabilitation support, etc.). It is supposed to be used in. In such a scene, there may be a plurality of users for each supporter. Although the terminal used by the supporter displays image information of multiple users in real time, detailed user status such as which user is training effectively and which user is tired. It is difficult to grasp, and as a result, the effect is diminished. In addition, it is difficult to communicate between the supporter and the user based on the content of the training conducted, and there is also a problem that the continuity is low. The group analysis unit 217 solves such a problem.

The group analysis unit 217 analyzes the evaluation information performed by the evaluation unit 213 on each user's image, and stores the analyzed information in the group analysis information storage unit 235. The group analysis unit 217 is close to (or a standard) the number of repetitions of the training performed and the reference value based on the evaluation information evaluated by the evaluation unit 213 regarding the physical exercise (including the exercise using the tool) performed by the user. Create a ranking of users whose physical exercises (or what they are doing may be analyzed in real time), such as the order (far from the value), were effective (or ineffective). In addition, each user may be categorized into a high-ranking group, a low-ranking group, and the like. In addition, the user's fatigue level increases as the position, movement speed, movement distance, joint angle, etc. of the tool part and body part move away from the reference values while continuing physical exercise. It may be determined that the patient is present, and the degree of fatigue may be estimated based on the degree of deviation from the reference value. Furthermore, the time to complete a predetermined number of iterations may be measured for each set, and the degree of fatigue may be determined by increasing the completion time, or the degree of fatigue may be estimated by comparing with the increased time or the completion time of the previous set. You may.

Further, the group analysis unit 217 may evaluate the time until the physical exercise actually starts, such as a signal for the supporter to start or end the exercise, and estimate the degree of concentration of the user. Further, the group analysis unit 217 may calculate a deviation value from the evaluation information among users who are participating in training or the like at the same time, or may create a ranking based on the deviation value. Further, the group analysis unit 217 may calculate a deviation value from the evaluation information among users who have performed the same physical exercise at different timings, or may create a ranking based on the deviation value. Further, from the ranking, the degree of fatigue, the degree of concentration, etc., a user who needs guidance or a user who is considered to be more effective in terms of continuity by making a direct comment may be specified. Further, the group analysis unit 217 may compare the evaluation information when the user has previously performed the same physical exercise with the evaluation information evaluated from the image of the newly performed physical exercise.

The group analysis information storage unit 235 stores the group analysis information performed by the group analysis unit 217. FIG. 16 is a diagram showing a configuration example of group analysis information stored in the group analysis information storage unit 235. Group analysis information largely includes ranking information, status information, and caution information. The ranking information includes ranking based on the evaluation information, ranking, and category information such as whether the ranking is included in the upper group or the lower group. The situation information includes the degree of fatigue and the degree of concentration. Attention information includes information such as users who need to be instructed in particular.

The group analysis information presentation unit 218 presents the group analysis information stored in the group analysis information storage unit 235 to the user terminal 10 and the supporter terminal 40. The group analysis information presentation unit 218 is a group analysis information presenting unit 218 between users who are simultaneously performing physical exercises with respect to the user terminal 10, and between users including users who are performing similar physical exercises at different timings. Presents categories such as rankings, deviation values, high ranks, and low ranks. Further, the group analysis information presentation unit 218 may also present the information stored in the evaluation information storage unit 132. Further, the group analysis information presentation unit 218 may present the ranking and deviation value information of each user to the supporter terminal 40, color the users at the top (or bottom) of the ranking, and so on. It may be presented in an easy-to-see manner (such as displaying the screen on which the user is displayed in a large size or presenting it at the top).

FIG. 23 is a diagram showing an example of a screen created by presenting the group analysis information presentation unit 218 to the user terminal 10. The figure illustrates the case where a moving image is captured in squat mode. As shown in the figure, on the screen 46, the rank within the own group is displayed by a line 461. The line 462 shows the number of exercises performed. Line 463 displays a real-time video (or recorded video) of the trainer as a model.

FIG. 24 is a diagram showing an example of a screen created by the group analysis information presentation unit 218 presenting to the supporter terminal 40. On the supporter terminal 40, the evaluation result of each user, the group analysis information, and the like are displayed together with the moving image of each user.

FIG. 25 is a diagram showing an example of a screen created by the group analysis information presentation unit 218 presenting to the supporter terminal 40. Images of users are arranged on the screen 47, and users who need more guidance (including support) in the group are displayed in an easy-to-understand manner by evaluation and group analysis. On the screen 47, as an example, a mark is displayed on the upper right of the screen showing a user who needs guidance.

FIG. 26 is a diagram showing an example of a screen created by the group analysis information presentation unit 218 presenting to the supporter terminal 40. The screen 48 is a screen showing the details of each user, and includes evaluation results (bones, posture values, tool orientation values, number of repetitions of physical exercise, etc.), group analysis results (ranking within the group, changes thereof, etc.), and the relevant screens. A function for viewing a recorded image (or a part of a moving image such as a few seconds before or after the time when the evaluation is lowered) such as an advice transmitted by the improvement measure information transmitting unit 216 is displayed for the user. Further, the screen may be a combination of the screen 47 and the screen 48.

The reference information request receiving unit 219 receives the reference information request transmitted from the user terminal 10. The reference information request receiving unit 219 registers information including image data included in the received reference information request (hereinafter, referred to as image information) in the image data storage unit 231. FIG. 12 is a diagram showing a configuration example of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data in association with a user ID indicating a photographed user. The image data was included in the reference information request. It is also assumed that the reference information request does not include image data.

The reference information determination unit 220 searches for reference information satisfying the user's needs based on the mode, purpose, image information included in the reference information request, the physical information, the evaluation information, and the like, and causes the user terminal 10 to search for the reference information. Send. Further, the reference information determination unit 220 receives information selected from any of the reference information transmitted by the reference value selection information transmission unit 120, and even if the image analysis unit 212 performs analysis based on the selected reference information. good. Further, the reference information determination unit 220 may transmit candidate reference information to the user based on the purpose, physical information, basic information, etc., even if there is no reference information request. Further, the reference information determination unit 220 may present an option for the user to present another reference value together with the option of the reference value, and in that case, "increase the strength" or "long time". You may present a reference option for selecting a new reference value, such as "continue."

The reference information determination unit 220 is stored in the reference information storage unit as reference information when, for example, the weight lifting mode is included as the reference information request and the purpose is "muscle hypertrophy of the upper part of the pectoralis major muscle". Select the standard information that matches the training of the upper part of the pectoralis major muscle from the standard information of. In this case, for example, reference information 1 "for training the upper part of the pectoralis major muscle: pushing the arm diagonally upward", reference information 2 "for training inside the pectoralis major muscle: closing the arm in front", and reference information 3 "lower part of the pectoralis major muscle". : The reference information such as "push the arm diagonally downward" is stored in the reference information storage unit 232. The reference information determination unit 220 transmits reference information according to the purpose to the user terminal 10, and when the user selects the reference information, the reference information determination unit 220 accepts the selection and the evaluation unit 213 uses the reference information for evaluation. It becomes.

Further, the reference information determination unit 220 is stored in the reference information storage unit as reference information when, for example, the weight lifting mode is included as the reference information request and the target is “muscle hypertrophy of the upper part of the pectoralis major muscle”. , The reference information suitable for the training of the upper part of the pectoralis major muscle is selected from a plurality of reference information, but the reference information may be further selected based on the physical information. In this case, for example, reference information 1 "for training the upper part of the pectoralis major muscle: pushing the arm diagonally upward", reference information 2 "for training inside the pectoralis major muscle: closing the arm in front", reference information 3 "lower part of the pectoralis major muscle". Standard information such as "push the arm diagonally downward" is stored, and if the inside of the pectoralis major muscle is not trained to some extent between each standard information, the effect is weak even if the upper part of the pectoralis major muscle is trained. , Or when there is a risk of injury, the reference information determination unit 220 further uses, for example, the user's physical information at that time (for example, muscle strength, muscle strength inside the pectoralis major muscle, etc.). First, reference information 2 “for training inside the pectoralis major muscle: close the arm in front” may be transmitted to the user terminal 10. In this case, further, send training steps and advice such as "First train the inside of the pectoralis major muscle (that is, reference information 2), and then train the upper part of the pectoralis major muscle (that is, reference information 1)". You may.

Further, the reference information determination unit 220 is stored in the reference information storage unit as reference information when, for example, the weight lifting mode is included as the reference information request and "muscle hypertrophy of the upper part of the pectoralis major muscle" is included as the purpose. , The reference information suitable for the training of the upper part of the pectoralis major muscle is selected from a plurality of reference information, but the reference information may be further selected based on the evaluation information. In this case, for example, reference information 1 "for training the upper part of the pectoralis major muscle: pushing the arm diagonally upward", reference information 2 "for training inside the pectoralis major muscle: closing the arm in front", and reference information 3 "lower part of the pectoralis major muscle". Standard information such as "push the arm diagonally downward" is stored, and it is difficult to train the upper part of the pectoralis major muscle between each standard information, so first of all, the movement to train the inside of the pectoralis major muscle When there is a relationship such as that it is better to practice, the reference information determination unit 220 is further included in, for example, the user's evaluation information (which can be regarded as a skill) at that time (for this reason, the reference information). The image information included in the request may be analyzed by the image analysis unit 212 and evaluated by the evaluation unit 213.) Difficulty level first based on information such as tool movement information (for example, whether the barbell can be raised and lowered smoothly). Low reference information 2 “For training inside the pectoralis major muscle: Close the arm in front” may be transmitted to the user terminal 10. In this case, further training steps and advice such as "First practice the movement to train the inside of the pectoralis major muscle (that is, reference information 2), and then train the upper part of the pectoralis major muscle (that is, reference information 1)". You may also send it together.

Further, the reference information determination unit 220 is stored in the reference information storage unit as reference information when, for example, the weight lifting mode is included as the reference information request and "muscle hypertrophy of the upper part of the pectoralis major muscle" is included as the purpose. , The reference information suitable for the training of the upper part of the pectoralis major muscle is selected from a plurality of reference information, and the reference information may be further selected based on the past training history information included in the physical information. In this case, for example, reference information 1 "for training the upper part of the pectoralis major muscle: pushing the arm diagonally upward", reference information 2 "for training inside the pectoralis major muscle: closing the arm in front", reference information 3 "lower part of the pectoralis major muscle". Standard information such as "push the arm diagonally downward" is stored, and if the inside of the pectoralis major muscle is not trained to some extent between each standard information, the effect is weak even if the upper part of the pectoralis major muscle is trained. In addition, if there is a relationship such as, or there is a risk of injury, the reference information determination unit 220 may further use, for example, the user's past training history information (for example, the number of times the pectoralis major muscle medial training has been performed, or already. Improvement measure When the training history more than a certain number of times is confirmed based on the number of times the information transmission unit 216 has transmitted to the user terminal 10 so that the user can perform medial training of the pectoralis major muscle, etc., the reference information 1 " "For pectoralis major upper training: pushing the arm diagonally upward" may be transmitted to the user terminal 10. In this case, further training steps and advice such as "Because the inside of the pectoralis major muscle (that is, reference information 2) has already been sufficiently trained, let's train the upper part of the pectoralis major muscle (that is, reference information 1)". May be sent together.

Further, the reference information determination unit 220 is stored in the reference information storage unit as reference information when, for example, the weight lifting mode is included as the reference information request and the purpose is "muscle hypertrophy of the upper part of the pectoralis major muscle". , Select the reference information according to the training of the upper part of the pectoralis major muscle from a plurality of reference information, but send the body exercise menu or the like to the user terminal 10 so as to perform one or more physical exercises, and the user takes an image. Criteria information may be selected based on the evaluation of physical exercise. In this case, for example, reference information 1 "for pectoralis major upper training A: 20 kg barbell pushes the arm diagonally upward", reference information 2 "for pectoralis major upper training B: 40 kg barbell, the arm is slanted". Criteria information such as "push up" and reference information 3 "pectoral muscle training basics: push-up 30 times" are stored, and if the pectoral muscles are trained to some extent between each reference information, the next step is taken. Further, when there is a relationship such as advancing, the reference information determination unit 220 first transmits "suspension" to the user terminal 10, causes the user to image an image of physical exercise, and the image analysis unit receives the image. , Count the number of times, etc. When a certain standard is exceeded, the standard information determination unit 220 transmits the standard information 1 "A for training the upper part of the pectoralis major muscle: push the arm diagonally upward with a 20 kg barbell" to the user terminal 10 as the next step. .. Further, if the standard is not exceeded, the standard information 3 “pectoral muscle training basics: push-up 30 times” for clearing the standard may be transmitted to the user terminal 10. In this case, further training steps such as "Let's do basic pectoral muscle training with push-ups (reference information 3) first to train the upper part of the pectoralis major muscle (that is, reference information 1 or 2)" You may also send advice and so on.

Further, the reference information determination unit 220 presents the mode, purpose, and image information included in the reference information request together with the recommendation level of the reference information to be presented to the user based on the physical information, the evaluation information, and the like. You may. In this case, the degree of recommendation may be indicated by a numerical value such as a numerical value out of 100 points or a ratio in 100%, or may be indicated by a degree such as 3 out of 5 levels, or ○, ×, △ It may be indicated by a mark such as, thumb-up, thumb-down, etc., or a symbol or color indicating an impression of the degree of conformity such as color, or the number of users who have selected the reference value in the past, the user concerned. The evaluation for the reference value obtained from the above may also be presented.

Further, the reference information determination unit 220 may derive the recommendation level of the reference information presented to the user from the degree of matching between the purpose and the purpose set in the reference value, and the relationship between the evaluation information and the reference value ( As evaluation information, if the swing speed of the bat is fast, the degree of conformity may be high if the reference value is for a person with a fast swing speed of the bat, etc.), or the relationship between the physical information and the reference value ( In the case of left-handed as physical information, if the batting standard value for the left batter is high, the degree of conformity is high. It may be derived from (Naru, etc.), or these may be used alone or in combination to derive the degree of recommendation.

Although the present embodiment has been described above, the above embodiment is for facilitating the understanding of the present invention, and is not for limiting the interpretation of the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention also includes an equivalent thereof.

For example, in the present embodiment, the image is analyzed by the server device 20, but the present invention is not limited to this, and the image is analyzed by the user terminal 10 to specify the positional relationship of each part and each part. May be good.

Further, in the present embodiment, the positions of the parts and parts are assumed to be positions on the two-dimensional image, but the position is not limited to this and may be a three-dimensional position. For example, when the user terminal 10 includes a depth camera in addition to the camera 106, the three-dimensional positions of parts and parts are specified based on the image from the camera 106 and the depth map from the depth camera. be able to. Further, for example, a three-dimensional image may be estimated to specify a three-dimensional position of a part or a part. It is also possible to provide a depth camera instead of the camera 106 and specify the three-dimensional position only from the depth map from the depth camera. In this case, the depth map may be transmitted from the user terminal 10 together with the image data or instead of the image data to the server device 20, and the image analysis unit 212 of the server device 20 may analyze the three-dimensional position. can.

Further, in the present embodiment, an image of the body of the user who is exercising using the tool is transmitted from the user terminal 10 to the server device 20, but the present invention is not limited to this, and the user terminal 10 is characterized by the image. The amount may be extracted and the feature amount may be transmitted to the server device 20, or the user terminal 10 estimates the part of the tool and the part of the body based on the feature amount, and the absolute position of the part and the part ( It may be the position on the XY coordinates of the image, it may be the actual size distance from the reference position (for example, the ground, toes, head, center of gravity of the body, etc.), or it may be the position in any other coordinate system. It is also possible to acquire the relative positional relationships between the plurality of parts and the plurality of parts for a plurality of parts, and transmit these absolute positions and relative positional relationships to the server device 20. You may.

Further, in the present embodiment, the content prepared on the server device 20 side is provided as the improvement measure information, but the present invention is not limited to this, and for example, the reference value is included to be correct based on the reference value. Marks and bones that are movements and postures (positions and orientations of each part, positions and angles of each part, etc.) may be superimposed and displayed on a moving image or a still image extracted from the moving image. This makes it possible to easily grasp what kind of movement and posture should be taken.

Further, in the present embodiment, the part and orientation of the tool, the position or movement of the body part, etc. (position over time) are evaluated, but the present invention is not limited to this, and the position of the tool worn by the user is not limited to this. It may be specified and evaluated.

Further, in the present embodiment, contents such as advice are provided for improvement measures, but for example, tool recommendations may be performed. In this case, the server device 20 stores the reference value of the tool and the size (length, etc.) of the tool in association with the physical information (height, weight, etc.) of the user, and the user uses the image data. The feature amount of the tool is extracted, the shape of the tool is specified, the size of the tool is estimated based on the shape and the size of the user included in the physical information (for example, height, etc.), and the estimated size of the tool is used. If the difference from the reference value is equal to or greater than a predetermined threshold value, a tool of the size of the reference value can be recommended. Furthermore, even if you recommend a tool that suits your purpose from information such as the conditions for the tool itself (barbell weight, etc.), how to use the tool, physical conditions (flexibility, etc.), the position and orientation of the tool part, and movement. good.

Further, in the present embodiment, contents such as advice are provided for improvement measures, but for example, the physical exercise being performed may be interrupted. In this case, the server device 20 stores the reference value at which the physical exercise should be interrupted in association with the physical information (purpose, height, weight, etc.) of the user, and the physical exercise performed by the user from the image data. When the number of times and speed (for example, the speed of lifting the barbell is extremely slowed down, or the number of times the barbell is lifted too many times) deviates from the standard value, the physical exercise is interrupted. In this case, a comment may be given to the user terminal 10 to cancel, the user may be notified by changing the display such as turning off the screen, or a sound such as an alert sound may be emitted. Alternatively, the user may be notified by vibration.

Further, in the present embodiment, contents such as advice are provided for improvement measures, but for example, physical exercise for determination of illness and its improvement may be presented. In this case, the server device 20 extracts a candidate for a disease presumed to be developed by the user from the symptom input by the user in the physical information and the evaluation information, and presents a screening test for narrowing down. When the user conducts a screening test and narrows down the name of the disease, he / she consults a doctor, exercises for improvement, tools for physical exercise, and recommends items such as meals. May be good.

Further, by estimating the position of the part of the tool, the server device 20 can estimate the speed, acceleration, moving distance, trajectory, etc. of the tool. Further, the server device 20 can estimate the number of patterns as the number of operations using the tool by extracting the pattern of the change in the position of the tool in time series.

Further, in the present embodiment, the movement is evaluated, but the present invention is not limited to this, and when a certain posture or movement is detected, a problem for the movement may be proposed. In this case, the server device 20 may store the task in place of the evaluation comment and output the task in association with one or a series of postures or movements.

Further, in the present embodiment, the exercise is evaluated, but the exercise is not limited to this, and training, rehabilitation, and the like to be performed when the movement of a certain tool, the direction, the posture, or the movement of a part of the body are detected. You may present the content to improve the physical exercise according to the purpose such as performance, stretching, strength training, posture, etc., which are the preparation stages thereof. In this case, the server device 20 correlates with the movement of one or a series of tool parts, the orientation of the tool parts, the posture of the body, or the movement of the body parts, and instead of the evaluation comment, the content of the training or the like is performed. It may be stored and the contents may be output.

Further, in the present embodiment, the motion is evaluated, but the present invention is not limited to this, and the motion performed by the user can be automatically detected. In this case, the server device 20 stores the position and posture (position of each part of the body) of each part of the tool that performs a predetermined operation such as a shoot or a pass as reference information, and analyzes the tool from the image. By comparing the position of a part or a part of the body with the reference information, it is possible to identify the action performed by the user in the image.

Further, in the present embodiment, the image captured in the past is analyzed to evaluate the motion, but the present invention is not limited to this, and when the analysis process is performed in real time and a predetermined motion is detected, the next action is taken. You may try to recommend the tactics that should be done. In this case, the tactics may be stored in place of the evaluation comment in association with the posture or movement, and the tactics may be output in real time.

Further, in the present embodiment, the supporter terminal 40 may also serve as the image pickup terminal 50 or may also serve as the output terminal 60. In this case, the supporter terminal 40 has a shape such as a glasses type, a contact lens type, a hat type, an HMD (head-mounted display), and the range close to the supporter's field of view due to the image pickup function provided in the supporter terminal 40. Is imaged, and the captured image is sent to the server device 20 via the communication network 30. The evaluation unit 213 and the group analysis unit 217 process the image, and the result is sent to the supporter terminal 40 or the user terminal 10 via the communication network 30. As shown in FIG. 27, the supporter terminal 40 uses, for example, a virtual image projection method, a retinal projection method, other methods, and brain activity such as an electroencephalogram to transfer characters, images, images, and the like to the brain. Information is output to the supporter or the user via an interface such as BMI (Brain Machine Interface) that is input by direct stimulation without going through the sensory organs. By performing this communication and processing at high speed, when the supporter sees the user's physical movement, it is possible to confirm the evaluation of the user's physical movement and the evaluation within the group in almost real time. In addition, the user can also check the evaluation of his / her physical exercise and the evaluation within the group in almost real time. From the supporter's field of view, the user's physical movement is seen as a real image, and the result of processing by the evaluation unit 213 and the group analysis unit 217 is viewed by the supporter with the naked eye through the supporter terminal 40. The supporter may be able to see it by superimposing it on the field of view. Further, when the supporter terminal 40 is an HMD or the like, the image captured by the supporter terminal 40 may be presented to the supporter in a form in which the results of processing by the evaluation unit 213 and the group analysis unit 217 are superimposed. ..

Further, in the present embodiment, the user terminal 10 may also serve as the image pickup terminal 50 or may also serve as the output terminal 60. In this case, the user terminal 10 is in the shape of, for example, a spectacle type, a contact lens type, a hat type, an HMD (head mounted display), or the like, and an image pickup function provided in the user terminal 10 captures a range close to the user's field of view. The captured image is sent to the server device 20 via the communication network 30. The evaluation unit 213 and the group analysis unit 217 process the image, and the result is sent to the user terminal 10 or the supporter terminal 40 via the communication network 30. As shown in FIG. 25, the supporter terminal 40 uses, for example, a virtual image projection method, a retinal projection method, other methods, and brain activity such as an electroencephalogram to transfer characters, images, images, and the like to the brain. Information is output to the user or supporter via an interface such as BMI (Brain Machine Interface) that is input by direct stimulation without going through the sensory organs. By performing this communication and processing at high speed, when the user sees his / her body movement through a mirror or the like, he / she can confirm the evaluation of his / her body movement and the evaluation in the group in almost real time.

Buying and selling of reference information Further, in the present embodiment, a plurality of reference values may exist, and the user may select the reference value for a fee.

Buying and selling of reference information Further, in the present embodiment, the reference value may include one created based on physical exercise (including one performed by using a tool) of a skilled person such as a professional athlete.

Buying and selling of reference information Further, in the present embodiment, the user terminal 10, the supporter terminal 40, the image pickup terminal 50, and the output terminal 60 may be wearable terminals such as eyeglasses, and the results of imaging, analysis, and evaluation by the wearable terminal are performed. May be output to a wearable terminal, but a non-wearable user terminal 10 such as a mobile terminal, a supporter terminal 40, an image pickup terminal 50, and an output terminal 60 may exist at the same time, and the non-wearable terminal may also be present. The result of analysis and evaluation may be displayed.

Further, in the present embodiment, the execution of a predetermined function and the storage of information are performed by the user terminal 10 or the server device 20, but the present invention is not limited to this, and the execution of the function and the storage of information by either device are not limited to this. You may do memory. Alternatively, the functional unit and the storage unit may be distributed and provided in a form different from the present embodiment.

10 User terminal 20 Server device 30 Communication network 40 Supporter terminal 50 Imaging terminal 60 Output terminal 101 CPU
102 Memory 103 Storage device 104 Communication interface 105 Output device 106 Camera input device 111 Imaging unit 112 Evaluation request transmission unit 113 Evaluation information reception unit 114 Evaluation display unit 115 Checkpoint display unit 116 Improvement measure request transmission unit 117 Improvement measure information reception unit 118 Improvement measure display unit 119 Reference value request transmission unit 120 Reference value selection information transmission unit 130 Physical information storage unit 131 Image storage unit 132 Evaluation information storage unit 133 Improvement measure storage unit 201 CPU
202 Memory 203 Storage device 204 Communication interface 205 Output device 206 Camera input device 211 Evaluation request reception unit 212 Image analysis unit 213 Evaluation unit 214 Evaluation information transmission unit 215 Improvement measure request reception unit 216 Improvement measure information transmission unit 217 Group analysis unit 218 Group analysis presentation unit 219 Reference value request reception unit 220 Reference value determination unit 231 Image data storage unit 232 Reference information storage unit 233 Evaluation condition information storage unit 234 Improvement measure Information storage unit 235 Group analysis information storage unit

Claims

It is a guidance support system that supports the guidance given by the supporter to the user who performs physical exercise.
A reference value storage unit that stores a reference value related to the position of at least one body part or tool part, and a reference value storage unit.
A part specifying part that identifies the part or the part by analyzing an image of the user's physical exercise,
An evaluation unit that determines the evaluation value of the physical exercise by comparing the position of the portion or the portion in the image and the reference value.
Equipped with
An imaging function that captures an image of the user's physical exercise,
A supporter terminal having a display function for displaying the evaluation value, and
Guidance support system characterized by being equipped with.
The supporter terminal must be a wearable computer.
The guidance support system according to claim 1, wherein the guidance support system is characterized by the above.
There are multiple reference values,
A reference value determining unit that presents one or more reference values to the user based on at least one of the user's purpose, physical information, and the evaluation value.
The guidance support system according to claim 1 or 2, wherein the system is provided with.
The reference value selection information transmission unit selected by the user from the reference value candidates, and
The guidance support system according to claim 1 to 3, further comprising.
It is a guidance support method that supports the guidance given by the supporter to the user who performs physical exercise.
A reference value storage step that stores a reference value relating to the position of at least one body part or tool part,
A site part identification step for identifying the part or the part by analyzing an image of the user's physical exercise,
An evaluation step of comparing the position of the part or the part in the image and the reference value to determine the evaluation value of the physical exercise, and the evaluation step.
Equipped with
An imaging function that captures an image of the user's physical exercise,
A supporter terminal having a display function for displaying the evaluation value, and
Guidance support method characterized by providing.