JP2022129615A - Exercise support system and exercise support method - Google Patents

Abstract

To provide an exercise support system capable of supporting exercise more effectively than ever before.SOLUTION: An exercise support system 10 is composed of a home device 20, a facility device 30, an exercise support device 40, etc. In the home device 20, a first control part 21d displays a support video for supporting exercise of a user generated on the basis of a detection result by a camera 21a in a display unit 21b, etc. in conjunction with the display of a training video. In the facility device 30, a second control part 31a calculates a difference between movements of the user detected by cameras 36a-36d and movements indicated by the training video, and displays a support video for supporting the exercise of the user generated on the basis of the difference in a projector 34 in conjunction with the display of the training video. Difference data indicating the difference is feedbacked to the home device 20 from the facility device 30 through the exercise support device 40.SELECTED DRAWING: Figure 2

Description

The present invention relates to an exercise support system and an exercise support method that support exercise training.

Due to the recent spread of infectious diseases, it has become difficult to go to fitness gyms as before in order to avoid the three Cs and infections. ing. As one of such solutions, a technology has been proposed that enables users to visually compare a desired model training video and the user's movement on a display using a smartphone (for example, Patent Document 1). reference).

JP 2020-195573 A

However, according to the technique disclosed in Patent Literature 1, although the difference between the model training video and the user's movement can be recognized to some extent, objective evaluation is possible, but there is a problem that the accuracy of the difference is low. In the technique disclosed in Patent Document 1, a user's motion is detected by a camera mounted on a smartphone. This is because it is difficult to detect with accuracy. Therefore, the technique disclosed in Patent Document 1 is not sufficient to support effective exercise or encourage users to make exercise a habit.

Therefore, the present invention has been made in view of such a situation, and is an exercise device capable of supporting exercise more effectively than in the past and encouraging the user to make exercise a habit and improve continuity. An object is to provide a support system and an exercise support method.

In order to achieve the above object, an exercise support system according to one aspect of the present invention includes an in-home device that supports exercise training by a user at home, and a facility device that supports exercise training by the user within a facility. an exercise support device connected to the in-home device and the facility device via a communication path; a first display unit for displaying an image; a first control unit for controlling the training image held in the storage unit to be displayed on the first display unit; a first detection unit that detects the movement of the user exercising in accordance with the training video displayed on the first display unit; The facility device performs control to display the support video for supporting the user's exercise, which is generated based on the above, on the first display unit in conjunction with the display of the training video, and the facility device displays the video. a display unit; a second control unit that controls display of the training image held in the storage unit on the second display unit; a second detection unit that detects movement with higher accuracy than the first detection unit; and the second control unit further controls the movement of the user detected by the second detection unit and the training image. and control to display on the second display section an assistance image for assisting the user's exercise, which is generated based on the calculated difference, in conjunction with the display of the training image. and the exercise support device acquires difference data indicating the difference calculated by the second control unit, generates a model training video based on the acquired difference data, and the in-home device, A training video generated by the exercise support device is obtained through the communication channel and stored in the storage unit.

In order to achieve the above object, an exercise support method according to one aspect of the present invention includes an in-home device for supporting exercise training by a user at home, and a facility device for supporting exercise training by the user in a facility. and an exercise support device connected to at least the in-home device via a communication path, and provided in any one of the in-home device, the facility device, and the exercise support device, and holding a training video serving as a model. a step of causing the in-home apparatus to display a training image held in the storage unit on a first display unit; detecting the movement of the user exercising in accordance with the training video displayed on the first display unit using a; and generating the in-home device based on the detection result by the first detection unit a step of displaying a support video for supporting the exercise of the user on the first display unit in conjunction with the display of the training video; a step of causing the facility device to display, using a second detection unit, the movement of the user exercising in accordance with the training video displayed on the second display unit at a higher level than the first detection unit; the step of detecting with accuracy; and the facility device calculating a difference between the motion of the user detected by the second detection unit and the motion indicated by the training video, and generating the training video based on the calculated difference. a step of displaying a support video for supporting a user's exercise on the second display unit in conjunction with the display of the training video; a step of acquiring and generating a model training video based on the acquired difference data; and storing in a storage unit.

ADVANTAGE OF THE INVENTION The present invention provides an exercise support system and an exercise support method that can more effectively support exercise than in the past, and encourage a user to make exercise a habit to improve continuity.

FIG. 1 is an overall configuration diagram of an exercise support system according to an embodiment. FIG. 2 is a block diagram showing the detailed configuration of the exercise support system shown in FIG. 1. As shown in FIG. FIG. 3 is a flow chart showing the operation of the in-home device in the exercise support system according to the embodiment. FIG. 4 is a diagram showing how the in-home device operates in the exercise support system according to the embodiment. FIG. 5 is a flow chart showing the operation of the facility device in the exercise support system according to the embodiment. FIG. 6 is a diagram showing how the facility device operates in the exercise support system according to the embodiment. FIG. 7A is a diagram showing an example arrangement of cameras, floor load cells, and wall load cells at a user training location. FIG. 7B is a diagram showing an example of user movement detection by the second control unit using the camera, the floor surface load cell, and the wall surface load cell. FIG. 7C shows an example of detection of the center-of-gravity position of the user by the floor surface load cell. FIG. 8 is a flow chart showing details of steps S23a and S23b in FIG. FIG. 9 is a diagram showing a display example of a support image and a training image by the projector of the facility device according to the embodiment. FIG. 10 is a flow chart showing the movement of the facility device that changes the training video according to the state of the user who is exercising. FIG. 11 is a block diagram showing a configuration of an exercise support device in the exercise support system according to the modified example of the embodiment. 12A is a communication sequence diagram showing exchanges between an in-home device and an exercise support device in the exercise support system according to the modification of the embodiment; FIG. 12B is a communication sequence diagram showing exchanges between the facility device and the exercise support device in the exercise support system according to the modification of the embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that each of the embodiments described below is a specific example of the present invention. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present invention. Also, each figure is not necessarily strictly illustrated. In each figure, substantially the same configurations are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.

FIG. 1 is an overall configuration diagram of an exercise support system 10 according to an embodiment. The exercise assistance system 10 is a system that assists a user in exercise training, and includes an in-home device 20 that assists the user in in-home exercise training, a fitness gym (office, commercial facility, educational institution, medical institution, nursing care facility, etc.). , attractions, etc.), and is connected to the home device 20 and the facility device 30 via a communication path such as the Internet. and an exercise support device 40 serving as an exercise server.

The facility apparatus 30 can precisely and accurately support the user's training by sensing the motion of the user's exercise with higher accuracy than the in-home apparatus 20 . In this exercise support system 10, the user's movement that is highly accurately sensed by the facility device 30 is reflected, that is, the user's individuality is reflected, and the exercise support system 10 provides a model training video that effectively supports the user's exercise. It is characterized in that it is generated by the support device 40 and fed back to the in-home device 20 .

FIG. 2 is a block diagram showing the detailed configuration of the exercise support system 10 shown in FIG. 1. As shown in FIG.

The in-home device 20 includes a terminal device 21 and peripheral devices controlled by the terminal device 21 (lighting device 22, sound device 23, projector 24, air conditioner 25).

The terminal device 21 is, for example, a mobile terminal such as a smart phone or a tablet, and includes a camera 21a, a display 21b, a first storage section 21c, and a first control section 21d.

The display 21b is an example of a first display unit for displaying a training image serving as a model and a support image for assisting the user's exercise, and is, for example, a liquid crystal display.

The first storage unit 21c is a storage unit for holding training images that serve as models, and is, for example, a semiconductor non-volatile memory.

The camera 21a is an example of a first detection unit that detects the movement of the user exercising in accordance with the training video displayed on the display 21b, and is, for example, a digital imaging device.

The first control unit 21d controls the display 21b and/or the projector 24 to display the training video held in the first storage unit 21c, and supports the user's exercise based on the detection result of the camera 21a. and control to display the generated support image on the display 21b and/or the projector 24 in conjunction with the display of the training image. The first control unit 21d includes a memory for storing programs and the like, a CPU for executing the programs, an input/output interface connected to peripheral components, a communication unit 21e, and the like. The communication unit 21e communicates with peripheral devices (the lighting device 22, the sound device 23, the projector 24, the air conditioner 25), the exercise support device 40, and the facility device 30, which are external devices, by wireless or wired communication. A communication interface, for example a communication card for WiFi (registered trademark) or a mobile phone network.

Although the first storage unit 21c has been described as being provided in the in-home device 20, it is provided in the third storage unit 47 of the exercise support device 40 or the second storage unit 31c of the facility device 30, and the A training video may be transmitted to the terminal device 21 and displayed on the display 21b and/or the projector 24 by the first control unit 21d.

The lighting device 22 is a device that illuminates the space where the user is present, and is, for example, an LED lighting device such as a ceiling light. It may be used as a device that exhibits a performance effect that increases the user's desire to exercise by performing color.

The audio device 23 is a speaker that provides sound such as music to the space where the user is present, and under the control of the terminal device 21, reproduces the sound included in the training video or the support video to increase the user's desire to exercise. It is used as a device that exerts a staging effect to make it happen.

The projector 24 is an example of a first display unit for displaying model training images and support images under the control of the terminal device 21 in place of the display 21b included in the terminal device 21 or together with the display 21b. is. Similarly, under the control of the first terminal device 21, it may be used as a device that exhibits a performance effect that increases the user's desire to exercise in conjunction with training video or support video. An example of the first display unit may be a TV compatible with AirPlay (registered trademark).

The air conditioner 25 is a device that adjusts the air in the space where the user is, such as an air conditioner, an air purifier, a circulator, a fan, etc., and is suitable for the user to exercise under the control of the terminal device 21. It is used as a device that maintains the environment such as room temperature.

The terminal device 21 is not necessarily a mobile terminal, and may be a stationary terminal device such as a desktop PC. Also, as components of the exercise support system 10, both the display 21b of the terminal device 21 and the projector 24 are not necessarily required, and either one of them may be provided. Further, as a component of the exercise support system 10, the camera 21a is not limited to the one provided in the terminal device 21, but instead or in addition, an external camera such as a web camera may be used. In addition, the peripheral devices (lighting device 22, sound device 23, projector 24, air conditioner 25) do not need to be independent and separate devices. It may also be an integrated smart light. Further, the peripheral devices (lighting device 22, sound device 23, projector 24, air conditioner 25) are not necessarily essential components of the exercise support system 10. FIG.

The facility device 30 includes a controller 31 and peripheral devices (lighting device 32, sound device 33, projector 34, cameras 36a to 36d, air conditioner 35, floor load cell 37a, wall load cell 37b) controlled by the controller 31. be done.

The controller 31 is a device for controlling peripheral devices, for example, a computer device such as a PC, and includes a second storage section 31c and a second control section 31a.

The second storage unit 31c is a storage unit for holding training images that serve as models, and is, for example, a semiconductor non-volatile memory.

The second control unit 31a controls the display of the training image held in the second storage unit 31c on the projector 34, and the user's movement detected by the cameras 36a to 36d, the floor surface load cell 37a, and the wall surface load cell 37b. Calculate the difference from the movement shown by the training video, generate a support video for supporting the user's exercise based on the calculated difference, and display the generated support video on the projector 34 in conjunction with the display of the training video. to control the The second control unit 31a includes a memory for storing programs and the like, a CPU for executing the programs, an input/output interface connected to peripheral components, a communication unit 31b, and the like. The communication unit 31b communicates with peripheral devices (lighting device 32, sound device 33, projector 34, cameras 36a to 36d, air conditioner 35, floor load cell 37a, wall load cell 37b), which are external devices, by wireless or wired communication. , the exercise support device 40, and the in-home device 20, such as a communication card for WiFi (registered trademark) or a mobile phone network.

The lighting device 32 is a device that illuminates the space where the user is present, and is, for example, an LED lighting device such as a ceiling light. It is used as a device that exerts a performance effect that increases the user's desire to exercise by performing. In addition, full color toning is included in the toning here.

The audio device 33 is a speaker that provides sound such as music to the space where the user is present, and under the control of the controller 31, reproduces the sound included in the training video or the support video to increase the user's desire to exercise. It is used as a device that exerts a staging effect for

The cameras 36a to 36d detect the movement of the user exercising in accordance with the training video displayed by the projector 34 with higher accuracy than the first detection unit (that is, the camera 21a) constituting the in-home device 20. An example of a unit, for example, a digital imaging device. More specifically, the cameras 36a to 36d are high-performance cameras having at least one function of a wider angle of view and a higher frame rate than the camera 21a of the in-home device 20.

The projector 34 is an example of a second display unit for displaying model training images and support images under the control of the controller 31 . Also, under the control of the controller 31, it is used as a device that exhibits a performance effect that increases the user's desire to exercise in conjunction with the training video or the support video.

The air conditioner 35 is a device that adjusts the air in the space where the user is, such as an air conditioner, an air purifier, a circulator, and a fan. It is used as a device that maintains the environment such as

The floor surface load cell 37a and the wall surface load cell 37b are examples of force sensors included in the second detection unit that detect the pressing force of the user on the floor surface and wall surface of the facility, respectively, and are two-dimensionally arranged load cell sensors. It is a seat that incorporates

The facility device 30 is provided with four cameras 36a to 36d as the second detection unit. You may provide only a stand, and you may provide two, three, five or more. In addition, the peripheral devices (lighting device 32, sound device 33, projector 34, cameras 36a to 36d, air conditioner 35, floor surface load cell 37a, wall surface load cell 37b) do not need to be independent and separate devices. , the lighting device 32 may be an integrated smart light that incorporates the sound device 33 and the projector 34 . Moreover, the lighting device 32, the acoustic device 33, the air conditioner 35, the floor surface load cell 37a, and the wall surface load cell 37b are not necessarily essential constituent elements of the exercise support system 10. FIG.

Although the second storage unit 31c has been described as being provided in the facility device 30, the second storage unit 31c is provided in the exercise support device 40, transmits the training video to the facility device 30 via the communication path, and the second control unit 31a , may be displayed on the projector 34 .

The exercise support device 40 is a server device having the in-home device 20 and the facility device 30 as clients. It is a computer device composed of peripheral devices and the like, and functionally includes a training video generation unit 41 . Further, a third storage unit 47 may be provided that holds training images that serve as models.

The training video generation unit 41 acquires, via a communication channel, difference data indicating the difference between the motion of the user and the motion shown by the training video, which is calculated by the second control unit 31a of the facility device 30, and generates the acquired difference data. Based on the data, a model training video is generated. For example, the training image generation unit 41 holds the same training images as those used in the facility device 30 in the third storage unit 47 or the like. The training video generation unit 41 adds an image such as an arrow indicating a direction to reduce the difference indicated by the difference data acquired from the facility device 30 to the training video, or generates a guidance voice to reduce the difference. Generate and include in training videos.

As a result, the terminal device 21 downloads a model training video from the exercise support device 40, stores it in the first storage unit 21c, and reproduces it when the user exercises at home. Effective training using a guided training video that reflects the individuality of the user and makes effective use of the difference data generated by the facility device 30 having the second detection units (that is, the cameras 36a to 36d). can be done.

Note that the training video generation unit 41 may acquire the difference data from the facility device 30 off-line, for example, via a portable recording medium owned by the user without using a communication channel.

Next, the operation of the exercise support system 10 configured as described above will be described.

FIG. 3 is a flowchart showing the operation of the in-home device 20 in the exercise support system 10 according to the embodiment (that is, one example of the exercise support method). FIG. 4 is a diagram showing how the in-home device 20 operates in the exercise support system 10 according to the embodiment.

The first control unit 21d of the in-home device 20 downloads a training video as a model from the exercise support device 40, television broadcasting, or websites on the Internet via the communication unit 21e, and stores it in the first storage unit 21c. (S10).

The first control unit 21d performs control to display the training image stored in the first storage unit 21c on the display 21b or the projector 24 (S11). At this time, the first control unit 21d may cause the sound device 23 to reproduce the sound contained in the training video, analyze the intensity of the instructor's movement in the training video, and according to the analysis result, 22 and the acoustic device 23 may be controlled to give a dramatic effect to the user, and the air conditioner 25 may be controlled to adjust the room temperature to a more comfortable one.

The first control unit 21d uses the camera 21a to detect the movement of the user exercising according to the training image displayed by the display 21b or the projector 24 (S12). That is, the first control unit 21d controls the camera 21a to capture the movement of the user who is exercising.

The first control unit 21d generates a support image for supporting the user's exercise based on the detection result by the camera 21a (S13). The support image may be an image showing the movement of the user captured by the camera 21a, or a human body shape model (that is, a skeleton model) generated by analyzing the image by the first control unit 21d. ) or other processed moving images. The human body shape model is, for example, a model represented by a plurality of segments corresponding to the skeleton of the human body and a plurality of joints corresponding to the joints of the human body.

Then, as shown in FIG. 4, the first control unit 21d displays the generated support image 24a on the display 21b and/or the projector 24 in conjunction with the display of the training image 24b, and the support image includes sound. If so, the audio device 23 is controlled to reproduce the sound (S14).

In this way, the user can train at home while comparing his/her movement with the training video that serves as a model. At this time, by downloading, as a model training video, a training video generated by the exercise support device 40 and reflecting the user's individuality using the difference data (S10), effective exercise can be achieved. It is supported, promotes habituation of exercise, and enhances continuity.

In step S10, the first control unit 21d selects the model stored in the third storage unit 47 of the exercise support device 40 or the second storage unit 31c of the facility device 30 from the television broadcast or the website on the Internet. The training image may be controlled to be displayed on the display 21b or the projector 24 (S11).

FIG. 5 is a flow chart showing the operation of the facility device 30 in the exercise support system 10 according to the embodiment (that is, one example of the exercise support method). FIG. 6 is a diagram showing how the facility device 30 operates in the exercise support system 10 according to the embodiment.

The second control unit 31a of the facility device 30 downloads a model training video from the exercise support device 40, television broadcasting, or websites on the Internet via the communication unit 31b, and stores it in the second storage unit 31c. (S20).

The second control unit 31a controls the projector 34 to display the training image stored in the second storage unit 31c (S21). At this time, the second control unit 31a may cause the sound device 33 to reproduce the sound contained in the training video, analyze the intensity of the instructor's movement in the training video, and according to the analysis result, 32 and the acoustic device 33 may be controlled to give a performance effect to increase the user's desire to exercise, and the air conditioner 35 may be controlled to adjust the room temperature to a more comfortable one.

The second control unit 31a uses the second detection unit (that is, the cameras 36a to 36d, the floor surface load cell 37a, and the wall surface load cell 37b) to enhance the movement of the user exercising according to the training image displayed by the projector 34. Accurately detected (S22). Details of this will be described later.

Then, the second control unit 31a calculates the difference between the movement of the user detected by the second detection unit (that is, the cameras 36a to 36d, the floor surface load cell 37a, and the wall surface load cell 37b) and the movement indicated by the training video ( S23a) Based on the calculated difference, a highly accurate assistance image for assisting the user's exercise is generated (S23b). For example, the second control unit 31a generates a human body shape model of the user, calculates muscle strength based on the calculated difference, and generates the generated human body shape model and the calculated muscle strength as a support image. Details of this will be described later.

Then, as shown in FIG. 6, the second control unit 31a causes the projector 34 to display the generated support image 51 in conjunction with the display of the training image 52, and if the support image 51 includes sound, , the sound is controlled to be reproduced by the sound device 33 (S24).

Although the second storage unit 31c has been described as being provided in the facility device 30, it is provided in the exercise support device 40, transmits the training video to the controller 31 via the communication path, and the second control unit 31a You may make it display on the projector 34. FIG.

7A to 7C are diagrams for explaining the details of step S22 in which the motion of the user is detected with high accuracy using the cameras 36a to 36d, the floor surface load cell 37a and the wall surface load cell 37b.

FIG. 7A is a diagram showing an arrangement example of the cameras 36a to 36d, the floor load cell 37a and the wall load cell 37b in the place (that is, space) where the user trains. (a) of FIG. 7A is a horizontal view of the place (that is, the space) where the user trains. (b) of FIG. 7A is a view of the place (that is, the space) where the user trains from the vertical direction. A camera 36a that zooms in on the user's face is arranged in front of the user, and cameras 36b constituting Kinect (trademark of Microsoft Corporation) that capture the movement of the user are installed in front of the user and on the left and right walls. 36d is arranged, a sheet-shaped floor surface load cell 37a is arranged on the floor, and a sheet-shaped wall surface load cell 37b is arranged on the wall surface in front of the user.

FIG. 7B is a diagram showing an example of user movement detection by the second control unit 31a using the cameras 36a to 36d, the floor surface load cell 37a, and the wall surface load cell 37b. Here, the positions of the joints of the human body model generated for the images captured by the cameras 36b to 36d are indicated by circles.

FIG. 7C shows an example of detection of the center-of-gravity position of the user by the floor surface load cell 37a. (a) of FIG. 7C shows a user standing stationary on the floor load cell 37a (left figure) and the center of gravity position of the user detected by the floor load cell 37a at that time (circle 37c in the right figure). )It is shown. FIG. 7C (b) shows the user moving on the floor load cell 37a (left figure) and the center of gravity position of the user detected by the floor load cell 37a at that time (circle 37c in the right figure). It is

FIG. 8 is a flow chart showing details of steps S23a and S23b in FIG.

The second control unit 31a generates a human body shape model of the user and calculates muscle strength by analyzing images showing the movement of the user obtained by the cameras 36b to 36d (S23a1). Similarly, the second control unit 31a analyzes the training video to generate a human body shape model of the instructor in the training video and calculate muscle strength (S23a2).

More specifically, for the generation of the human body shape model, the second control unit 31a uses YOLO (You Only Look Once) or SSD (Single Look Once) capable of high-speed object detection from the images obtained by the cameras 36b to 36d. A user's region is extracted by the method of Shot Multibox Detector. Then, for the extracted region, for example, using a technique such as CPN (Cascaded Pyramid Network), skeletal points are extracted from the human image of each frame, each skeletal point is connected, and a plurality of body segments and joints are provided. A three-dimensional human body shape model is generated. As for the muscle strength, the second control unit 31a calculates the joint force of each joint when the human body shape model exercises, and calculates the joint force of each joint in the human body model for a plurality of muscles supporting the joint. From the muscle model (muscle and muscle strength database) that determines the volume and the position where the muscle attaches to the body segment, the muscle strength obtained from the joint force for each joint in the human body shape model is converted to the cross-sectional area of multiple muscles that support the joint. The muscle strength of each muscle is calculated by distributing the muscle strength accordingly (see Japanese Patent Application Laid-Open No. 2013-68979 (Patent Document 2)).

Then, the second control unit 31a calculates the difference between the human body model and muscle strength of the user and the instructor (S23a3), and attaches an image such as an arrow indicating the direction to reduce the difference to the calculated human body model of the user. By adding and processing, it is generated as a support video (S23b1). At this time, the second control unit 31a may generate a voice explaining the calculated difference or a voice guiding to reduce the difference and include it in the support video.

Next, the second control unit 31a corrects the user's muscle strength calculated from the image using the pressing force detected by the floor surface load cell 37a and the wall surface load cell 37b (S23b2). For example, the second control unit 31a corrects the muscle strength of the user calculated from the image to be closer to the muscle strength calculated from the pressing force detected by the floor surface load cell 37a and the wall surface load cell 37b.

Further, the second control unit 31a corrects the center-of-gravity position of the generated human body shape model from the pressing force detected by the floor surface load cell 37a and the wall surface load cell 37b (S23b3). For example, the second control unit 31a corrects the center-of-gravity position of the human body shape model of the user calculated from the image to be closer to the center-of-gravity position detected by the floor surface load cell 37a.

The human body shape model and muscle strength thus corrected are displayed as a support image on the projector 34 in conjunction with the display of the training image, and the audio included in the support image is reproduced by the acoustic device 33 .

FIG. 9 is a diagram showing a display example of the support image 51 and the training image 52 by the projector 34 of the facility device 30 according to the embodiment. As shown in this figure, the assistance image 51 is displayed on the left side of the wall in front of the user, and approximates the outline 51a of the exercising user, the human body shape model 51b, the muscle strength 51c, and the training image 52 that serves as a model. It is composed of an arrow 51d indicating a direction. A training video 52 is displayed on the right side of the wall in front of the user and consists of an instructor's outline 52a in the video, a human body shape model 52b, and muscle strength 52c.

In this way, by using the facility device 30 installed in a fitness gym or the like, the user can know the difference from the training video with high precision and multifacetedness compared to training at home. It is possible to efficiently perform training that approximates the movements of the instructor shown in the training video that serves as a model.

Note that the facility device 30 has a function of changing the training video according to the state of the user who is exercising. FIG. 10 is a flow chart showing the movement of the facility device 30 that changes the training video according to the state of the user who is exercising.

The second control unit 31a estimates the user's emotion from the facial image of the user in exercise captured by the camera 36a (S30). Specifically, the second control unit 31a extracts, from the user's face image, physiological data such as the color of the face and heart rate associated with changes in blood flow, and non-physiological data such as facial expressions specified by the positions of facial feature points. Acquiring data, calculating the user's arousal value and comfort value based on the acquired data, and estimating the user's feelings such as "relaxed" and "suffering" from the training situation of the user, The degree of the estimated emotion (“relaxation”, “spiciness”) is calculated (see Japanese Patent Application Laid-Open No. 2017-144222 (Patent Document 3)).

Then, when the relaxed feeling indicated by the estimated emotion is greater than the threshold (that is, when the emotion with the highest matching score is "relaxed" and the matching score exceeds the threshold), ( If Yes in S31), the displayed training video is changed to a training video with a higher exercise load (S32). For example, the second control unit 31a speeds up the playback speed of the training video.

In addition, when the spiciness indicated by the estimated emotion is greater than the threshold (that is, when the emotion with the highest matching score is “spicy” and the matching score exceeds the threshold), ( If Yes at S33), the displayed training video is changed to a training video with a lower exercise load (S34). For example, the second control unit 31a slows down the playback speed of the training video.

By changing the training video based on such emotion estimation, the exercise load applied to the user becomes a value according to the user's exercise capacity and condition, and exercise support for the user is effectively provided.

In the above embodiment, the in-home device 20 and the facility device 30 execute various processes such as generating a support video for supporting the user's exercise. , the exercise support device may perform various processes as a server device.

FIG. 11 is a block diagram showing the configuration of an exercise support device 40a in an exercise support system 10a according to a modification of the embodiment. This exercise support device 40a includes a human body shape model generation unit 42, a muscle strength calculation unit 43, a support image generation unit 44, and an emotion estimation unit 45, in addition to the training video generation unit 41 included in the exercise support device 40 according to the above embodiment. Prepare.

The human body shape model generation unit 42 generates a human body shape model of the user by analyzing an image showing the movement of the user obtained by the camera, which is sent from the in-home device 20 and the facility device 30, or stored in advance. By analyzing the training video that the instructor is doing, the human body shape model of the instructor in the training video is generated.

The muscle strength calculation unit 43 calculates the muscle strength of the user by analyzing the video showing the movement of the user obtained by the camera, which is sent from the in-home device 20 and the facility device 30, and calculates the strength of the user. By analyzing the video, the muscle strength of the instructor in the training video can be calculated.

The functions of the human body shape model generation unit 42 and the muscle force calculation unit 43 correspond to the processes of steps S23a1 and S23a2 in the flowchart of FIG. 8 in the above embodiment.

The support image generation unit 44 calculates the difference between the human body shape model and muscle strength of the user and the instructor generated and calculated by the human body shape model generation unit 42 and the muscle strength calculation unit 43, and displays an arrow or the like indicating the direction to reduce the difference. The image is added to the generated human body shape model of the user and processed to generate a support video. At this time, the user's muscle strength calculated by the muscle strength calculator 43 may be corrected using the pressing force detected by the floor surface load cell 37a and the wall surface load cell 37b sent from the facility device 30, or the pressing force may be corrected. The position of the center of gravity of the human body model generated by the human body model generator 42 may be corrected from the pressure. The function of the support image generator 44 corresponds to the processing of steps S23a3 to S23b3 in the flowchart of FIG. 8 in the above embodiment.

The emotion estimating unit 45 estimates the user's emotion from the face image of the exercising user, which is sent from the facility device 30 and captured by the camera 36a. The function of this emotion estimation unit 45 corresponds to the process of step S30 in the flowchart of FIG. 10 in the above embodiment.

FIG. 12A is a communication sequence diagram showing exchanges between the in-home device 20 and the exercise support device 40a in the exercise support system 10a according to the modification of the embodiment. (a) of FIG. 12A shows a sequence in which a process of uploading an image showing the movement of the user obtained by the camera 21a of the in-home device 20 from the in-home device 20 to the exercise support device 40a is repeated. In (b) of FIG. 12A, the support video generated by the human body shape model generation unit 42, the muscle strength calculation unit 43, and the support video generation unit 44 of the exercise support device 40a is streamed from the exercise support device 40a to the in-home device 20. It shows the sequence to be delivered. The sequence shown in (a) of FIG. 12A and the sequence shown in (b) of FIG. 12A are performed in parallel. With such a sequence, the in-home device 20 can support the user's exercise in real time using the support video generated by the exercise support device 40a.

FIG. 12B is a communication sequence diagram showing exchanges between the facility device 30 and the exercise support device 40a in the exercise support system 10a according to the modification of the embodiment. (a) of FIG. 12B shows an image showing the movement of the user obtained by the cameras 36a to 36d of the facility device 30 from the facility device 30 to the exercise support device 40a, and the motion of the user by the floor load cell 37a and the wall surface load cell 37b. It shows a sequence in which the process of uploading the detected pressing force data of the user is repeated. In (b) of FIG. 12B, the support video generated by the human body shape model generation unit 42, the muscle strength calculation unit 43, and the support video generation unit 44 of the exercise support device 40a is streamed from the exercise support device 40a to the facility device 30. A sequence is shown in which the data representing the emotion estimated by the emotion estimation unit 45 of the exercise support device 40a is repeatedly downloaded. The sequence shown in (a) of FIG. 12B and the sequence shown in (b) of FIG. 12B are performed in parallel. With such a sequence, the facility device 30 can support the user's exercise in real time using the support video generated by the exercise support device 40a, and adjust the load of the training video according to the user's emotion.

The repetition of the sequence in FIGS. 12A and 12B is performed, for example, in a predetermined unit (predetermined number of frames, predetermined time, each scene in the video) in the video uploaded to the exercise support device 40a.

In addition, the in-home device 20, the facility device 30, and the exercise support device 40a acquire exercise images of athletes or instructors from television broadcasts or websites on the Internet, and generate a human body shape model from the acquired exercise images. may be generated to calculate muscle strength, and a training video to which the generated human body shape model and the calculated muscle strength are added may be generated.

As described above, the exercise support system 10 according to the present embodiment includes the in-home device 20 that supports the user's exercise training at home, the facility device 30 that supports the user's exercise training in the facility, An exercise support device 40 connected to at least the in-home device 20 via a communication path, and a memory provided in any one of the in-home device 20, the facility device 30, and the exercise support device 40 and holding a training video serving as a model. The in-home device 20 includes a first display unit such as a display 21b and/or a projector 24 that displays an image, and a first display unit that controls to display the training image held in the storage unit on the first display unit. It has a control unit 21d and a first detection unit such as the camera 21a that detects the movement of the user exercising in accordance with the training video displayed on the first display unit. The facility device 30 performs control to display a support video for supporting the user's exercise, which is generated based on the detection result of the detection unit, on the first display unit in conjunction with the display of the training video, and the facility device 30 displays the video. a second display unit such as a projector 34, a second control unit 31a that controls to display the training image held in the storage unit on the second display unit, and a training image displayed on the second display unit. and a second detection unit such as the cameras 36a to 36d that detects the movement of the user exercising with higher accuracy than the first detection unit of the in-home device 20. The second control unit 31a further includes the second detection unit calculating the difference between the movement of the user detected by the and the movement shown by the training video, and interlocking the support video for supporting the user's exercise, which is generated based on the calculated difference, with the display of the training video 2, the exercise support device 40 acquires difference data indicating the difference calculated by the second control unit 31a of the facility device 30, and serves as a model based on the acquired difference data. A training video is generated, and the in-home device 20 acquires the training video generated by the exercise support device 40 via the communication channel and stores it in the storage unit.

The storage units are provided as a first storage unit 21c and a second storage unit 31c in the in-home apparatus 20 and the facility apparatus 30, respectively. The second control unit 31d of the facility device 20 performs control to display the training image stored in the second storage unit 31c on the second display unit, and The in-home device 30 may acquire the training video generated by the exercise support device 40 via the communication channel and store it in the first storage unit 21c.

As a result, information about the difference between the movement indicated by the training video and the user's movement obtained by high-precision sensing in the facility device 30 is fed back to the in-home device 20 via the exercise support device 40 . In other words, with the in-home apparatus 20, the user can exercise by using a training video that reflects the user's individuality and is suitable for the user. Therefore, an exercise support system is realized that can support exercise more effectively than before, and encourage the user to make exercise a habit to improve continuity.

In addition, in the in-home device 20, the display 21b as the first display unit, the first storage unit 21c, the first control unit 21d, and the camera 21a as the first detection unit operate one terminal device 21 such as a smartphone. Configure. As a result, the user can easily train for exercise at home using a terminal device such as a smartphone.

In addition, in the in-home device 20, the first display unit is the projector 24, and the first storage unit 21c, the first control unit 21d, and the camera 21a as the first detection unit are connected to the projector 24 via a communication path. A single terminal device 21 may be configured. This allows the user to exercise at home in an environment close to the facility apparatus 30 using the large screen of the projector 24 .

Further, in the facility device 30, the second control unit 31a further performs control to provide at least one of video and audio that increases the user's desire to exercise in the space where the user is present in conjunction with the display of the training video. do. As a result, the facility apparatus 30 provides a presentation effect to the user who is exercising, thereby increasing the user's desire to exercise.

In addition, the first detection unit of the in-home device 20 is one camera 21a, and in the facility device 30, the second detection unit has at least one function of a wider angle of view and a higher frame rate than the camera 21a of the in-home device 20. , and the second display is the projector 34 . As a result, the facility device 30 can sense the movement of the user with high accuracy and effectively support the movement of the user.

In addition, the exercise support device 40 acquires an exercise video of an athlete or an instructor from a television broadcast or a website on the Internet, generates a human body shape model from the acquired exercise video, calculates muscle strength, and generates a human body shape model. And a training video is generated using the calculated muscle strength. As a result, high-quality training videos can be generated without spending much time and money compared to outsourcing to a specialized production company.

Further, in the facility device 30, the second control unit 31a generates a human body shape model of the user based on the calculated difference, calculates the muscle strength, and uses the generated human body shape model and the calculated muscle strength as a support image. The image is displayed on the projector 34 as the second display unit. As a result, in the facility device 30, the user can intuitively grasp the difference between the training video and the movement of the user by checking the displayed human body shape model and muscle strength, and can improve the skill efficiently. .

Also, the second detection unit includes a floor surface load cell 37a and a wall surface load cell 37b as force sensors that detect the pressing force of the user against at least one of the floor surface and wall surface of the facility. ) The second control unit 31a corrects the calculated muscle strength from the pressing force detected by the floor surface load cell 37a and the wall surface load cell 37b. As a result, an accurate muscle force that reflects the actual pressing force applied by the user is calculated.

Further, the second control unit 31a corrects the center-of-gravity position of the generated human body shape model from the pressing force detected by the floor surface load cell 37a and the wall surface load cell 37b as force sensors. As a result, an accurate center-of-gravity position reflecting the actual weight shift by the user is calculated.

Also, the second detection unit includes a camera 36a that captures the user's face, and the second control unit 31a estimates the user's emotion from the user's face captured by the camera 36a, and detects the relaxation indicated by the estimated emotion. If the feeling is greater than the threshold, the displayed training video is changed to a training video with a higher exercise load, and if the difficulty indicated by the estimated emotion is greater than the threshold, the displayed training video to a training video with a lower exercise load. As a result, the exercise load given to the user has a value corresponding to the exercise capacity and condition of the user, and the exercise support for the user is effectively provided.

Also, the present invention may be implemented as a program that the indoor device 20 or the facility device 30 has. The program is a program executed by a computer provided in the in-home device 20 or the facility device 30 in the exercise support system 10, and includes a training video display step of controlling to display the training video held in the storage unit on the display unit; A detection step of using a detection unit to detect movement of a user exercising in accordance with a training video displayed on a display unit; and a support image display step for controlling display on the display unit.

Such a program realizes the in-home device 20 or the facility device 30 of the exercise support system that can support more effective exercise than before, encourage the user to make exercise a habit, and improve continuity. be.

Further, a communication step of transmitting the detection result in the detection step to the exercise support device 40a via a communication channel and acquiring a support image generated by the exercise support device 40a based on the transmitted detection result, In the support image display step, the support image acquired in the communication step is displayed on the display unit in conjunction with the display of the training image. This implements a client program that operates in cooperation with the exercise support device 40a as a server device.

Also, the present invention may be implemented as a program that the indoor device 20 or the facility device 30 has. The program is a program executed by a computer provided in the in-home device 20 or the facility device 30, and obtains an exercise video of an athlete or an instructor from a television broadcast or a website on the Internet, and generates a human body shape model from the obtained exercise video. is generated to calculate muscle strength, a training video is generated using the generated human body shape model and the calculated muscle strength, and stored in the first storage unit 21c or the second storage unit 31c. As a result, high-quality training videos can be generated without spending much time and money compared to outsourcing to a specialized production company.

Further, the present invention includes an in-home device 20 that supports exercise training by a user at home, a facility device 30 that supports exercise training by a user in a facility, and the in-home device 20 and the facility device via a communication path. An exercise support system 10 comprising: an exercise support device 40 connected to a network 30; and a storage unit provided in any one of the in-home device 20, the facility device 30, and the exercise support device 40 and holding a training video serving as a model. in which the in-home device 20 causes the training video held in the storage unit to be displayed on a first display unit such as the display 21b; a step of detecting the movement of the user exercising in accordance with the training video displayed on the first display unit using the a step of displaying on the first display unit a support image for providing support in conjunction with the display of the training image; a step of the facility device 30 displaying the training image held in the storage unit on the second display unit; The device 30 uses the second detection unit such as the cameras 36a to 36d to detect the movement of the user exercising in accordance with the training video displayed on the second display unit with higher accuracy than the first detection unit of the in-home device 20. and the facility device 30 calculates the difference between the movement of the user detected by the second detection unit and the movement indicated by the training video, and supports the user's exercise generated based on the calculated difference. a step of displaying a support image for training on a second display unit such as the projector 34 in conjunction with the display of the training image; a step of generating a model training video based on the acquired difference data; and the in-home device 20 acquiring the training video generated by the exercise support device 40 via a communication channel and storing it in a storage unit. step.

As a result, information about the difference between the movement indicated by the training video and the movement of the user, which is obtained by high-precision sensing in the facility device 30, is fed back to the in-home device 20 via the exercise support device 40, making exercise easier than before. An exercise support system is realized that can promote habituation and improve continuity.

Although the exercise support system and the exercise support method according to the present invention have been described above based on the embodiments and modifications, the present invention is not limited to these embodiments and modifications. As long as it does not deviate from the gist of the present invention, there are also embodiments and modifications applied to various modifications that a person skilled in the art can think of, and other forms constructed by combining some components in the embodiments and modifications. , are included within the scope of the present invention.

For example, in the above-described embodiment, the training video generator 41 is provided in the exercise support device 40 , but may be provided in the terminal device 21 of the in-home device 20 . In that case, the terminal device 21 of the in-home device 20 acquires the difference data generated by the facility device 30 instead of the exercise support device 40, and based on the acquired difference data, prepares a model training video. It is generated and stored in the first storage unit 21c.

Further, in the above-described embodiment, the facility device 30 is provided with the floor surface load cell 37a and the wall surface load cell 37b, but they may be provided in the in-home device 20 as well. In this case, the user's pressing force data detected by the floor load cell and the wall surface load cell provided in the in-home device 20 is uploaded to the exercise support device 40a, and the emotion estimation unit 45 of the exercise support device 40a performs emotion estimation. Data indicating the estimated emotion is downloaded to the in-home device 20 and used to adjust the load of the training video in the home.

In addition, the method of estimating emotion is not limited to the one described in the above embodiment, and the user's facial image is learned by deep learning when the exercise load is low or when the exercise load is high. By comparing with a database, the emotion may be estimated. In this case, in addition to making the multi-layer network learn the feature amount, machine learning may be performed to designate the degree of opening of the eyes, the distance between the eyebrows, the wrinkles between the eyebrows, the mouth, etc. as the feature amount.

Also, the exercise support devices 40 and 40a are not limited to being composed of one device, and may be composed of a plurality of server devices distributed according to the processing functions, for example.

The present invention is an exercise assistance system that assists a user in exercising using a training video, and in particular, assists the exercise more effectively than before, encourages the user to make exercise a habit, and enhances continuity. can be used as an exercise support system including an in-home device and a facility device.

10, 10a exercise support system 20 home device 21 terminal device 21a, 36a to 36d camera 21c first storage unit 21d first control unit 24, 34 projector 24a, 51 support video 24b, 52 training video 30 facility device 31a second control unit 31c Second storage unit 40, 40a Exercise support device

Claims (17)

an in-home device for assisting a user in-home in exercise training;
a facility device that supports exercise training by the user in the facility;
an exercise support device connected to the in-home device and the facility device via a communication path;
a storage unit provided in any one of the in-home device, the facility device, and the exercise support device and holding a training video serving as a model;
with
The in-home device is
a first display unit that displays an image;
a first control unit that controls to display the training video held in the storage unit on the first display unit;
a first detection unit that detects the movement of the user exercising in accordance with the training video displayed on the first display unit;
The first control unit further causes the first display unit to display a support image for supporting the user's exercise, which is generated based on the detection result of the first detection unit, in conjunction with the display of the training image. to control the display on the
The facility device includes:
a second display unit that displays an image;
a second control unit that controls to display the training video held in the storage unit on the second display unit;
a second detection unit that detects the movement of the user exercising in accordance with the training video displayed on the second display unit with higher accuracy than the first detection unit;
The second control unit further calculates a difference between the motion of the user detected by the second detection unit and the motion indicated by the training video, and generates motion of the user based on the calculated difference. control to display a support video for supporting on the second display unit in conjunction with the display of the training video,
The exercise support device acquires difference data indicating the difference calculated by the second control unit, generates a model training video based on the acquired difference data,
The in-home device acquires a training video generated by the exercise support device via the communication channel and stores the training video in the storage unit.
Exercise support system. The storage units are provided in the in-home apparatus and the facility apparatus as a first storage unit and a second storage unit, respectively. Perform control to display on the first display unit,
The second control unit of the facility device performs control to display the training video held in the second storage unit on the second display unit,
The in-home device acquires a training video generated by the exercise support device via the communication path and stores the training video in the first storage unit.
The exercise support system according to claim 1. The first display unit, the first storage unit, the first control unit, and the first detection unit constitute one mobile terminal,
The exercise support system according to claim 2. The first display unit is a projector,
The first storage unit, the first control unit, and the first detection unit constitute one mobile terminal connected to the projector via a communication path,
The exercise support system according to claim 2. The second control unit further performs control to provide at least one of video and audio that increases the user's motivation to exercise in the space where the user is present, in conjunction with the display of the training video.
The exercise support system according to any one of claims 1 to 4. The first detection unit is one camera,
The second detection unit is one or more cameras having at least one function of a wider angle of view and a higher frame rate than the first detection unit,
The second display unit is a projector,
The exercise support system according to any one of claims 1 to 5. The exercise support device acquires an exercise video of an athlete or an instructor from a television broadcast or a website on the Internet, generates a human body shape model from the acquired exercise video, calculates muscle strength, and generates the human body shape model. and generating the training video using the calculated muscle strength;
The exercise support system according to any one of claims 1-6. The second control unit generates a human body shape model of the user based on the calculated difference, calculates muscle strength, and displays the generated human body shape model and the calculated muscle strength as the support image in the second display. display in the
The exercise support system according to any one of claims 1-7. The second detection unit includes a force sensor that detects the pressing force of the user against at least one of the floor surface and wall surface of the facility,
The exercise support system according to claim 8. The second control unit corrects the calculated muscle force from the pressing force detected by the force sensor.
The exercise support system according to claim 9. The second control unit corrects the center-of-gravity position of the generated human body shape model from the pressing force detected by the force sensor.
The exercise support system according to claim 9. The second detection unit includes a camera that captures the user's face,
The second control unit estimates an emotion of the user from the user's face photographed by the second detection unit, and displays a relaxed feeling indicated by the estimated emotion if it is greater than a threshold. changing the currently displayed training video to a training video with a higher exercise load, and if the difficulty indicated by the estimated emotion is greater than a threshold, changing the displayed training video to a training video with a lower exercise load;
The exercise support system according to any one of claims 1-11. A program executed by a computer provided in the in-home device in the exercise support system according to any one of claims 1 to 12,
a training image display step of controlling to display the training image held in the storage unit on the first display unit;
a detection step of using the first detection unit to detect the movement of the user exercising in accordance with the training video displayed on the first display unit;
a support video display step of controlling to display a support video generated based on the detection result by the first detection unit on the first display unit in conjunction with the display of the training video;
program. Furthermore, a communication step of transmitting the detection result in the detection step to the exercise support device via a communication path and acquiring a support video generated by the exercise support device based on the transmitted detection result,
In the support image display step, the support image acquired in the communication step is displayed on the first display unit in conjunction with the display of the training image.
14. A program according to claim 13. A program executed by a computer provided in the in-home device or facility device in the exercise support system according to any one of claims 1 to 12,
An exercise video of an athlete or an instructor is acquired from a television broadcast or a website on the Internet, a human body shape model is generated from the acquired exercise video, muscle strength is calculated, and the generated human body shape model and the muscle strength are used. generating a training video and storing it in the storage unit;
program. An in-home device for supporting exercise training by a user at home, a facility device for supporting exercise training by the user in a facility, and exercise connected to at least the in-home device and the facility device via a communication path An exercise support method using an exercise support system comprising: a support device; and a storage unit that is provided in any one of the in-home device, the facility device, and the exercise support device and stores a training video serving as a model,
a step of causing the in-home device to display the training video held in the storage unit on a first display unit;
a step in which the in-home device uses a first detection unit to detect movement of the user exercising in accordance with a training video displayed on the first display unit;
A step in which the in-home device displays, on the first display unit, an assistance image for assisting the user's exercise, which is generated based on the detection result of the first detection unit, in conjunction with the display of the training image. When,
a step of causing the facility device to display the training video held in the storage unit on a second display unit;
a step in which the facility device uses a second detection unit to detect, with higher accuracy than the first detection unit, the movement of the user exercising in accordance with the training video displayed on the second display unit;
The facility device calculates a difference between the motion of the user detected by the second detection unit and the motion indicated by the training image, and generates a motion based on the calculated difference to assist the user in exercising. a step of displaying the support image of on the second display unit in conjunction with the display of the training image;
a step in which the exercise support device acquires difference data indicating the difference calculated by the facility device, and generates a model training video based on the acquired difference data;
a step in which the in-home device acquires a training video generated by the exercise support device via the communication channel and stores the training video in the storage unit;
Exercise support method. 17. The exercise support method by the exercise support system according to claim 16, wherein the storage units are provided as a first storage unit and a second storage unit in the in-home device and the facility device, respectively, wherein the in-home device comprises , performing control to display the training image held in the first storage unit on the first display unit;
the facility device performing control to display the training video held in the second storage unit on the second display unit;
17. The exercise support method according to claim 16, further comprising a step in which said in-home device acquires a training video generated by said exercise support device via said communication path and stores it in said first storage unit.

Images