JP2023008347A - Attachment device and body motion supporting system - Google Patents

Abstract

To make it possible to accurately grasp motion of a user's body.SOLUTION: An attachment device attached to a user's body comprises: a sensor for detecting a movement; a feedback device; a storage device for storing a classifier for calculating accuracy of motion of a part at which the attachment device is attached by taking, as input, a type of motion performed by the user and a detection value of the sensor; and a microcontroller. The microcontroller acquires the type of motion performed by the user from an information processor, acquires the detection value from the sensor, calculates the accuracy by giving the motion and the detection value to the classifier, and actuates the feedback device depending on the accuracy.SELECTED DRAWING: Figure 1

Description

The present invention relates to a wearable device and a body movement assistance system.

Guidance is provided by measuring exercise postures (see Patent Document 1).

Japanese Patent Application Laid-Open No. 2021-037168

However, in the system described in Patent Document 1, it is difficult to accurately grasp the posture of the user.

SUMMARY OF THE INVENTION The present invention has been made in view of such a background, and an object of the present invention is to provide a technique capable of accurately grasping the movement of a user's body.

The main invention of the present invention for solving the above problems is a wearable device worn on a user's body, comprising a sensor for detecting exercise, a feedback device, a motion type performed by the user, and a detection value of the sensor. and a microcontroller, wherein the microcontroller determines the type of action performed by the user. obtaining from an information processing device, obtaining the detected value from the sensor, providing the operation and the detected value to the classifier to obtain the accuracy, and operating the feedback device according to the accuracy; characterized by

Other problems disclosed by the present application and solutions thereof will be clarified by the section of the embodiment of the invention and the drawings.

ADVANTAGE OF THE INVENTION According to this invention, a motion of a user's body can be grasped|ascertained correctly.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the whole structural example of the body movement assistance system of this embodiment. 3 is a diagram showing a hardware configuration example of the mounting device 30. FIG. 3 is a diagram showing a software configuration example of the wearing device 30. FIG. 2 is a diagram illustrating an example hardware configuration of a user terminal 10; FIG. 2 is a diagram showing a software configuration example of the user terminal 10; FIG. It is a figure explaining operation|movement of the body movement assistance system of this embodiment.

<Overview of the invention>
The contents of the embodiments of the present invention are listed and explained. The present invention has, for example, the following configurations.
[Item 1]
A wearable device worn on a user's body, comprising:
a sensor that detects motion;
a feedback device;
a storage device that stores a classifier for determining the accuracy of movement of the part to which the wearable device is worn based on input of the type of motion performed by the user and the detection value of the sensor;
a microcontroller, and
The microcontroller
acquiring the action type performed by the user from an information processing device;
obtaining the detected value from the sensor;
applying the motion and the detected value to the classifier to determine the accuracy;
operating the feedback device according to the accuracy;
A wearable device characterized by:
[Item 2]
A system for supporting a user's body movement,
comprising a wearable device worn on the user's body and an information processing device communicably connected to the wearable device,
The wearable device comprises:
a sensor that detects motion;
a feedback device;
with
The information processing device is
a storage unit that stores the correct positions of the key points of the body;
a photographing unit that acquires a photographed image of the user;
a detection unit that detects a first position candidate of the key point from the captured image;
a motion acquisition unit that acquires the motion detected by the sensor;
an estimator for estimating a second candidate location of the keypoint from the motion;
a position determination unit that determines a position of the keypoint based on the first and second position candidates;
an accuracy determining unit that compares the position and the correct position to determine the accuracy of the position;
a controller for controlling the wearable device to operate the haptic feedback device according to the accuracy;
to provide
A body movement support system characterized by:
[Item 3]
The body movement support system according to item 2,
The sensor is an inertial measurement device that detects three-dimensional inertial motion with nine degrees of freedom,
The storage unit further includes a classifier that obtains a second accuracy of the motion of the keypoint, which is different from the first accuracy of the position, by inputting the type of motion of the body and the value detected by the sensor. remember,
The information processing device further includes an instruction unit that instructs the user to perform the specific action type,
The accuracy determination unit obtains the second accuracy by providing the type of motion that has issued the instruction and the detection value by the sensor to the classifier,
the control unit operating the feedback device according to the second accuracy;
A body movement support system characterized by:
[Item 4]
The body movement support system according to item 3,
The information processing device is
a feedback storage unit that stores text, video, or audio feedback for the user in association with the first and second accuracy ranges;
a feedback output unit that reads and outputs the feedback corresponding to the first and second accuracies;
A body movement support system, further comprising:

<Overview of the system>
A body movement support system according to an embodiment of the present invention will be described below. The body movement support system of this embodiment is intended to guide the user in correct body movement. For example, it can teach correct yoga poses. In the body movement support system of this embodiment, the user wears a device (hereinafter referred to as a wearable device) on each part such as a wrist, ankle, and thigh, and the wearable device detects the movement and posture of each part. Validate user posture and movement. In addition, in the body movement support system of the present embodiment, a user terminal equipped with a camera such as a smartphone is used to photograph the user's posture, detect the position of the user's body part, and detect it from the measurement value and the image from the wearable device. It is also possible to verify the validity of the user's posture and movement based on the position of the body part. Furthermore, in the body movement support system of this embodiment, the wearable device is equipped with a tactile feedback device. can be notified.

FIG. 1 is a diagram showing an example of the overall configuration of the body movement support system of this embodiment. The body movement support system of this embodiment includes a user terminal 10 and a wearing device 30 .

The user terminal 10 is, for example, a computer such as a smart phone, smart watch, tablet computer, or personal computer. The user terminal 10 issues instructions to the user about body movements (for example, yoga poses to be taken). The user terminal 10 has a photographing function, and can photograph the user, detect specific parts of the user (hereinafter referred to as key points) from the photographed image, and evaluate the posture of the user.

The wearing device 30 is a device worn on the user's body. The wearing device 30 is worn on a part such as a wrist, an ankle, or a thigh. The wearable device 30 provides a sensor that detects the movement and posture of the worn part and feedback to the user (in this embodiment, tactile feedback is assumed, but feedback such as sound and light may be used). It has a tactile feedback device (such as a vibrator) and a microcontroller that verifies whether the user's body part position and movement are correct.

<Wearing device>
FIG. 2 is a diagram showing a hardware configuration example of the mounting device 30. As shown in FIG. Note that the illustrated configuration is an example, and other configurations may be employed. The wearable device 30 comprises an IMU 311 , a communication unit 312 , a haptic feedback device 313 and a microcontroller 314 .

The IMU 311 is a sensor that detects motion. In this embodiment, the IMU 311 is an inertial measurement unit that detects three-dimensional inertial motion. In this embodiment, the IMU 311 includes a three-dimensional acceleration sensor, a three-dimensional gyro sensor, and a three-dimensional magnetic sensor, and is assumed to be capable of measuring nine degrees of freedom.

The communication unit 312 communicates with the user terminal 10 . The communication unit 312 may perform short-range communication using, for example, BlueTooth (registered trademark), or may perform communication such as WiFi.

The tactile feedback device 313 is a device that makes it possible to obtain tactile feedback by applying force, vibration, movement, or the like to the user.

Microcontroller 314 is the device that controls wearable device 30 . The microcontroller 314 can implement various functions by executing programs stored in the memory 315 . The memory 315 is, for example, a flash memory that stores various data and programs.

FIG. 3 is a diagram showing a software configuration example of the wearable device 30. As shown in FIG. The wearable device 30 includes a motion type acquisition unit 321 , a detection value acquisition unit 322 , an accuracy estimation unit 323 , a feedback unit 324 , a detection value transmission unit 325 , a command reception unit 326 and a classifier storage unit 331 .

The action type acquisition unit 321 acquires the action type performed by the user from an external device. The motion type is, for example, a yoga pose. The motion type acquisition unit 321 can acquire the motion type from the user terminal 10, for example.

The detected value acquiring unit 322 acquires values (detected values) of the six degrees of freedom detected by the IMU 311 .

The detected value transmission unit 325 transmits the detected value to the outside. In this embodiment, the detected value transmission unit 325 transmits the detected value to the user terminal 10 .

The classifier storage unit 331 stores a classifier that is a learning model that has been learned by machine learning. The classifier stored in the classifier storage unit 331 is a learning model for inferring the accuracy of the movement of the part to which the wearable device 30 is worn based on input of the type of action performed by the user and the detection value of the IMU 311. . Note that the classifier storage unit 331 may store a classifier in association with an action type. In this case, the detection value of the IMU 311 is input to the classifier for each motion type without inputting the motion type. The classifier can be a learning model that has already been learned for each part of the body, and the classifier for each part on which the wearable device 30 is worn is preliminarily registered in the classifier storage unit 331 .

The accuracy estimation unit 323 estimates the accuracy of the motion of the user's body. In this embodiment, the accuracy estimation unit 323 estimates the accuracy of the position, posture, and movement of the part to which the wearable device 30 is worn. The accuracy estimator 323 can infer the accuracy by providing the classifier with the motion type and the detection value acquired by the detection value acquisition unit 322 .

A feedback unit 324 operates the haptic feedback device 313 according to the accuracy. The feedback unit 324 can, for example, control the haptic feedback device 313 (eg, set operation parameters) so that the lower the accuracy, the stronger the feel (eg, the larger, stronger, faster vibration).

The command receiving unit 326 receives commands for operating the haptic feedback device 313 from the outside. Feedback unit 324 can also operate haptic feedback device 313 in response to a command.

<User terminal 10>
FIG. 4 is a diagram showing a hardware configuration example of the user terminal 10. As shown in FIG. Note that the illustrated configuration is an example, and other configurations may be employed. A user terminal includes a CPU 101 , a memory 102 , a storage device 103 , a communication interface 104 , a camera 105 and a touch panel display 106 . The storage device 103 is, for example, a hard disk drive, solid state drive, flash memory, etc., which stores various data and programs. The communication interface 104 is an interface for connecting to a communication network. Examples include a USB (Universal Serial Bus) connector and an RS232C connector for serial communication. The touch panel display 106 is a device for inputting/outputting data. Each functional unit of the user terminal 10, which will be described later, is implemented by reading a program stored in the storage device 103 into the memory 102 and executing it. is implemented as part of the storage provided by

FIG. 5 is a diagram showing a software configuration example of the user terminal 10. As shown in FIG. The user terminal 10 includes an instruction output unit 111, an imaging unit 112, a position detection unit 113, a detection value acquisition unit 114, a position estimation unit 115, a position determination unit 116, an accuracy determination unit 117, a control unit 118, a feedback output unit 119, An instruction storage unit 131, a correct position storage unit 132, a classifier storage unit 133, and a feedback storage unit 134 are provided.

The instruction storage unit 131 stores information (hereinafter referred to as instruction information) including a body movement instruction (instruction) for the user. Instruction information can include timing, instructions, and action types. Instructions can be, for example, textual information, audio, still or moving images, or a combination thereof. The instructions can be, for example, the next pose to do in yoga. Timing is timing for outputting an instruction. The timing can be, for example, time information, and can be expressed, for example, as elapsed time from the start of exercise. The motion type is the type of body motion instructed by the instruction. For instructions other than body movement (for example, explanation), the action type may not be set, or information indicating a special type may be set.

The correct position storage unit 132 stores information (hereinafter referred to as correct information) including correct positions of key parts (hereinafter referred to as key points) for specifying the posture of the user's body. The correct information can include the correct position of the part in association with the information specifying the part. The correct position can be represented, for example, as a relative position from the standard position of the body, and can be, for example, a coordinate value of three-dimensional coordinates with the standard position (for example, a specific point on the floor) as the origin. Further, the correct information can include not only the position but also the posture (correct posture). Instead of or in addition to the correct position and correct posture, correct data expressed by a vector with nine degrees of freedom that can be measured by the IMU 311 of the wearing device 30 may be included.

The classifier storage unit 133 stores a classifier as a learning model created by machine learning. The classifier stored in the classifier storage unit 133 is a model for inferring the accuracy of movement of key points of the body based on input of the type of body motion and the detected value by the IMU 311 .

Note that the classifier storage unit 133 may store a classifier in association with a motion type. Accuracy can be inferred.

The feedback storage unit 134 stores information (hereinafter referred to as feedback information) for giving text, video, or audio feedback to the user. The feedback information includes user-defined values associated with accuracy ranges, such as "a little more", "more", "a little earlier", "more earlier", "a little sooner", and "faster". Advice on improving performance can be included. The accuracy range may be a position accuracy range, a pose accuracy range, and/or a motion accuracy range.

The instruction output unit 111 issues an instruction (instruction) to the user to perform a specific action type. In this embodiment, the instruction output unit 111 can read and output instructions stored in the instruction storage unit 131 in accordance with timing. Note that the instruction output unit 111 may output voice data obtained by reading the text data.

The photographing unit 112 acquires a photographed image of the user. The imaging unit 112 can acquire a captured image by controlling the camera 105 .

The position detection unit 113 detects the first position candidate of the keypoint from the captured image. The position detection unit 113 detects a key point from the captured image by image analysis using a known CV (Computer Vision) technique, detects a standard position (for example, a specific point on the floor), and detects a relative position from the standard position. Position can be calculated.

The detected value acquisition unit 114 acquires motion (detected value) detected by the IMU 311 of the wearable device 30 . The detection value acquisition unit 114 can communicate with each of the wearable devices 30 and acquire detection values.

A position estimation unit 115 estimates a second position candidate of the keypoint from the detected value. A known technique can be used for the process of estimating the keypoint position from the detection value of the IMU 311 .

A position determination unit 116 determines the position of the keypoint based on the first and second position candidates. If the first position candidate can be detected from the captured image, the position determination unit 116 may determine the first position candidate as the position of the keypoint. When the position determination unit 116 cannot detect the first position candidate from the captured image (for example, when the key point is in a blind spot and is invisible from the camera 105), the position determination unit 116 selects the second position candidate as a key point. You may make it determine as a position of a point. Also, the first position candidate detected from the captured image may be corrected based on the detection value from the IMU 311 . Also, the position determination unit 116 can obtain the attitude of the key point and the motion (speed etc.) per unit time from the detected values.

The accuracy determination unit 117 compares the determined positions with the correct positions stored in the correct position storage unit 132 to determine the accuracy (first accuracy) of the keypoint positions. The accuracy determination unit 117 compares not only the position but also the posture with the correct posture stored in the correct position storage unit 132 to evaluate the accuracy (second accuracy), and evaluates the first accuracy related to the position. The final accuracy can also be determined according to the accuracy and the second accuracy related to the attitude. The accuracy determination unit 117 can determine the first accuracy, the second accuracy, and/or the final accuracy for each part.

The accuracy determination unit 117 can also determine the accuracy (third accuracy) related to the movement of the part. The third degree of accuracy can be inferred by providing the classifier stored in the classifier storage unit 133 with the type of motion and the detected value from the IMU 311 . The accuracy determination unit 117 may determine the final accuracy according to the first to third accuracy.

The controller 118 controls the mounting device 30 to operate the tactile feedback device 313 of the mounting device 30 according to the first degree of accuracy, the second degree of accuracy and/or the final degree of accuracy. For example, the control unit 118 can operate the haptic feedback device 313 such that the lower the accuracy, the larger the movement. The control unit 118 can operate the haptic feedback device 313 by transmitting a command to the wearing device 30 corresponding to the part.

Controller 118 can also operate the haptic feedback device according to the second degree of accuracy.

The feedback output unit 119 can read feedback corresponding to accuracy from the feedback storage unit 134 and output it.

<Action>
FIG. 6 is a diagram for explaining the operation of the body movement support system of this embodiment.

The user terminal 10 outputs an instruction (S401). For example, in yoga, it can tell you what pose to do next. As described above, the user terminal 10 can output instructions stored in the instruction storage unit 131 in time.

The user terminal 10 transmits the action type to the wearable device 30 (S402). The user terminal 10 can acquire the action type corresponding to the instruction from the instruction storage unit 131 .

The wearable device 30 acquires the detection value of the IMU 311 (S421), gives the detection value from the IMU 311 to the classifier corresponding to the action type received from the user terminal 10, and infers the accuracy (S422). When the classifier receives the motion type as the feature amount, the motion type and the detection value may be given to the classifier. The wearing device 30 operates the haptic feedback device 313 according to the obtained accuracy (S423). For example, the operation of the haptic feedback device 313 can be controlled such that the lower the accuracy, the greater the feel.

Also, the mounting device 30 transmits the detection value from the IMU 311 to the user terminal 10 (S424).

The user terminal 10 activates the camera 105 to photograph the user (S403), and detects the first position of the key point from the photographed image (S404). Also, the user terminal 10 estimates the second position of the key point based on the detection value received from the wearable device 30 (S405). The user terminal 10 determines the position of the key point based on the first position and the second position (S406), and compares the determined position with the correct position to determine accuracy (S407). The user terminal 10 transmits a feedback command in which parameters corresponding to the accuracy are set to the wearing device 30 (S408), and the wearing device 30 operates the haptic feedback device 313 according to the command (S425). For example, a parameter can be set such that the lower the accuracy, the greater the feel.

The user terminal 10 also reads the feedback according to the accuracy from the feedback storage unit 134 and outputs the read feedback (S409).

As described above, according to the body movement support system of the present embodiment, the wearable device 30 worn on the user's body autonomously determines whether the motion of the wearable part is correct or wrong, and uses the tactile feedback device 313 to determine whether the user's motion is correct or not. It can give direct feedback to the body. As a result, the user can correct the positions and movements of the body parts to achieve a correct posture.

In addition, according to the body movement support system of the present embodiment, the user terminal 10 such as a smartphone can be used to determine the accuracy of the user's body posture from the captured image, and the advice corresponding to the accuracy can be returned. can be done. Also, even if the body keypoints are not captured by the camera, the IMU detection values from the wearable device 30 can be used to estimate the positions of the keypoints and determine the accuracy of the posture.

Although the present embodiment has been described above, the above-described embodiment is intended to facilitate understanding of the present invention, and is not intended to limit and interpret the present invention. The present invention can be modified and improved without departing from its spirit, and the present invention also includes equivalents thereof.

For example, in the present embodiment, the feedback (advice) according to accuracy is prepared in advance in association with the range of accuracy, but a generator that generates advice may be used. In this case, it is possible to create a generator based on the positions and postures of the keypoints and the advice by machine learning, and give the positions of the keypoints to the generator to generate the advice.

Further, in the present embodiment, the classifier is stored in the wearable device 30 in advance, but the classifier may be transmitted from the user terminal 10 to the wearable device 30 and installed. Also, the user terminal 10 may acquire a classifier for each part to be installed in the wearable device 30 and a classifier to be stored in the classifier storage unit 133 from a server device (not shown).

In the present embodiment, the user terminal 10 compares the positions and orientations of key points (parts) with the correct positions and correct orientations to obtain accuracy. A pose may be given to infer the first and second degrees of accuracy. The learning model in this case includes a learning model for obtaining the first degree of accuracy, which is learned using the position as input data and the first degree of accuracy as teacher data, and the posture (or the detection values of the 9 degrees of freedom of the IMU 311). and a learning model for obtaining a second degree of accuracy, which is learned using the input data and the second degree of correctness as teacher data.

111 instruction output unit 112 imaging unit 113 position detection unit 114 detection value acquisition unit 115 position estimation unit 116 position determination unit 117 accuracy determination unit 118 control unit 119 feedback output unit 131 instruction storage unit 132 correct position storage unit 133 classifier storage unit 134 feedback storage unit 311 IMU
312 communication unit 313 tactile feedback device 314 microcontroller 321 motion type acquisition unit 322 detection value acquisition unit 323 accuracy estimation unit 324 feedback unit 325 detection value transmission unit 326 command reception unit 331 classifier storage unit

Claims (4)

A wearable device worn on a user's body, comprising:
a sensor that detects motion;
a feedback device;
a storage device that stores a classifier for determining the accuracy of movement of the part to which the wearable device is worn based on input of the type of motion performed by the user and the detection value of the sensor;
a microcontroller, and
The microcontroller
acquiring the action type performed by the user from an information processing device;
obtaining the detected value from the sensor;
applying the motion and the detected value to the classifier to determine the accuracy;
operating the feedback device according to the accuracy;
A wearable device characterized by: A system for supporting a user's body movement,
comprising a wearable device worn on the user's body and an information processing device communicably connected to the wearable device,
The wearable device comprises:
a sensor that detects motion;
a feedback device;
with
The information processing device is
a storage unit that stores the correct positions of the key points of the body;
a photographing unit that acquires a photographed image of the user;
a detection unit that detects a first position candidate of the key point from the captured image;
a motion acquisition unit that acquires the motion detected by the sensor;
an estimator for estimating a second candidate location of the keypoint from the motion;
a position determination unit that determines a position of the keypoint based on the first and second position candidates;
an accuracy determining unit that compares the position and the correct position to determine the accuracy of the position;
a controller for controlling the wearable device to operate the haptic feedback device according to the accuracy;
to provide
A body movement support system characterized by: The body movement support system according to claim 2,
The sensor is an inertial measurement device that detects three-dimensional inertial motion with nine degrees of freedom,
The storage unit further includes a classifier that obtains a second accuracy of the motion of the keypoint, which is different from the first accuracy of the position, by inputting the type of motion of the body and the value detected by the sensor. remember,
The information processing device further includes an instruction unit that instructs the user to perform the specific action type,
The accuracy determination unit obtains the second accuracy by providing the type of motion that has issued the instruction and the detection value by the sensor to the classifier,
the control unit operating the feedback device according to the second accuracy;
A body movement support system characterized by: The body movement support system according to claim 3,
The information processing device is
a feedback storage unit that stores text, video, or audio feedback for the user in association with the first and second accuracy ranges;
a feedback output unit that reads and outputs the feedback corresponding to the first and second accuracies;
A body movement support system, further comprising:

Images