TWI823561B - Multiple sensor-fusing based interactive training system and multiple sensor-fusing based interactive training method - Google Patents

Abstract

A multiple sensor-fusing based interactive training system includes a posture sensor, a sensing module, an arithmetic module and a display module. The posture sensor is configured to sense posture data and myoelectric data related to the training motion of the user. The sensing module is configured to output limb torque data based on posture data, and output muscle group activation time data based on myoelectric data. The arithmetic module is configured to convert the limb torque data and the muscle group activation time data into torque-skeletal coordinate system and muscle strength eigenvalues-skeletal coordinate system based on the skeleton coordinate system, perform a fusion calculation on the torque-skeletal coordinate system and the muscle strength eigenvalues-skeletal coordinate system, calculate evaluation data for the training action based on the result of the fusion calculation, and determine the training motion corresponding to one of sporting motions based on the evaluation data. The display module is configured to display the evaluation data and the sporting motion.

Description

Multi-mode perception collaborative training system and multi-mode perception collaborative training method

The present disclosure relates to a training system, and in particular, to a multi-mode perception collaborative training system and a multi-mode perception collaborative training method.

At present, more and more people are exercising regularly, and gyms of all sizes can be found almost everywhere. There are a variety of fitness equipment in the gym. After many users go to the gym, they just follow the simple instructions on the fitness equipment and start using the fitness equipment without the guidance of a coach. There are endless cases of sports injuries caused by improper use of fitness equipment.

Or even with the guidance of a coach, when training independently without a coach, it may be due to incorrect posture, mismatch between the muscle groups used and the training movements, or the incorrect order of the muscle groups in the training movements. Cause sports injuries.

In addition, when athletes perform sports training, coaches or bystanders can only initially judge whether there is a risk of sports injuries based on the movement posture of the athletes. However, there is no way to quantify the effectiveness of sports as an indicator to provide coaches and athletes with ways to discuss improvements.

The multi-modal perception collaborative training system provided by the present disclosure includes a body posture sensor, a sensing module, a computing module and a display module. Posture sensors include inertial sensors and myoelectric sensors. The inertial sensors are used to sense multiple posture data related to the user's training movements. The myoelectric sensors are used to sense various posture data related to the user's training movements. Multiple EMG data. The sensing module is used to output limb torque data based on posture data, and output muscle group activation time data based on electromyographic data. The computing module is used to convert limb torque data and muscle group activation time data into torque-skeleton coordinates and muscle force characteristic values-skeleton coordinates based on skeleton coordinates, and perform fusion calculations on torque-skeleton coordinates and muscle force characteristic values-skeleton coordinates. , calculate evaluation data for training actions based on the results of the fusion algorithm, and determine based on the evaluation data that the training action corresponds to one of multiple known sports actions. The display module is used to display evaluation data and known motion actions.

The multi-modal perception collaborative training method provided by the present disclosure includes: sensing multiple posture data related to the user's training actions through the inertial sensor of the posture sensor, and using the myoelectric sensor of the posture sensor Sense multiple electromyographic data related to the user's training actions; output multiple limb torque data based on posture data, and output multiple muscle group activation time data based on the electromyographic data; convert these limb torque data based on the skeleton coordinate system is the moment-skeleton coordinate system; convert the muscle group activation time data into muscle force characteristic value-skeleton coordinate system according to the skeleton coordinate system; perform fusion calculation on the moment-skeleton coordinate system and muscle force characteristic value-skeleton coordinate system; according to the fusion calculation Calculate evaluation data for the training action as a result; determine based on the evaluation data that the training action corresponds to one of multiple known sports actions; and display the evaluation data and the known sports actions.

Some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The component symbols cited in the following description will be regarded as the same or similar components when the same component symbols appear in different drawings. These embodiments are only part of the disclosure and do not disclose all possible implementations of the disclosure.

FIG. 1 is an architectural diagram of a multi-modal perception collaborative training system 1 according to an embodiment of the present disclosure. Please refer to FIG. 1 . The multi-modal perception collaborative training system 1 includes a posture sensor 10 , a sensing module 20 , a computing module 30 and a display module 40 . The multi-modal perception collaborative training system 1 senses data related to the user's training actions through the body posture sensor 10 and processes the sensed data to determine that the user's ongoing training actions belong to multi-modal perception collaborative training. Which of the sports actions have been built into the training system 1, and instantly displays data that the user can refer to when performing training actions. The feedback from the multi-modal perception collaborative training system 1 can help users determine whether their posture during training is correct, whether the main muscle groups used match the training movements, and whether the order of the muscle groups in the training movements is correct, etc.

The body posture sensor 10 includes an inertial sensor 110, a myoelectric sensor 120a and a myoelectric sensor 120b. The inertial sensor 110 is used to sense multiple posture data related to the user's training movements. The inertial sensor 110 can see the user's training movements and is installed on the user's body or limbs. For example, when the user is running, the inertial sensor 110 can be disposed on the user's waist, outside of the leg, on the shoe, etc. Postural data related to running movements include stride frequency, stride length, vertical amplitude, body tilt angle, foot contact time, foot movement trajectory, etc. These postural data are all related to the economy of running, and It can effectively check the user's posture while running. In practice, the inertial sensor 110 is, for example, an acceleration sensor (G-Sensor), an angular velocity sensor, a gyroscope (gyro sensor), a cadence sensor and other dynamic sensors, but is not limited thereto.

The myoelectric sensor 120a and the myoelectric sensor 120b are used to sense a plurality of myoelectric data related to the user's training actions. The myoelectric sensor 120a and the myoelectric sensor 120b can observe the user's training movements, and are attached or worn on the user's core muscle groups or related muscle groups, such as the left and right thighs, left and right calves, left and right arms, and back muscles on both sides. Or the chest muscles on both sides, etc., to collect muscle electromyographic data. Practically speaking, the sensor that collects myoelectric data can be a contact or non-contact myoelectric sensor, which will not be described in detail here.

Please refer to Figure 1 again. The sensing module 20 is coupled to the posture sensor 10 for outputting a plurality of limb torque data according to the posture data and outputting a plurality of muscle group activation time data according to the electromyographic data. Specifically, the sensing module 20 performs spatial coordinate conversion according to the plurality of posture data sensed by the inertial sensor 110, and then outputs a plurality of limb torque data. In addition, the sensing module 20 performs dynamic electromyography (EMG) processing on the plurality of myoelectric data sensed by the myoelectric sensor 120a and the myoelectric sensor 120b, and then outputs multiple muscle group activation time data.

The computing module 30 is coupled to the sensing module 20 for converting the limb torque data and muscle group activation time data into torque-skeleton coordinates and muscle force characteristic values-skeleton coordinates according to the skeleton coordinate system, and calculates the torque-skeleton coordinates sum The muscle strength characteristic value-skeleton coordinates are fused and calculated, and the evaluation data of the training actions are calculated based on the results of the fusion calculation. Based on the evaluation data, it is determined that the training action corresponds to one of multiple known sports actions. Detailed description will be explained later. Practically speaking, the computing module 30 can be a central processing unit (CPU), a digital signal processor (Digital Signal Processing, DSP), multiple microprocessors (microprocessors), one or more combined digital signal processing Microprocessor, controller, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), any other type of integrated circuit, State machines, processors based on Advanced RISC Machine (ARM) and the like, but may not be limited thereto.

The display module 40 is coupled to the computing module 30 for displaying evaluation data and known motion actions. In one embodiment, the display module 40 can display evaluation data and known sports actions in text, icons, and charts. In another embodiment, the display module 40 can further display the evaluation data and known sports actions using sounds, images, text, icons, and charts. Practically speaking, the display module 40 may be an electronic device with a display function such as a monitor, a tablet computer, or a personal computer, or may be a display device on a treadmill, but is not limited thereto.

FIG. 2 is a schematic diagram of the computing module 30 performing fusion calculation in the multi-modal perception collaborative training system 1 according to an embodiment of the present disclosure. Please refer to Figures 1 and 2 at the same time. The skeleton coordinate system 31 is input into the calculation module 30, and multiple limb torque data and multiple muscle group activation time data are synchronized and successively input into the calculation module 30. The computing module 30 performs coordinate system conversion 301 on the multiple limb torque data according to the skeleton coordinate system 31, and converts the multiple limb torque data into a force-skeleton coordinate system; after that, the computing module 30 performs conversion 302 to convert the force-skeleton coordinate system The system is converted into a moment-skeleton coordinate system. At the same time, the computing module 30 performs coordinate system conversion 301 on the multiple muscle group activation time data according to the skeleton coordinate system 31, and converts the multiple muscle group activation time data into the muscle force activation time-skeleton coordinate system; after that, the computing module 30 performs conversion 303 to convert the muscle force activation time-skeleton coordinate system into the muscle force characteristic value-skeleton coordinate system.

Next, the computing module 30 continues to perform a fusion calculation 304 on the moment-skeleton coordinate system and the muscle force characteristic value-skeleton coordinate system, calculates evaluation data for the user's training actions based on the results of the fusion calculation 304, and outputs the evaluation data to the display Mod 40.

Specifically, the evaluation data is a type of data that is used to quantify the user's exercise effectiveness after the computing module 30 converts and fuses multiple limb torque data and multiple muscle group activation time data. In one embodiment, the computing module 30 uses K-Mean Clustering (KMC) to perform fusion calculation 304 on the moment-skeleton coordinate system and the muscle force characteristic value-skeleton coordinate system.

As shown in FIG. 1 , the computing module 30 determines that the user's training action corresponds to one of a plurality of known exercise actions based on the evaluation data, and outputs the determined known exercise action to the display module 40 . The display module 40 displays the evaluation data and the determined known sports actions to provide the user with the ability to observe whether the posture when performing training actions is correct, whether the main muscle groups used match the training actions, and the sequence of muscle group output of the training actions. Whether it is correct etc.

In one embodiment, the user can connect to social data through the multi-modal perception collaborative training system 1 described in the present disclosure, and upload the evaluation data and known sports actions to the social website or training website. Or users can interact with other users who perform the same known sports actions on social networking websites or training websites based on each other's evaluation data. Alternatively, users can conduct online discussions with remote sports coaches based on the evaluation data they upload to social networking sites or training websites.

In one embodiment, if the training action being performed by the user is the same as the known exercise action determined by the computing module, which means that the user's posture data and myoelectric data match the performance that the known exercise action should have, then the user You can confirm that the posture of the training movements you are performing is correct, the main muscle groups used match the training movements, and the order of the muscle groups in the training movements is correct.

In another embodiment, if the training action being performed by the user is different from the known exercise action determined by the computing module, it may mean that the user's posture data and myoelectric data do not completely match the known exercise action. display, the user can further confirm or adjust the posture of the training action he is performing, the main muscle groups used, or further confirm or adjust the correct order of the muscle groups in the training action, etc. Alternatively, the user can further confirm whether the inertial sensor 110, the myoelectric sensor 120a and the myoelectric sensor 120b are set in the correct position.

FIG. 3 is a block diagram of a multi-modal perception collaborative training system 1 according to an embodiment of the present disclosure. Please refer to FIG. 3. When the training action performed by the user does not require the use of training equipment, the posture sensor 10 includes a body inertia sensor 110a, a myoelectric sensor 120a and a myoelectric sensor 120b. The body inertial sensor 110a is the same as the inertial sensor 110 disposed on the user's body or limbs, the myoelectric sensor 120a and the myoelectric sensor 120b are attached or worn on the user's core muscles or related to the above. The myoelectric sensors above the muscle groups will not be described again here.

In one embodiment, when the training action performed by the user does not require the use of training equipment, the body inertia sensor 110a will output a plurality of posture data to the sensing module 20, and the sensing module 20 will After the multiple posture data sensed by 110a undergo spatial coordinate conversion 21, multiple limb torque data are output. In addition, the sensing module 20 performs dynamic EMG processing 22 based on the plurality of myoelectric data sensed by the myoelectric sensor 120a and the myoelectric sensor 120b, and then outputs a plurality of muscle group activation time data.

Since the body posture sensor 10 only includes the body inertia sensor 110a, the skeleton coordinate system input to the computing module 30 is the human skeleton coordinate system 31a. Multiple limb torque data and multiple muscle group activation time data are synchronized and successively input to the computing module 30. The computing module 30 converts multiple limb torque data and multiple muscle group activation time data according to the human skeleton coordinate system 31a. and fusion calculation, calculate the evaluation data, and output the evaluation data to the display module 40 .

In one embodiment, when only the body inertia sensor 110a is disposed on the user's body, the human skeleton coordinate system 31a corresponds to the user's skeleton, including human bones, muscles, etc. The user's skeleton can be obtained through the photography device 32 .

FIG. 4 is a block diagram of a multi-modal perception collaborative training system 1 according to another embodiment of the present disclosure. Please refer to Figure 4. When the training action performed by the user requires the use of training equipment, the posture sensor 10 includes a body inertia sensor 110a, an equipment inertia sensor 110b, a myoelectric sensor 120a and a myoelectric sensor 120b. The equipment inertia sensor 110b is visible to the user's training movements and is provided on the training equipment used by the user to perform training movements, such as a bat, a bat, etc., to sense multiple parameters related to the user's training movements. Posture data. The body inertial sensor 110a is the same as the inertial sensor 110 disposed on the user's body or limbs, the myoelectric sensor 120a and the myoelectric sensor 120b are attached or worn on the user's core muscles or The electromyographic sensors above the relevant muscle groups will not be described again here.

In one embodiment, when the training action performed by the user requires the use of training equipment, both the body inertia sensor 110a and the equipment inertia sensor 110b will output posture data to the sensing module 20, and the sensing module 20 will After the spatial coordinate conversion 21 is performed on the plurality of posture data sensed by the body inertia sensor 110a and the equipment inertia sensor 110b, a plurality of limb torque data are output. In addition, the sensing module 20 performs dynamic EMG processing 22 based on the plurality of myoelectric data sensed by the myoelectric sensor 120a and the myoelectric sensor 120b, and then outputs a plurality of muscle group activation time data.

Since the body posture sensor 10 includes both the body inertia sensor 110a and the equipment inertia sensor 110b, the skeleton coordinate system input to the computing module 30 is the human body/equipment skeleton coordinate system 31b. Multiple limb torque data and multiple muscle group activation time data are synchronized and successively input to the computing module 30. The computing module 30 converts multiple limb torque data and multiple muscle group activation time data according to the human body/equipment skeleton coordinate system 31b. Conversion and fusion calculations are performed, evaluation data is calculated, and the evaluation data is output to the display module 40 .

In one embodiment, when the body inertia sensor 110 a is disposed on the user's body and the equipment inertia sensor 110 b is disposed on the training equipment, the human body/equipment skeleton coordinate system 31 b not only corresponds to the user's body skeleton , including human bones, muscles, etc., and also corresponds to the equipment skeleton of training equipment. The user's body frame and the equipment frame of the training equipment can be obtained through the camera device 32 .

FIG. 5 is a block diagram of the multi-modal perception collaborative training system 1 updating training data and error data according to an embodiment of the present disclosure. Please refer to FIGS. 1 and 5 at the same time. The body posture sensor 10 includes at least two sensors disposed on the user's body. The sensors disposed on the user's body may be one of an inertial sensor and a myoelectric sensor. One or a combination thereof. As shown in FIG. 5 , the body posture sensor 10 includes a sensor 151 , a sensor 152 , a sensor 153 and a sensor 154 , wherein the sensor 151 , the sensor 152 , the sensor 153 and the sensor 154 The sensor 154 can be an inertial sensor or a myoelectric sensor respectively. In other words, the sensor 151, the sensor 152, the sensor 153 and the sensor 154 can all be inertial sensors or all myoelectric sensors. , some of the sensors 151, 152, 153 and 154 may also be inertial sensors, and the remaining sensors may be myoelectric sensors.

Please continue to refer to Figure 5. The body posture sensor 10 will output a plurality of sensing data to the sensing module 20, where the sensing data and the sensors included in the body posture sensor 10 are inertial sensors or data output by myoelectric sensors. corresponding. The sensing module 20 outputs multiple limb torque data and/or multiple muscle group activation time data according to the sensing data. The computing module 30 converts and integrates multiple limb torque data and multiple muscle group activation time data.

In one embodiment, the multi-modal perception collaborative training system 1 further includes a motion simulation model module 50 , a training data database 60 and a motion model database 70 . The motion simulation model module 50 is coupled to the computing module 30 . The training data database 60 is coupled to the computing module 30 and the motion simulation model module 50, and includes training data corresponding to a variety of known motion models and error data, where the error data is used to determine whether the user's training actions are incorrect actions or dangerous. action. The motion model database 70 is coupled to the motion simulation model module 50 and includes a plurality of known motion models. Multiple known motion models are pre-established motion models based on more than four inertial sensors or myoelectric sensors. In practice, the motion simulation model module 50 can be a micro-processor or an embedded controller, and the training data database 60 and the motion model database 70 can be stored in a memory or a hard disk. media, and this disclosure is not limited thereto.

The computing module 30 determines the user's exercise context based on the number of body posture sensors used by the user, such as running, aerobic exercise, core muscle training without equipment, etc. After the computing module 30 determines the user's motion situation, it performs motion model matching with the motion simulation model module 50 based on the motion situation. The motion simulation model module 50 selects one of the known motion models from the motion model database 70 based on the motion situation. One, the purpose is to find the motion model corresponding to the training action the user is performing.

After the motion simulation model module 50 selects a known motion model corresponding to the training action being performed by the user from the motion model database 70, the motion simulation model module 50 self-trains from the training data database 60 based on the selected known motion model. The training data corresponding to the selected known motion model is read and returned to the computing module 30 . The computing module 30 compares the evaluation data with the training data corresponding to the selected known motion model to calculate the similarity between the user's training actions and the selected known motion model.

When the similarity is greater than or equal to the similarity threshold (for example, 0.5), the computing module 30 determines that the user's training actions conform to the selected known motion model, and stores the evaluation data to the training data database 60 to update the corresponding selected The training data of the motion model is known and the AI model is established. At the same time, the computing module 30 will output the evaluation data and the known motion actions corresponding to the selected known motion model to the display module 40, and the display module 40 will further display the evaluation data and the known motion actions.

On the contrary, when the similarity is less than the similarity threshold (for example, 0.5), the computing module 30 determines that the user's training actions do not conform to all known motion models included in the motion model database 70 . At this time, the computing module 30 stores the evaluation data into the training data database 60 to update the error data. At the same time, the computing module 30 will output the evaluation data and the incorrect movement action information to the display module 40, and the display module 40 will further display the evaluation data and the incorrect movement action information. Among them, the error movement action message is used to remind the user to further confirm or adjust the training action posture they are performing, the main muscle groups used, or to further confirm or adjust the correct order of the muscle groups in the training action, etc. Alternatively, the user can further confirm whether the posture sensor 10 is set in the correct position.

FIG. 6 is a schematic diagram of using an inertial sensor 110 (such as an acceleration sensor) to determine deflection in a multi-mode perception collaborative training system according to an embodiment of the present disclosure; FIG. 7 is a schematic diagram of an implementation according to the present disclosure. The example shows a block diagram of using the inertial sensor 110 to determine the degree of deviation in the multi-modal perception collaborative training system. Please refer to Figure 6 first. When the inertial sensor 110 is fixed on the sensing carrier 16, and the sensing carrier 16 is set on the user's body, limbs or clothing by means of straps or wear, during movement During the process, the inertial sensor 110 may be deflected due to the user's body shaking or limb swinging. Once a relative offset d occurs between the inertial sensor 110 and the user's body or limbs, the accuracy of the posture data will be affected.

Please refer to Figures 6 and 7 at the same time. The inertial sensor 110 has an offset sensing unit 111 . When the sensing carrier 16 that fixes the inertial sensor 110 is attached to the user's body or limbs, the sensing data measured by the inertial sensor 110 is acceleration A1, and the offset sensing unit 111 records the acceleration A1 Set as standard reference value. When a relative offset d occurs between one side of the sensing carrier 16 on which the inertial sensor 110 is fixed and the user's body or limbs, the sensing data measured by the inertial sensor 110 is acceleration A2. Once the acceleration A2 and the acceleration A1 have an error e, the offset sensing unit 111 senses offset data corresponding to the acceleration A2 and outputs it to the sensing module 20 .

The sensing module 20 outputs the limb torque data to the computing module 30 according to the posture data and offset data. The computing module 30 compares the evaluation data with the training data corresponding to the selected known motion model, and determines the accuracy of the fixed inertial sensor 110 It is sensed whether the relative offset d between the carrier 16 and the user's body exceeds the offset threshold. When the relative offset d is not greater than the offset threshold, it means that the offset degree of the sensing vehicle 16 does not affect the accuracy of the posture data, and the multi-modal perception collaborative training system 1 will continue to operate. On the contrary, when the relative offset d is greater than the offset threshold, it means that the offset degree of the sensing vehicle 16 has affected the accuracy of the posture data, and the computing module 30 will output an abnormal signal to the display module 40, and the display module 40 A sensor setting error message is displayed.

When the user is using sports equipment, a force sensor can also be installed on the sports equipment to sense the user's force output status when performing training actions and detect whether the force output status of the left and right sides of the user's body is balanced. Once the force output on the left and right sides of the user's body is unbalanced, the multi-mode sensing collaborative training system described in the present disclosure can further issue a warning to remind the user to pay attention to the imbalance in the force output on the left and right sides of the body.

FIG. 8A is a schematic diagram of using the posture sensor and the force sensor 80 to calculate the left and right balance in the multi-modal perception collaborative training system according to an embodiment of the present disclosure. As shown in Figure 8A, when using the training equipment in Figure 8A, the user needs to push the flat plate on the fitness equipment with both feet at the same time. The multi-modal perception collaborative training system further includes a force sensor 80. The force sensor 80 is provided on the training equipment and coupled to the sensing module to sense multiple mechanical data corresponding to the user's training movements. (e.g. pressure sensing signal). When the user pushes the flat plate on the fitness equipment with both feet at the same time, the force sensor 80 senses a plurality of mechanical data (such as pressure sensing signals) corresponding to the user's training actions, where these mechanical data can be Corresponding to the total pressure sensing signal exerted by both feet of the user on the tablet at the same time, or corresponding to the pressure sensing signal of the user's left foot and the pressure sensing signal of the user's right foot respectively, depending on the force sensor The setting method or quantity of 80.

Please continue to refer to Figure 8A. When the user is using sports equipment, the myoelectric sensor 120a and the myoelectric sensor 120b are respectively disposed on the left half and the right half of the user's body, and the equipment inertia sensor 110b is disposed on the sports equipment. The myoelectric sensor 120a and the myoelectric sensor 120b are used to sense a plurality of left half electromyographic data (such as left thigh muscle signals) and a plurality of right half electromyographic data (such as right thigh muscle signal) corresponding to the user's training movements. muscle signals), and the equipment inertial sensor 110b is used to sense multiple postural data (such as acceleration signals) related to the user's training movements.

8B is another schematic diagram of using the posture sensor and the force sensor 80 to calculate the left and right balance in the multi-modal perception collaborative training system according to an embodiment of the present disclosure. As shown in Figure 8B, when the user is running, since the running action is to step on the ground with both feet in an alternating manner, that is, only one foot touches the ground at a time, the balance of the output of the legs will affect the effectiveness of running. , and even affect the safety when running. Therefore, it is necessary to detect the left and right balance of the user's legs separately.

The myoelectric sensor 120a and the myoelectric sensor 120c are respectively disposed on the left thigh and left calf of the user's body, and the myoelectric sensor 120b and the myoelectric sensor 120d are respectively disposed on the right thigh and right calf of the user's body, and in A body inertia sensor 110a is provided on the back side of the user's waist. The myoelectric sensor 120a and the myoelectric sensor 120c are used to sense a plurality of left half electromyographic data (such as left calf muscle signals) corresponding to the user's training movements. The myoelectric sensor 120b and the myoelectric sensor 120d is used to sense a plurality of right half electromyographic data (such as right calf muscle signals) corresponding to the user's training movements, and the body inertia sensor 110a is used to sense a plurality of right side electromyographic data related to the user's training movements. Individual posture data (such as stride length, stride frequency, vertical amplitude, body tilt angle, foot contact time, foot movement trajectory, etc. posture amplitude changes).

As shown in FIG. 8B , when the user uses the training equipment (i.e., the treadmill) in FIG. 8B , the user's feet step on the tracks of the treadmill in an alternating manner. The force sensor 80 is disposed on the track of the treadmill and coupled to the sensing module for sensing a plurality of force data (such as pressure sensing signals) corresponding to the user's training movements. Since the user does not step on both feet of the treadmill at the same time when running, the force sensor 80 can sense multiple mechanical data corresponding to the user's training movements (such as cadence pressure). measurement signals), especially these mechanical data respectively correspond to the cadence pressure sensing signal of the user's left foot and the cadence pressure sensing signal of the user's right foot.

FIG. 9 is a block diagram illustrating the use of the posture sensor and the force sensor 80 to calculate the left and right balance in the multi-modal perception collaborative training system according to an embodiment of the present disclosure. The sensing module 20 outputs multiple pressure data based on multiple mechanical data, and outputs multiple limb torque data based on multiple posture data. At the same time, the sensing module 20 outputs a plurality of left half muscle group activation time data and a plurality of right half muscle group activation time data respectively according to the plurality of left half muscle group activation time data and the plurality of right half half muscle group activation time data. The computing module 30 calculates the left half force output value based on multiple pressure data, multiple limb torque data, and multiple left half muscle group activation time data, and simultaneously based on multiple pressure data, multiple limb torque data, and multiple right half muscle groups. The starting time data is used to calculate the output value of the right half. Then, the computing module 30 performs another fusion calculation 361 on the left half output data and the right half output data, and calculates the left and right balance corresponding to the user's training movements based on the result of the other fusion calculation 361 .

When the left and right balance is less than or equal to the balance threshold, the computing module 30 determines the force balance of the left and right halves of the user's body, and continues to calculate the left and right balance corresponding to the user's training actions based on the results of another fusion calculation. . On the contrary, when the left and right balance is greater than the balance threshold, the computing module 30 determines that the force output of the left and right halves of the user's body is unbalanced, and the display module 40 displays the evaluation data and imbalance information to remind the user to pay attention to the left and right sides of the body. The side output force is unbalanced.

As shown in FIG. 1 , in one embodiment, the multi-modal perception collaborative training system 1 further includes a physiological information sensor 90 coupled to the sensing module 20 for sensing multiple physiological data of the user, such as using It collects information data such as body temperature, heart rate, respiration, skin moisture content, sweat, etc. when the user performs training movements, and transmits these physiological data to the computing module 30 . The computing module 30 can monitor the physiological conditions of the user when performing training actions based on the physiological data, and determine whether to issue a warning signal to remind the user to stop the training actions. In practice, the physiological information sensor 90 may be a contact or non-contact sensor that captures the above physiological values, and this disclosure is not limited thereto.

FIG. 10 is a schematic diagram of motion recognition using the body inertial sensor 110a and the myoelectric sensor 120a in the multi-modal perception collaborative training system according to an embodiment of the present disclosure. Please refer to Figure 10. In one embodiment, the user can use the multi-modal perception collaborative training system to perform motion recognition. The body inertia sensor 110a is arranged on the glove, and the myoelectric sensor 120a is arranged on the wrist strap.

When the user wants to perform the dart throwing action, the glove equipped with the body inertia sensor 110a can be put on the user's dart throwing hand, and the wrist strap equipped with the myoelectric sensor 120a can be fixed to the user's dart throwing hand. on the wrist. The body inertia sensor 110a is used to sense multiple posture data related to hand movements when the user throws darts. The myoelectric sensor 120a is used to sense multiple electromyographic data related to when the user throws darts.

In addition, the photography device 32 is used to obtain images of the user's posture when throwing darts. When the user throws a dart, the user's dart-throwing hand will first raise, extend back, and then throw the dart forward. Therefore, the posture image of the user when throwing the dart includes the movement trajectory of the user's hand. In addition, when the user throws a dart, the user's body may also use the force of body rotation to throw the dart. Therefore, the posture image of the user when throwing the dart also includes the rotation trajectory of the user's body.

Next, please refer to Figures 3 and 10 at the same time. The body inertia sensor 110a will output posture data to the sensing module 20. The sensing module 20 performs spatial analysis based on the multiple posture data sensed by the body inertia sensor 110a. After coordinate conversion 21, multiple limb torque data are output. In addition, the sensing module 20 performs dynamic EMG processing 22 based on the plurality of electromyographic data sensed by the myoelectric sensor 120a, and then outputs multiple muscle group activation time data.

The body posture sensor 10 includes both the body inertia sensor 110a and the myoelectric sensor 120a, so the skeleton coordinate system input to the computing module 30 is the human skeleton coordinate system 31a. Multiple limb torque data and multiple muscle group activation time data are synchronized and successively input to the computing module 30. The computing module 30 converts multiple limb torque data and multiple muscle group activation time data according to the human skeleton coordinate system 31a. and fusion calculation, calculate the evaluation data, and output the evaluation data to the display module 40 .

In addition, in one embodiment, the computing module 30 can output the posture and posture images of the user when throwing darts to the display module 40 (such as a mobile device). The user can view the posture and posture images through the display module 40, and even By slowing down the posture images, you can watch the movement of the hands and the rotation of the body. The computing module 30 can also be combined with the artificial intelligence AI motion analysis module to perform motion analysis on the user's posture and posture images when throwing darts, combine the evaluation data to quantify the effectiveness of the motion, and provide the user with adjustment suggestions for hand motions and body rotations. .

FIG. 11 is a flowchart of the multi-modal perception collaborative training method 2 according to an embodiment of the present disclosure. As shown in Figure 11, the multi-mode sensing collaborative training method 2 includes steps S310 to S380.

In step S310, multiple posture data related to the user's training actions are sensed through at least one inertial sensor of the plurality of posture sensors, and sensed through at least one myoelectric sensor of the posture sensors. Measure multiple electromyographic data related to the user's training movements. In step S320, a plurality of limb torque data are output according to the posture data, and a plurality of muscle group activation time data are output according to the electromyographic data.

In step S330, the limb torque data is converted into a moment-skeleton coordinate system according to the skeleton coordinate system. In step S340, the muscle group activation time data is converted into muscle strength characteristic value-skeleton coordinate system according to the skeleton coordinate system. This disclosure does not limit the execution order of step S330 and step S340, and step S330 and step S340 can also be performed simultaneously. In step S350, a fusion calculation is performed on the moment-skeleton coordinate system and the muscle force characteristic value-skeleton coordinate system. Among them, the fusion calculation is performed using K-Mean Clustering (KMC) on the moment-skeleton coordinate system and the muscle force eigenvalue-skeleton coordinate system. In step S360, evaluation data is calculated for the training action according to the results of the fusion calculation.

In step S370, it is determined based on the evaluation data that the training action corresponds to one of a plurality of known sports actions. In step S380, the evaluation data and known motion actions are displayed.

FIG. 12 is a flow chart for calculating evaluation data in a multi-modal perception collaborative training method according to an embodiment of the present disclosure. As shown in FIG. 12 , in step S321 , coordinate system conversion is performed on multiple limb torque data and multiple muscle group activation time data according to the skeleton coordinate system. In step S330, the limb torsion data is converted into a force-skeleton coordinate system according to the skeleton coordinate system, and then in step S331, the force-skeleton coordinate system is converted into a moment-skeleton coordinate system according to the skeleton coordinate system. In addition, in step S340, the muscle group activation time data is converted into muscle force activation time-skeleton coordinate system according to the skeleton coordinate system, and then in step S341, the muscle force activation time-skeleton coordinate system is converted according to the skeleton coordinate system. is the muscle force characteristic value-skeleton coordinate system. It should be noted that steps S330 to S331 and steps S340 to S341 are two separate processes, which can be performed simultaneously or separately.

Once the muscle force activation time-skeleton coordinate system and muscle force characteristic value-skeleton coordinate system are converted according to the skeleton coordinate system, in step S350, a fusion calculation is performed on the moment-skeleton coordinate system and the muscle force characteristic value-skeleton coordinate system. Next, in step S360, evaluation data is calculated for the training action according to the results of the fusion calculation.

In one embodiment, when the inertial sensor is disposed on the user's body, the skeleton coordinate system corresponds to the user's skeleton, and the user's skeleton can be obtained through the photography device. In one embodiment, when the inertial sensor is disposed on the training equipment, the skeleton coordinate system further corresponds to the skeleton of the training equipment, and the skeleton of the training equipment can be obtained through the photography device.

In one embodiment, the multi-modal perception collaborative training method further includes determining the user's movement context based on the number of posture sensors used by the user, and selecting one of a plurality of known movement models based on the movement context.

In one embodiment, the multi-modal perception collaborative training method further includes comparing the evaluation data with training data corresponding to the selected known motion model to calculate the similarity between the user's training actions and the selected known motion model. When the similarity is greater than or equal to the similarity threshold, it is determined that the user's training actions conform to the selected known motion model, the training data corresponding to the selected known motion model is updated with the evaluation data, and the evaluation data and known motion actions are displayed. . On the contrary, when the similarity is less than the similarity threshold, it is determined that the user's training actions do not conform to each of all known motion models, the evaluation data is updated to error data, and the evaluation data and error motion action messages are displayed.

In one embodiment, the inertial sensor has an offset sensing unit, and the inertial sensor is disposed on the user's body, and the multi-mode sensing collaborative training method further includes when the inertial sensor and the user's body When a relative offset is generated, multiple offset data are sensed, and multiple limb torque data are output based on multiple posture data and multiple offset data. Compare the evaluation data with training data corresponding to the selected known motion model to determine whether the relative offset between the inertial sensor and the user's body exceeds an offset threshold. When the relative offset is greater than the offset threshold, a sensor setting abnormal message is displayed.

In one embodiment, the multi-modal perception collaborative training method further includes sensing a plurality of mechanical data corresponding to the user's training actions through force sensors respectively provided on the training equipment. At least one inertial sensor senses a plurality of posture data corresponding to the user's training movements, and senses the user's training through at least two myoelectric sensors respectively provided on the left half and right half of the user's body. Multiple left half electromyography data and multiple right half electromyography data corresponding to the action. Output multiple pressure data based on multiple mechanical data, output multiple limb torque data based on multiple posture data, and output multiple left half muscle group activation times based on multiple left half EMG data and multiple right half EMG data. Data and activation time data of multiple right hemimuscle groups. The left half force output value is calculated based on multiple pressure data, multiple limb torque data, and multiple left half muscle group activation time data, and calculated based on multiple pressure data, multiple limb torque data, and multiple right half muscle group activation time data. Right half output value. Another fusion calculation is performed on the left half output data and the right half output data, and the left and right balance corresponding to the user's training movements is calculated based on the result of the other fusion calculation.

In one embodiment, when the left and right balance is less than or equal to the balance threshold, it is determined that the force output of the left and right halves of the user's body is balanced, and the left and right corresponding to the user's training movements are continuously calculated based on the results of another fusion calculation. balance. On the contrary, when the left and right balance is greater than the balance threshold, it is determined that the force output of the left and right halves of the user's body is unbalanced, and the evaluation data and imbalance information are displayed.

In one embodiment, the multi-modal sensing collaborative training method further includes sensing multiple physiological data of the user, such as the user's body temperature, heart rate, respiration, skin moisture content, sweat and other information data when performing training actions. Based on the These physiological data monitor the user's physiological conditions when performing training actions and determine whether to issue a warning signal to remind the user to stop training actions.

To sum up, the multi-modal sensing collaborative training system and multi-modal sensing collaborative training method provided by the present disclosure can provide gym users with the ability to know whether the postures of using fitness equipment are correct and whether the postures used are correct without the guidance of a coach. Whether the muscle groups match the training movements, or whether the order of the muscle groups in the training movements is correct, avoids sports injuries, and can also quantify the effectiveness of sports and become an indicator to provide coaches and athletes with ways to discuss improvements.

1: Multi-modal perception collaborative training system 10: Body posture sensor 110:Inertial sensor 110a: Body inertia sensor 110b: Equipment inertial sensor 111: Offset sensing unit 120a, 120b, 120c, 120d: Myoelectric sensor 151, 152, 153, 154: Sensor 16: Sensing vehicle 2: Multi-modal perception collaborative training method 20: Sensing module 21: Space coordinate conversion 22: Dynamic EMG processing 30:Computational module 301: Coordinate system conversion 302, 303: Conversion 304: Fusion Calculus 31: Skeleton coordinate system 31a:Human skeleton coordinate system 31b: Human body/equipment skeleton coordinate system 32: Photography installation 361: Another fusion calculus 40:Display module 50:Motion simulation model module 60:Training data database 70:Kinematic model database 80: Mechanical sensor 90: Physiological information sensor A1, A2: acceleration d: relative offset e: error S310～S380, S321, S330～S331, S340～S341: steps

FIG. 1 is an architectural diagram of a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram of the computing module performing fusion calculation in the multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 3 is a block diagram of a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 4 is a block diagram of a multi-modal perception collaborative training system according to another embodiment of the present disclosure. FIG. 5 is a block diagram of a multi-modal perception collaborative training system updating training data and error data according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram of using an inertial sensor to determine the degree of deviation in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 7 is a block diagram of using an inertial sensor to determine the degree of deviation in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 8A is a schematic diagram of using body posture sensors and force sensors to calculate left and right balance in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. 8B is another schematic diagram of using a posture sensor and a force sensor to calculate left and right balance in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 9 is a block diagram illustrating the use of body posture sensors and force sensors to calculate left and right balance in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 10 is a schematic diagram of motion recognition using body inertia sensors and myoelectric sensors in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. FIG. 11 is a flow chart of a multi-modal perception collaborative training method according to an embodiment of the present disclosure. FIG. 12 is a flow chart for calculating evaluation data in a multi-modal perception collaborative training method according to an embodiment of the present disclosure.

1: Multi-modal perception collaborative training system

10: Body posture sensor

110:Inertial sensor

120a, 120b: Myoelectric sensor

20: Sensing module

30:Computational module

40:Display module

50:Motion simulation model module

60:Training data database

70:Kinematic model database

80: Mechanical sensor

90: Physiological information sensor

Claims (20)

A multi-mode sensing collaborative training system includes: multiple posture sensors, including: at least one inertial sensor for sensing multiple posture data related to the user's training actions; and at least one myoelectric sensor , used to sense a plurality of myoelectric data related to the training action of the user; the sensing module is coupled to the posture sensors, and is used to output a plurality of limb torque data according to the posture data, and Output multiple muscle group activation time data according to the electromyographic data; the computing module is coupled to the sensing module to perform the following steps: convert the limb torque data into a torque-skeleton coordinate system according to the skeleton coordinate system ; Convert the muscle group activation time data into a muscle force characteristic value-skeleton coordinate system according to the skeleton coordinate system; perform a fusion calculation on the moment-skeleton coordinate system and the muscle force characteristic value-skeleton coordinate system; according to the fusion calculation Calculate evaluation data for the training action as a result; and determine based on the evaluation data that the training action corresponds to one of a plurality of known sports actions; and a display module coupled to the computing module to display the evaluation data and the already Know sports actions. The multi-modal perception collaborative training system as described in claim 1, wherein the computing module is further used to perform the following steps: convert the limb torsion data into a force-skeleton coordinate system according to the skeleton coordinate system, and then convert the force- The skeleton coordinate system is converted into the moment-skeleton coordinate system; and the muscle group activation time data is converted into the muscle force activation time-skeleton coordinate system according to the skeleton coordinate system, and then the muscle force activation time-skeleton coordinate system is converted into The muscle force characteristic value-skeleton coordinate system. The multi-modal perception collaborative training system as claimed in claim 1, wherein when the at least one inertial sensor is disposed on the user's body, the skeleton coordinate system corresponds to the user's body skeleton, and the user's The body skeleton can be obtained through a photographic device. The multi-mode sensing collaborative training system as claimed in claim 3, wherein when the at least one inertial sensor is disposed on the training equipment, the skeleton coordinate system further corresponds to the equipment skeleton of the training equipment, and the The equipment skeleton can be obtained through the photography device. The multi-modal perception collaborative training system as described in claim 1 further includes: a motion model database including a plurality of known motion models; and a motion simulation model module coupling the motion model database and the computing module; The computing module determines the movement situation of the user based on the number of posture sensors used by the user; The computing module is paired with the motion simulation model module based on the motion scenario, and the motion simulation model module selects one of the known motion models from the motion model database based on the motion scenario. The multi-modal perception collaborative training system as described in claim 5 further includes: a training data database coupled to the computing module, the training data database including training data corresponding to each of the known motion models; Error data; wherein the computing module compares the evaluation data with the training data corresponding to the selected known motion model to calculate the similarity between the user's training actions and the selected known motion model. The multi-modal perception collaborative training system as described in claim 6, wherein when the similarity is greater than or equal to a similarity threshold, the computing module determines that the training action of the user conforms to the selected known motion model, and The evaluation data is stored in the training data database to update the training data corresponding to the selected known motion model, and the display module displays the evaluation data and the known motion; when the similarity is less than a similarity threshold When, the computing module determines that the user's training actions do not conform to the known motion models, and stores the evaluation data to the training data database to update the error data, the display module displays the evaluation data and Wrong motion action message. The multi-modal perception collaborative training system as claimed in claim 6, wherein the at least one inertial sensor has a deflection sensing unit, and the at least one inertial sensor is disposed on the user's body for when the at least one An inertial sensor and the user's When a relative offset occurs between the bodies, a plurality of offset data is sensed, and the sensing module outputs the limb torque data according to the posture data and the offset data; wherein the computing module compares the evaluation The data and the training data corresponding to the selected known motion model are used to determine whether the relative offset between the at least one inertial sensor and the user's body exceeds an offset threshold; wherein when the When the relative offset is greater than the offset threshold, the display module displays a sensor setting abnormal message. The multi-modal perception collaborative training system as claimed in claim 1, further comprising: a mechanical sensor coupled to the sensing module and disposed on the training equipment to sense the training action corresponding to the user. A plurality of mechanical data; wherein the at least one inertial sensor is disposed on the training equipment to sense the posture data corresponding to the training movements of the user; wherein the at least one myoelectric sensor is disposed respectively The left and right halves of the user's body are used to sense a plurality of left half electromyographic data and a plurality of right half electromyographic data corresponding to the user's training movements; wherein the sensing module A plurality of pressure data are output according to the mechanical data, and the limb torque data are output according to the posture data; wherein the sensing module outputs a plurality of pressure data according to the left half electromyography data and the right half electromyography data respectively. The left half muscle group activation time data and the plurality of right half muscle group activation time data; wherein the computing module is further used to perform the following steps: according to the pressure data, the limb torque data and the left half muscle group The group activation time data is used to calculate the left half output value; the right half output value is calculated based on the pressure data, the limb torque data and the right half muscle group activation time data; the left half output data and the right half output data are calculated Another fusion calculation; and calculating the left and right balance corresponding to the training action of the user according to the result of the other fusion calculation. The multi-modal perception collaborative training system as described in claim 9, wherein when the left and right balance is less than or equal to a balance threshold, the computing module determines that the left half and the right half of the user's body are balanced, and Continue to calculate the left and right balance corresponding to the training action of the user based on the result of the other fusion calculation; and when the left and right balance is greater than the balance threshold, the computing module determines the left half of the user's body And the right half output is unbalanced, and the display module displays the evaluation data and imbalance information. A multi-modal perception collaborative training method includes: sensing multiple posture data related to the user's training actions through at least one inertial sensor of multiple posture sensors; The myoelectric sensor senses a plurality of electromyographic data related to the training action of the user; outputs a plurality of limb torque data according to the posture data, and outputs a plurality of muscle group activation time data according to the electromyographic data; Convert these limb torsion data into moments according to the skeleton coordinate system - skeleton base coordinate system; convert the muscle group activation time data into a muscle force characteristic value-skeleton coordinate system according to the skeleton coordinate system; perform a fusion calculation on the moment-skeleton coordinate system and the muscle force characteristic value-skeleton coordinate system; according to the The result of the fusion calculation is to calculate evaluation data for the training action; determine based on the evaluation data that the training action corresponds to one of a plurality of known sports actions; and display the evaluation data and the known sports actions. The multi-modal perception collaborative training method as described in claim 11 further includes: converting the limb torsion data into a force-skeleton coordinate system according to the skeleton coordinate system, and then converting the force-skeleton coordinate system into the moment-skeleton coordinate system coordinate system; and convert the muscle group activation time data into muscle force activation time-skeleton coordinate system according to the skeleton coordinate system, and then convert the muscle force activation time-skeleton coordinate system into the muscle force characteristic value-skeleton coordinate system . The multi-modal perception collaborative training method as claimed in claim 11, wherein when the at least one inertial sensor is disposed on the user's body, the skeleton coordinate system corresponds to the user's body skeleton, and the user's The body skeleton can be obtained through a photographic device. The multi-modal sensing collaborative training method as described in claim 13, wherein when the at least one inertial sensor is installed on the training equipment, the skeleton coordinate system further corresponds to the equipment skeleton of the training equipment, and the The equipment skeleton can be obtained through the photography device. The multi-modal perception collaborative training method as described in claim 11, further comprising: determining the user's movement situation based on the number of body posture sensors used by the user; selecting a plurality of known movement models based on the movement situation One of them. The multi-modal perception collaborative training method as described in claim 15 further includes: comparing the evaluation data with the training data corresponding to the selected known motion model to calculate the training action of the user and the selected Similarity of known motion models. The multi-modal perception collaborative training method as described in claim 16, wherein when the similarity is greater than or equal to a similarity threshold, it is determined that the training action of the user conforms to the selected known motion model, and is updated with the evaluation data The training data corresponding to the selected known motion model displays the evaluation data and the known motion actions; when the similarity is less than a similarity threshold, it is determined that the user's training actions do not conform to the known motions. The model is updated with the evaluation data as error data, and displays the evaluation data and error motion action information. The multi-mode sensing collaborative training method as described in claim 16, wherein the at least one inertial sensor has an offset sensing unit, and the at least one inertial sensor is disposed on the user's body, and the multi-mode sensing The collaborative training method further includes: when a relative offset occurs between the at least one inertial sensor and the user's body, sensing a plurality of offset data, and outputting according to the posture data and the offset data The limb torque data; and comparing the evaluation data with the training data corresponding to the selected known motion model, determining the relative offset between the at least one inertial sensor and the user's body Whether an offset threshold is exceeded; when the relative offset is greater than the offset threshold, a sensor setting abnormal message is displayed. The multi-modal perception collaborative training method as described in claim 11 further includes: sensing a plurality of mechanical data corresponding to the training movements of the user through force sensors respectively provided on the training equipment; The at least one inertial sensor on the training equipment senses the body posture data corresponding to the training movements of the user; through the at least one muscle respectively provided on the left and right half of the user's body The electrical sensor senses a plurality of left half electromyography data and a plurality of right half electromyography data corresponding to the training action of the user; outputs a plurality of pressure data according to the mechanical data, and outputs a plurality of pressure data according to the posture data Output the limb torque data; output a plurality of left half muscle group activation time data and a plurality of right half muscle group activation time data respectively according to the left half muscle electromyography data and the right half half muscle electrical data; according to the pressure data , the limb torque data and the left half muscle group activation time data are used to calculate the left half output value; the right half output value is calculated based on the pressure data, the limb torque data and the right half muscle group activation time data; The left half output data and the right half output data perform another fusion calculation; and the left and right balance corresponding to the training action of the user is calculated according to the result of the other fusion calculation. The multi-modal perception collaborative training method as described in claim 19, wherein when the left and right balance is less than or equal to a balance threshold, it is determined that the left half and the right half of the user's body are in balance, and the force balance is continued based on the other. The result of a fusion calculation calculates the left and right balance corresponding to the user's training action; and when the left and right balance is greater than the balance threshold, it is determined that the left half and the right half of the user's body are unbalanced, Displays the evaluation data and imbalance message.

Images