TWI464696B - Method and system for motion comparison - Google Patents

Description

Motion comparison method and action comparison system

The disclosure relates to a multi-modal action stream comparison method and system for adjusting weights by weight vectors and computing according to a stream node comparison algorithm.

In a commercially available somatosensory fitness game, the virtual personal trainer course will edit the personalized training course in response to the fitness training session entered by the user. These training sessions will give the user a score based on the user's actions and the actions of the virtual personal trainer course. Users can know if their actions are correct based on this score. However, the scores given are only for the specific fixed-point actions of the user, and the user cannot be given a smooth experience. In addition, because the training objectives set personal needs, users expect to try to figure out and follow the virtual personal trainer movement, can strengthen and train the movements of the hands, feet, specific parts of the body and the center of gravity. On the other hand, the user may also have the need to train the lower limb endurance (for example, the height of the station and the length of the exercise time), so as to improve the heart rate to achieve fitness.

Therefore, how to provide a continuous stream action comparison method that meets the needs of users is an issue of concern to those skilled in the art.

In order to solve the above problem, the exemplary embodiment of the present disclosure proposes a The weight vector adjustment weight and the multi-modal action stream comparison method and system for calculating the stream node comparison algorithm. Thereby, the continuous motion compliance of the individual body training parts of the user is improved, and the correct posture of the user is effectively improved.

An exemplary embodiment of the present disclosure provides an action comparison system, including a user action capture unit, a virtual trainer action information unit, a first calculation unit, a multi-dimensional weight filter, a second calculation unit, and a mapping unit. The user action capturing unit is configured to acquire a first stream action of the user and a plurality of nodes of the first stream action according to the plurality of images of the user. Each node of the first stream action includes a plurality of coordinates, and each node of the first stream action belongs to a body part. The virtual trainer action information unit is a plurality of nodes for providing a second stream action and a second stream action of a virtual trainer. Each node of the second stream action includes a plurality of coordinates, and each node of the second stream action belongs to one of the body parts. The first calculation unit is coupled to the user action extraction unit and the virtual trainer action information unit, and is configured to calculate a coordinate of the node of the first stream action and a node of the second stream action according to a stream node comparison algorithm. Multiple relationship values between the coordinates. The multi-dimensional weight filter is configured to obtain a plurality of weights by the weight vector according to the body part, the plurality of types of motion preset by the user, and the time information of the image. The second calculus unit is coupled to the first calculus unit and the multi-dimensional weight filter, and is configured to generate a comparison result value according to the result of multiplying the relationship values by the weights. The mapping unit is coupled to the second calculation unit for mapping the comparison result value to a similarity.

In another aspect, the disclosure proposes an action comparison method. Suitable for an electronic device. The action comparison method includes: obtaining a first stream action of the user and a plurality of nodes of the first stream action according to the plurality of images of the user, wherein each node of the first stream action includes a plurality of coordinates, And each node of the first stream action belongs to a body part; a second stream action of the virtual trainer and a plurality of nodes of the second stream action are provided, wherein each node of the second stream action comprises a plurality of coordinates And each node of the second stream action belongs to one of the body parts; the coordinates of the node of the first stream action and the coordinates of the node of the second stream action are calculated according to a stream node comparison algorithm a plurality of relationship values; a plurality of weights are obtained by the weight vector according to the body part, the type of motion preset by the user, and the time information of the image; and the result of multiplying the relationship value by the weight is generated to generate a Aligning the result values; and, mapping the result values to a similarity.

As described above, the action comparison method and the action comparison system proposed by the exemplary embodiment of the present disclosure can be used to allow a user to exercise a specific body part according to a user's setting when performing a whole body exercise. According to the similarity between the user and the virtual coach's action, the user can understand the correctness of the continuous action and make the exercise more effective.

FIG. 1 is a schematic diagram showing a virtual coach guiding a user's motion according to an exemplary embodiment.

Referring to FIG. 1 , the motion comparison device 100 includes a screen 120 and a sensor 130 . The sensor 130 is for sensing the action of the user 140. For example, sensor 130 includes a camera or an infrared sensor. When the user 140 starts After the action comparison device 100, a virtual coach 132 is displayed on the screen 120. The virtual coach 132 will begin to demonstrate one or more actions whereby the user 140 learns the action. An action message is also displayed on the screen 120, including the similarity of the action or other related information (eg, action type, time, rating, etc.), whereby the user 140 can know whether his or her action is correct.

2 is a block diagram showing an action comparison device, according to an exemplary embodiment.

Referring to FIG. 2, the action comparison device further includes an action comparison system 200 including a user action extraction unit 210, a virtual coach action information unit 220, a first calculation unit 230, a multi-dimensional weight filter 240, and a The second calculation unit 250 and the mapping unit 260. In this exemplary embodiment, each unit in the motion comparison system 200 is implemented as a circuit.

The sensor 130 captures a plurality of images of the user 140 and transmits the images to the user action capturing unit 210. The user action capture unit 210 obtains a stream action (also referred to as a first stream action) of the user 140 and a plurality of nodes on the stream action according to the image, and each node belongs to a body part ( For example, head or hand). In an exemplary embodiment, the images include information on brightness and depth of field, and each node includes a plurality of coordinates representing a three-dimensional space. In other words, the streaming action of the user 140 is a continuous action made in three dimensions.

The virtual trainer action information unit 220 provides a streaming action (also referred to as a second streaming action) of a virtual coach and a plurality of nodes of the streaming action. Similarly, each node of the virtual instructor's streaming action belongs to a body part, and each node will include multiple coordinates. The first calculation unit 230 calculates the coordinates of the user's node according to a stream node comparison algorithm. Multiple relationship values between the coordinates of the nodes of the virtual trainer. These relationship values are used to indicate whether the action of the user 140 is similar to the action of the virtual coach 132.

In particular, the multi-dimensional weight filter 240 obtains a plurality of weights according to the body part to which the nodes belong, the type of motion preset by the user, and the time information of the image sensed by the sensor 130. For example, if the user wants to strengthen the muscles of the lower limbs, the weight of the body parts of the lower limbs will be larger (relative to the weight corresponding to the body parts of the upper limbs). For example, if the user presets the type of exercise to focus on the lower body of the foot, the multi-dimensional weight filter 240 will filter the weight corresponding to the weight of the lower body of the dance from a database. Alternatively, if the user wants to exercise the body part of the lower limb during the first five minutes of the exercise, the weight of the body part of the lower limb in the first five minutes will be relatively large.

The second calculus unit 250 multiplies the weights by the relationship values, and generates a comparison result value based on the multiplied result. The mapping unit 260 is coupled to the second calculation unit 250 for mapping the comparison result value to a similarity. For example, if the action of the user 140 is similar to the action of the virtual coach 132, the similarity will be higher. This similarity is displayed on the screen 120, letting the user 140 know if his or her action is correct. In particular, for each image captured by the sensor 130, the motion comparison system 200 produces a similarity and displays the similarity on the screen 120. Thereby, the user 140 can know at any time whether his or her continuous action is correct.

The operation of each unit in the motion comparison system 200 will be described in detail below by way of an exemplary embodiment.

First, the user 140 inputs one or more settings to the action comparison device 100. set. For example, the user 140 may select a sport type, exercise time, exercise intensity, or body part to be strengthened. Next, the sensor 130 continuously acquires the streaming image of the user. The streaming image includes a plurality of images, and each image corresponds to different time information. In an exemplary embodiment, the time information can be represented as the second few seconds. Alternatively, the time information can be used to represent the first image in the streaming image.

3 is a schematic diagram showing nodes of a plurality of body parts, according to an exemplary embodiment.

Referring to FIG. 3 , the user action capturing unit 210 obtains a streaming operation of the user 140 based on the streaming image acquired from the sensor 130 and calculates a plurality of nodes. In this exemplary embodiment, the number of these nodes is 15, and these nodes are 15 body parts that belong to the user, respectively. These nodes are a head node 1201, a neck node 1202, a trunk center node 1203, a right shoulder node 1204, a right hand axis node 1205, a right hand node 1206, a right hip node 1207, a right knee node 1208, a right foot node 1209, and a left shoulder node 1210. The left hand axis node 1211, the left hand node 1212, the left hip node 1213, the left knee node 1214, and the left foot node 1215. The nodes 1201 to 1215 are located on the main joint of the human body, and the user action capturing unit 210 can determine the continuous movement of the user from the streaming image, and calculate the nodes 1201 to 1215 from the continuous movement. However, in other example embodiments, the user action capture unit 210 may also calculate a greater or lesser number of nodes. Alternatively, the user action capturing unit 210 may also calculate nodes of other body parts, and the disclosure is not limited thereto. In this exemplary embodiment, each node includes coordinates in the x direction, coordinates in the y direction, and coordinates in the z direction. Here, the three coordinates of a node of the user are expressed as (x _u (t), y _u (t), z _u (t)), where t represents time information. On the other hand, the vector U is used here to represent the coordinates of all nodes of the user over a period of time.

On the other hand, the virtual instructor action information unit 220 provides a streaming action of the virtual instructor, a plurality of nodes of the streaming action and the coordinates of the nodes. For example, the motion comparison device 200 includes a database that stores stream images of a plurality of motion types. Each stream image includes multiple images at different times. Each image consists of 15 nodes, which belong to the above 15 body parts. Each node of the virtual trainer also includes coordinates in the x direction, coordinates in the y direction, and coordinates in the z direction. Here, corresponding to a time information t, the three coordinates of a node on the virtual trainer are represented as (x _v (t), x _v (t), z _v (t)), and the vector V is used to indicate The coordinates of all the nodes on the virtual trainer for a period of time.

4 is a schematic diagram showing the operation of an action comparison system according to an exemplary embodiment.

Referring to FIG. 4 , the user action capture unit 210 transmits the vector U to the first calculation unit 230 , and the virtual coach action information unit 220 transmits the vector V to the first calculation unit 230 . For each body part and each time information t, the first calculation unit 230 calculates the relationship value between the vector U and the vector V according to a stream node comparison algorithm. The stream node comparison algorithm can be a dynamic time warping (DTW), an Euclidean distance (ED) algorithm or a coefficient of correlation (CC) algorithm. . The basic principle of DTW is to compare the degree of similarity between two time series. The principle of ED is to calculate the distance between two points in a space. If the distance is larger, the two points are not similar. CC is used to determine how much the relationship between two time series is high. However, those skilled in the art should understand the content of these stream alignment algorithms and are not detailed here.

The stream node alignment algorithm employed by the first calculus unit 230 can be expressed as a function H _n (), which is a relationship value used to calculate the nth body part. Taking the Euclid distance algorithm and the head node as an example, the first calculus unit 230 calculates the three coordinates of the head node in the user according to the time information t (x _u (t), y _u (t), z _u (t)) The Euclid distance between the three coordinates (x _v (t), x _v (t), z _v (t)) of the head node in the virtual trainer, thereby generating a relationship value of 401 ( Expressed as H _{head node} (U, V)). Similarly, for other body parts, the first calculus unit 230 also calculates a corresponding relationship value (eg, relationship values 402 to 415).

On the other hand, the multi-dimensional weight filter 240 filters a plurality of weights by weight vectors according to the body part 416, the motion type 417, and the time information 418 set by the user. Here, the weight vector includes three dimensions of the body part 416, the motion type 417, and the time information 418. For example, a body part 416 can be represented by an integer from 1 to 15 (eg, a "1" is used to indicate a head node); and a different motion category 417 can also be represented as a discrete plurality of values (eg, "0" is used to represent the tai chi Boxing, with "1" for the national standard dance); time information 418 can be expressed in seconds of the film. The weight vector composed of the body part 416, the motion type 417, and the time information 418 can determine a position on the three-dimensional matrix 500 (as shown in FIG. 5). A weight on the three-dimensional matrix 500 can be represented by W _mtn , which represents the weight of a certain motion category (m), a certain time information (t) and a certain body part (n). Where m and n are positive integers and t is a real number. In another example embodiment, the weight vector may also include more dimensions. For example, the weight vector may include difficulty or a dimension preferred by the user, and the disclosure is not limited thereto.

Referring to FIG. 5, if the user does not want to consider the motion type 417 and the time information 418, the multi-dimensional weight filter 240 can obtain the weight based on the body part 416 (corresponding to the N-axis). For example, the national standard dance hand training, the weight corresponding to the hand node is relatively large; if the national standard dance foot training, the weight corresponding to the foot node is relatively large. On the other hand, if the user does not want to consider the time information 418, the multi-dimensional weight filter 240 can obtain the weight (corresponding to the M-N plane) based only on the motion category 417 and the body part 416. For example, Tai Chi Chuan, the weight of the node that focuses on the center of gravity of the hand and the lower part of the disk is relatively large; the weight of the node that focuses on the lower part of the foot is relatively large. If the user does not want to consider the sport category 417, the multi-dimensional weight filter 240 can obtain weights (corresponding to the N-T plane) based only on the body part 416 and the time information 418. For example, the user can set the movement of the upper body to be emphasized in the first half of the movement, and the weight corresponding to the upper body node is relatively large; while the movement of the lower body is emphasized in the second half of the movement, the weight corresponding to the lower body node is relatively large.

Here, the weights generated by the multi-dimensional weight filter 240 are real numbers, and the values of these weights are greater than or equal to 0.1 and less than or equal to 2. The greater the value of the weight, the more important the corresponding body part is. However, if the value of the weight is set too large (more than 2), the user's inaccurate motion will be amplified, which may result in the user not easily obtaining a high degree of similarity. However, in other exemplary embodiments, these weights may also be set to other values, and the disclosure is not limited thereto.

Referring back to FIG. 4, the multi-dimensional weight filter 240 transmits the obtained weights to the second calculation unit 250. The second calculus unit 250 will separate these weights Multiply the relationship values 401~415 and add the multiplied results to obtain a comparison result value. For example, the second calculation unit 250 generates a comparison result value according to the following equation (1).

Node is a collection of all body parts. In this exemplary embodiment, the second calculus unit 250 calculates a comparison result value for each image of the user's streaming action and each image of the virtual coach. The second calculation unit 250 transmits the comparison result values to the mapping unit 260.

FIG. 6 is a schematic diagram showing the operation of a mapping unit according to an exemplary embodiment.

Referring to FIG. 6, the mapping unit 260 receives a plurality of comparison result values, and obtains a maximum alignment result value and a minimum alignment result value from the comparison result values. In Fig. 6, the maximum alignment result value is expressed as the following equation (2), and the minimum alignment result value is expressed as the following equation (3).

Mapping unit 260 generates a similarity domain based on the minimum alignment result value and the maximum alignment result value. Mapping unit 260 also defines a similarity value field (eg, 0 to 100). The mapping unit 260 maps a comparison result value from the similarity definition domain to the phase based on the minimum alignment result value and the maximum alignment result value. Similarity value range. For example, when the stream node alignment algorithm employed by the first calculus unit 230 is DTW or ED, the maximum alignment result value will be mapped to 0, and the minimum alignment result value will be mapped to 100. On the other hand, when the stream node alignment algorithm employed by the first calculation unit 230 is CC, the maximum alignment result value is mapped to 100, and the minimum alignment result value is mapped to 0. When an alignment result value is between the minimum alignment result value and the maximum alignment result value, the comparison result value is mapped to a similarity between 0 and 100. The second calculus unit 250 can generate a similarity between 0 and 100 in a linear or non-linear manner, and the disclosure is not limited thereto. Moreover, in other exemplary embodiments, the mapping unit 260 may also define other similarity domain (for example, 0 to 10), and the disclosure is not limited thereto. Finally, mapping unit 260 transmits the generated similarity to screen 120.

FIG. 7 is a flow chart showing a method of motion comparison according to an exemplary embodiment.

Referring to FIG. 7, in step S702, a first stream action of the user and a plurality of nodes of the first stream action are acquired according to the plurality of images of the user, wherein each node of the first stream action includes a plurality of nodes. Coordinates, and each node of the first stream action belongs to a body part. In step S704, a plurality of nodes of the second stream action and the second stream action of the virtual trainer are provided, wherein each node of the second stream action includes a plurality of coordinates, and each node of the second stream action One of the above body parts. In step S706, a plurality of relationship values between the coordinates of the node of the first stream action and the coordinates of the node of the second stream action are calculated according to a stream node comparison algorithm. In step S708, a plurality of weights are obtained by the weight vector based on the body part, the type of motion preset by the user, and a time information of the image. In step In step S710, a result of multiplying the relationship values by these weights respectively produces a comparison result value. In step S712, the comparison result value is mapped to one similarity. However, the steps in FIG. 7 have been described in detail above, and will not be described again here.

In an exemplary embodiment, the steps in Figure 7 can be implemented as multiple programs. These codes are stored in a memory of an electronic device. The electronic device includes a processor for executing the code. The electronic device can be implemented as a computer, a television, a game console, or any embedded system. Alternatively, the steps in Figure 7 can be implemented as one or more circuits. The disclosure does not limit the steps in Figure 7 to be implemented in a software or hardware manner.

The action comparison system and the action method proposed by the exemplary embodiment, in particular, a multi-modal action stream comparison system and method, can select multiple by weight vector according to time information, body part and motion type Weights, and these weights are used to calculate a similar value. Thereby, the user can strengthen the body part to be exercised according to the instruction of the virtual coach, and improve the effect of the exercise.

While the present invention has been described above in terms of the preferred embodiments thereof, it is not intended to limit the invention, and the invention may be modified and modified without departing from the spirit and scope of the invention. The patent protection scope of the invention is subject to the definition of the scope of the patent application attached to the specification.

100‧‧‧ action comparison device

120‧‧‧ screen

130‧‧‧Sensor

132‧‧‧Virtual Coach

140‧‧‧Users

200‧‧‧Action comparison system

210‧‧‧User action capture unit

220‧‧‧Virtual Coach Action Information Unit

230‧‧‧First calculation unit

240‧‧‧Multidimensional weight filter

250‧‧‧Second calculation unit

260‧‧‧ mapping unit

1201~1215‧‧‧ nodes

401~415‧‧‧ relationship value

416‧‧‧ body parts

417‧‧‧Types of sports

418‧‧‧Time Information

500‧‧‧Three-dimensional matrix

S702, S704, S706, S708, S710, S712‧‧‧ steps of the action comparison method

FIG. 1 is a schematic diagram showing a virtual coach guiding a user's motion according to an exemplary embodiment.

2 is a block diagram showing an action comparison device, according to an exemplary embodiment.

3 is a schematic diagram showing nodes of a plurality of body parts, according to an exemplary embodiment.

4 is a schematic diagram showing the operation of an action comparison system according to an exemplary embodiment.

FIG. 5 is a schematic diagram showing a three-dimensional matrix according to an exemplary embodiment.

FIG. 6 is a schematic diagram showing the operation of a mapping unit according to an exemplary embodiment.

FIG. 7 is a flow chart showing a method of motion comparison according to an exemplary embodiment.

S702, S704, S706, S708, S710, S712‧‧‧ steps of the action comparison method

Claims (14)

An action comparison system includes: a user action capture unit configured to acquire a first stream action of the user and a plurality of nodes of the first stream action according to a plurality of images of a user, Each of the nodes of the first stream action includes a plurality of coordinates, and each of the nodes of the first stream action belongs to a body part; a virtual coach motion information unit is configured to provide a virtual coach a second stream action and a plurality of nodes of the second stream action, wherein each of the nodes of the second stream action comprises a plurality of coordinates, and each of the nodes of the second stream action belongs to One of the body parts; a first calculation unit coupled to the user action extraction unit and the virtual trainer action information unit for calculating the first stream according to a stream node comparison algorithm a plurality of relationship values between the coordinates of the nodes of the action and the coordinates of the nodes of the second stream action; a multi-dimensional weight filter for determining the body parts, the user Preset The plurality of motion types and the plurality of time information of the images are obtained by using a weight vector to obtain a plurality of weights; a second calculation unit is coupled to the first calculation unit and the multi-dimensional weight filter for The result of multiplying the weights by the relationship values respectively produces a comparison result value; and a mapping unit coupled to the second calculation unit for mapping the comparison result value to a similarity. The action comparison system as described in claim 1, wherein the number of the nodes of the user is 15, and the nodes of the virtual coach are The number is 15. The action comparison system of claim 1, wherein the weights are real numbers, and the weights have a value greater than or equal to 0.1 and less than or equal to 2. The action comparison system of claim 2, wherein the operation of the comparison result value by the second calculation unit comprises: the second calculation unit generating the comparison result value according to the equation (1), Where U is a vector including the coordinates of the nodes of the user, and V is a vector including the coordinates of the nodes of the virtual coach, and W _mtn is corresponding to the body parts. n body parts, the mth motion category of the motion categories, and the weight of the tth second of the time information, H _n () is a function corresponding to the stream node comparison algorithm and is used The relationship value corresponding to the nth body part is calculated, n and m are positive integers, t is a real number, and Node is a set formed by the body parts. The action comparison system according to claim 1, wherein the stream node comparison algorithm is a dynamic time warping algorithm, an Euclidean distance algorithm or a correlation coefficient algorithm. The action comparison system of claim 1, wherein the second calculation unit is further configured to generate a plurality of second comparison result values, and the mapping unit obtains a minimum of the second comparison result values. The comparison result value is compared with a maximum alignment result value, and the comparison result value is mapped to the similarity in a similarity value field according to the minimum alignment result value and the maximum alignment result value. The action comparison system as described in claim 6 of the patent application, wherein the The similarity value range is from 0 to 100. An action comparison method is applicable to an electronic device, comprising: acquiring a first stream action of the user and a plurality of nodes of the first stream action according to a plurality of images of a user, wherein the first Each of the nodes of the streaming action includes a plurality of coordinates, and each of the nodes of the first streaming action belongs to a body part; providing a second stream action of the virtual trainer and the second stream action a plurality of nodes, wherein each of the nodes of the second stream action includes a plurality of coordinates, and each of the nodes of the second stream action belongs to one of the body parts; The node comparison algorithm calculates a plurality of relationship values between the coordinates of the nodes of the first streaming action and the coordinates of the nodes of the second streaming action; according to the body parts, the a type of motion preset by the user and a plurality of time information of the images, wherein a plurality of weights are obtained by a weight vector; and a result obtained by multiplying the relationship values by the weights respectively produces a comparison result value; And the comparison result value is mapped to a similarity. The action comparison method of claim 8, wherein the number of the nodes of the user is 15, and the number of the nodes of the virtual coach is 15. The action comparison method as described in claim 8, wherein the weights are real numbers, and the weights have a value greater than or equal to 0.1 and less than or equal to 2. The action comparison method of claim 9, wherein the step of generating the comparison result value comprises: generating the comparison result value according to equation (1), Where U is a vector including the coordinates of the nodes of the user, and V is a vector including the coordinates of the nodes of the virtual coach, and W _mtn is corresponding to the body parts. n body parts, the mth motion category of the motion categories, and the weight of the tth second of the time information, H _n () is a function corresponding to the stream node comparison algorithm and is used The relationship value corresponding to the nth body part is calculated, n and m are positive integers, t is a real number, and Node is a set formed by the body parts. The action comparison method as described in claim 8, wherein the stream node comparison algorithm is a dynamic time warping algorithm, an Euclidean distance algorithm or a correlation coefficient algorithm. The action comparison method of claim 8, wherein the step of mapping the comparison result value to the similarity comprises: generating a plurality of second comparison result values; the mapping unit obtaining the second ratios And comparing a minimum alignment result value and a maximum alignment result value to the result value, and mapping the comparison result value to the similarity value field according to the minimum alignment result value and the maximum alignment result value degree. The action comparison method according to claim 13, wherein the similarity value range is from 0 to 100.