WO2023163104A1 - Joint angle learning estimation system, joint angle learning system, joint angle estimation device, joint angle learning method, and computer program - Google Patents

Abstract

[Problem] To identify the angle of a joint in a subject while more reducing a load on the subject than ever before. [Solution] This system is provided with: an inertial sensor 31 that is attached to a predetermined part in a subject 1B who is an object person of a learning for the purpose of determining an acceleration rate and an angular velocity of the predetermined part; inertial sensors 32, 33 that are respectively attached to the right and left thighs of the subject 1B for the purpose of determining acceleration rates and angular velocities respectively of the right and left thighs of the subject 1B; and a learning estimation device 2. The learning estimation device 2 calculates a correct angle formed between the predetermined part in the subject 1B and each of the right and left thighs of the subject 1B on the basis of the above-determined acceleration rates and angular velocities, and performs a mechanical learning using the acceleration rate and the angular velocity of the predetermined part in the subject 1B as input data and also using the correct angle as correct data, thereby generating a learned model. Subsequently, an acceleration rate and an angular velocity of a predetermined part in an estimation object person 1A are input into the learned model to estimate an object angle formed between the predetermined part in the estimation object person 1A and each of the right and left thighs of the estimation object person 1A.

Description

Joint angle learning and estimating system, joint angle learning system, joint angle estimating device, joint angle learning method, and computer program

The present invention relates to technology for estimating joint angles of individuals such as humans.

Technology that identifies human movements while walking is being used in various fields such as healthcare, sports, virtual reality, and animation. The identified movements are used, for example, as follows. It is used for patient diagnosis in the field of healthcare. Alternatively, in the field of sports, it is used for checking the form of athletes. Or, in the field of virtual reality, it is used to reproduce a user's movements with an avatar. Alternatively, in the field of animation, it is used to reproduce user movements with characters.

The following methods are conventionally known as methods for identifying movements during human walking. A marker is attached to a plurality of parts of the human body one by one, and movement is specified by photographing the trajectory of each marker during walking with a video camera. Alternatively, a sensor such as an IMU (Inertial Measurement Unit) is attached instead of the marker, and movement is specified based on the acceleration and angular velocity measured by each sensor. Once the positions and poses of segments such as hips, thighs, and shins are determined by these methods, the angles of the joints can be determined. For example, according to the method described in Patent Document 1, one inertial sensor is attached to each of the forearm and the upper arm, and the angle of the elbow joint is calculated based on the angular velocity and acceleration measured by each of these inertial sensors.

In addition, the following method has been proposed as a method for acquiring the angles of joints during human walking. According to the method described in Patent Literature 2, a wearing device having relatively rotatable arm portions, a waist attachment portion, and an angle sensor for detecting the relative rotation angle between the two is attached to the human body, and the angles of the joints of the human body are measured. identify.

Japanese Patent Publication No. 2021-503340 JP 2018-134724 A

As mentioned above, there are several methods for identifying the joint angles during human walking. However, in any method, it is necessary to attach a large number of sensors to the human body or wear a large-scale attachment. Alternatively, it is necessary to attach a large number of markers and shoot with a video camera. Thus, the conventional method imposes a burden on the subject to identify the angles of the joints.

In view of such problems, it is an object of the present invention to specify the joint angle while reducing the burden on the subject more than before.

An individual angle learning and estimating system according to one aspect of the present invention includes: a first inertial sensor attached to a predetermined portion of a learning target individual to obtain acceleration and angular velocity of the predetermined portion; Learning by performing machine learning using acquisition means for acquiring the angles of the joints of the target individual as correct angles, and using the acceleration and angular velocity of a predetermined part of the learning target individual as input data and the correct angles as correct data. a second inertial sensor attached to a predetermined part of an inference target individual to determine the acceleration and angular velocity of the predetermined part of the inference target individual; estimating means for estimating joint angles of the inference target individual by inputting accelerations and angular velocities of parts into the learned model.

The predetermined part is, for example, one of the waist, left thigh, right thigh, left shin, right shin, left leg, and right leg. A plurality of predetermined parts may be provided. For example, accelerations and angular velocities at three points, the waist, left leg, and right leg, may be obtained.

According to the present invention, the joint angle can be specified with less burden on the subject than before.

Attaching a second inertial sensor to one of the waist, left thigh, right thigh, left shin, right shin, left leg, and right leg to calculate the acceleration and angular velocity to estimate the angle of the joint of the target individual is more accurate. The angle of the joint can be specified well. A plurality of second inertial sensors may be attached. In particular, attaching a second inertial sensor to the hip, left leg, and right leg can provide more accurate results.

1 is a diagram showing an example of the overall configuration of a joint angle estimation system; FIG. It is a figure which shows the example of the hardware constitutions of a learning inference apparatus. It is a figure which shows the example of a functional structure of a learning inference apparatus. FIG. 11 is a flowchart for explaining an example of the flow of data set preparation processing; FIG. FIG. 4 is a diagram showing examples of mounting positions of a plurality of inertial sensors; FIG. 4 is a diagram showing an example of changes in acceleration and angular velocity in the moving coordinate system of the inertial sensor on the waist and an example of changes in angles of four joints of the subject. FIG. 4 is a diagram illustrating an example of the flow of machine learning processing; It is a figure which shows the example of a neural network. FIG. 10 is a diagram showing an example of a trained neural network; 4 is a flowchart for explaining an example of the overall processing flow of a joint angle estimation program; FIG. 10 is a diagram showing an example of MAE in each of the first to third patterns; FIG. 10 is a diagram showing an example of MAE when inertial sensors are fixed to respective parts; FIG. 10 illustrates an example of a method of controlling a prosthesis;

[1. Overall system configuration]
FIG. 1 is a diagram showing an example of the overall configuration of a joint angle estimation system 1. As shown in FIG. FIG. 2 is a diagram showing an example of the hardware configuration of the learning inference device 2. As shown in FIG. FIG. 3 is a diagram showing an example of the functional configuration of the learning inference device 2. As shown in FIG.

The joint angle estimating system 1 shown in FIG. 1 is a system for estimating joint angles when a person 1A to be inferred is walking by AI (Artificial Intelligence) technology. 3 and the like. The learning inference device 2 and each inertial sensor 3 are connected wirelessly.

The learning inference device 2 generates a learned model by machine learning, and estimates the angles of the joints of the inference target person 1A when the inference target person 1A is walking based on the learned model. A case where a personal computer is used as the learning inference device 2 will be described below as an example.

As shown in FIG. 2, the learning inference device 2 includes a main processor 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a network interface 24, a serial interface 25, and a wireless communication device 26. , a display 27, a keyboard 28, a pointing device 29, and the like.

In the ROM 22 or the auxiliary storage device 23, an operating system as well as computer programs such as a joint angle estimation program 40 and a sensor application 41 are installed.

According to the joint angle estimation program 40, the shift window value extraction unit 401, the joint angle calculation unit 402, the data set registration unit 403, the data set storage unit 404, the machine learning unit 405, the learned model storage unit 406, the joint Functions such as an angle inference unit 407 and an inference result output unit 408 are realized.

The RAM 21 is the main memory of the learning inference device 2. The RAM 21 is appropriately loaded with computer programs such as a joint angle estimation program 40 and a sensor application 41 .

The main processor 20 executes computer programs loaded into the RAM 21 . A GPU (Graphics Processing Unit), a CPU (Central Processing Unit), or the like is used as the main processor 20 .

The network interface 24 communicates with other devices by protocols such as TCP/IP (Transmission Control Protocol/Internet Protocol). As the network interface 24, a NIC (Network Interface Card) or a communication device for Wi-Fi is used.

The serial interface 25 communicates with peripheral devices by a serial communication method. A board conforming to standards such as USB (Universal Serial Bus) is used as the serial interface 25 .

The wireless communication device 26 communicates with peripheral devices by short-range wireless. A board conforming to standards such as Bluetooth is used as the wireless communication device 26 .

The serial interface 25 and wireless communication device 26 are used in particular to communicate with the inertial sensor 3 . The serial interface 25 and wireless communication device 26 are selectively used according to the standard of the inertial sensor 3 .

The display 27 displays a screen for inputting commands or data, a screen showing results of operations by the main processor 20, or the like.

The keyboard 28 and pointing device 29 are input devices for the operator to enter commands or data.

The inertial sensor 3 is a device that measures the acceleration and angular velocity of each of the three axes (X-axis, Y-axis, Z-axis) and is generally called an "IMU (Inertial Measurement Unit)" or an "inertial measurement unit." be. A commercially available IMU is used as the inertial sensor 3 . A case where an IMU provided by the MTw Awinda system of Xsens in the Netherlands is used will be described below as an example.

MTw Awinda has the following functions. Multiple (up to 30) IMUs can measure acceleration and angular velocity while maintaining synchronization accuracy of 1/100,000 second using a proprietary protocol. As will be described later, in this embodiment, the angle of the joint is calculated or estimated every 1/100th of a second, so these IMU measurement results can be used in almost accurate synchronization. Then, each IMU can transmit the measurement result to the personal computer by ZigBee.

In addition, MTw Awinda provides receiver and manager software for personal computers. The receiver is a ZigBee standard USB device for receiving data transmitted from the IMU. Manager software is a computer program for changing IMU settings and properties. A receiver is connected to the serial interface 25 , and manager software is installed as a sensor application 41 in the auxiliary storage device 23 .

The processing of each part of the learning inference device 2 shown in FIG. 3 and the processing of the inertial sensor 3 shown in FIG.

[2. Data set preparation]
FIG. 4 is a flowchart illustrating an example of the flow of data set preparation processing. FIG. 5 is a diagram showing examples of mounting positions of the plurality of inertial sensors 31-37. FIG. 6 is a diagram showing an example of changes in acceleration and angular velocity in the movement coordinate system of inertial sensor 31 on the waist and an example of changes in angles of four joints of subject 1B.

The shift window value extraction unit 401, the joint angle calculation unit 402, and the data set registration unit 403 (see FIG. 3) of the learning and inference device 2 extract sex, age, weight, A process of collecting information from a plurality of subjects 1B (1B1, 1B2, . . . ) of various heights and generating a data set for machine learning is executed according to the procedure shown in FIG. A case where information is collected from a subject 1B1 will be described below as an example.

The operator inputs age, sex, weight, and height to the learning inference device 2 via the keyboard 28 or pointing device 29 as attributes of the subject 1B1. The dataset registration unit 403 acquires an attribute vector C indicating these attributes (#101 in FIG. 4). Hereinafter, the attribute vector C of the v-th subject 1B will be referred to as "attribute vector Cv".

Furthermore, the operator collects data on the state of each of the waist, left and right thighs, left and right shins, and left and right legs while the subject 1B1 is walking as follows.

The operator prepares seven inertial sensors 3, one waist belt, two thigh belts, two shin belts, and two leg belts. These belts are exclusive to MTw Awinda, and after initializing with all seven inertial sensors 3 aligned in the same posture, the inertial sensors 3 are adjusted one by one to a predetermined posture. can be installed with At the time of this initialization, the attitude angles of all the inertial sensors come to have a certain attitude angle on the space (fixed) coordinate system, and generally the attitude is consistent with the space coordinate system.

After attaching one inertial sensor 3 to each of these belts, subject 1B1 attached these belts to his waist, left and right thighs, left and right shins, and left and right leg segments ( part). Thereby, as shown in FIG. 5, the inertial sensor 3 is fixed at each specific position of each segment.

Hereinafter, the inertial sensors 3 fixed to the segments of the waist, left thigh, right thigh, left shin, right shin, left leg, and right leg are referred to as “inertial sensor 31,” “inertial sensor 32,” . 37”.

The inertial sensors 31, 32, . . . , 37 are given identification numbers in advance. In this embodiment, it is assumed that the inertial sensors 31, 32, . It can be said that the identification numbers "1", "2", ..., "7" respectively identify the waist, left thigh, right thigh, left shin, right shin, left leg, and right leg.

When these inertial sensors 3 are fixed at specific positions of each segment of the subject 1B1, the subject 1B1 straightens the left hip joint, the right hip joint, the left knee joint, and the right knee joint so that the respective angles are zero degrees. stand. It is preferable to stand with the back of the head, back, buttocks and calves against a vertical wall. The state of standing straight like this is the standard state.

Then, the operator uses the sensor application 41 to calibrate the inertial sensors 31 to 37 when the subject 1B1 is in the reference state. This defines the position and posture of each of the seven segments of the hip, left thigh, right thigh, left shin, right shin, left foot, and right foot in the reference state, and the three-axis acceleration of each of these seven segments. and angular velocity can be obtained based on the measurement results of the inertial sensors 31-37. Attitude is represented by quaternions, but may also be represented by Euler angles. In the following, an example in which a posture is represented by a quaternion will be described. The traveling direction (frontal direction) of the subject 1B1 can be identified by the positions and postures of these seven segments, but it may also be identified by making the subject 1B1 walk a few steps.

After the calibration is completed, the position and posture in the reference state of each of the seven segments such as the waist are set in the learning inference device 2. Subject 1B1 then continues walking for about 10 minutes.

The inertial sensors 31 to 37 measure the acceleration and angular velocity of each of the three axes at a sampling rate of 100 Hz while walking, and calculate the quaternion of the posture of the inertial sensor itself. Then, by the inertial sensors 31 to 37 and the sensor application 41, the acceleration of the waist, left thigh, right thigh, left shin, right shin, left leg, and right leg at each time (every 1/100th of a second), Angular velocity and quaternion are obtained. These accelerations and angular velocities are input to the shift window value extraction unit 401 (see FIG. 3) as values in the moving coordinate system (local moving coordinate system coordinate system) for each inertial sensor 3, and the quaternions are used to calculate joint angles. It is input to section 402 .

Hereinafter, the acceleration at time t of each of the three axes of the moving coordinate system of the segment whose identification number is "s" will be described as "acceleration ax_s_t", "acceleration ay_s_t", and "acceleration az_s_t", and the angular velocity at time t will be "Angular velocity ωx_s_t", "Angular velocity ωy_s_t", and "Angular velocity ωz_s_t" are described, and a quaternion is described as "Quaternion Qs_t". Also, the time of the m-th measurement is represented as "m-1". Therefore, the times of the 1st, 2nd, 3rd, . . . measurements are "0", "1", "2", .

　While subject 1B1 was walking, the acceleration and angular velocity of each of the seven segments, such as the waist, changed. Furthermore, the angles of the joints of subject 1B1 also change. For example, the hip acceleration and angular velocity change as shown in FIG. 6(A). Also, the angles of the left hip joint, right hip joint, left knee joint, and right knee joint change as shown in FIG. 6(B). Note that these angles are calculated by the joint angle calculator 402 as described later.

In the learning inference device 2, when the acceleration and angular velocity of each segment are input to the shift window value extraction unit 401 and the quaternion of each segment is input to the joint angle calculation unit 402 (#102 in FIG. 4), the following processing is performed. done.

The shift window value extraction unit 401 extracts the values belonging to the shift windows 4A (4A1, 4A2, . ~#106).

The u-th shift window 4A is a window whose range is from time "20(u-1)" to time "99+20(u-1)". Therefore, the first shift window 4A1 is a window covering time 0 to 99 as shown in FIG. 6(A). The second shift window 4A2 is a window that covers a range shifted 0.2 seconds to the right from the shift window 4A1, that is, the range from time 20 to 119. Similarly, the third and subsequent shift windows 4A are windows in a range shifted rightward from the previous shift window 4A by 0.2 seconds.

The shift window value extraction unit 401 sets the first shift window 4A as the current window (#103, #104). Accelerations and angular velocities that fall within the current window are extracted from the accelerations and angular velocities received from the inertial sensor 31 (#105). That is, first, the acceleration and angular velocity that fall within the shift window 4A1 are extracted.

The extracted acceleration and angular velocity are represented by a 100x6 matrix. For example, the acceleration and angular velocity extracted from shift window 4A1 are

is represented by the matrix The extracted matrix is hereinafter referred to as "motion matrix B". In particular, the motion matrix B for the u-th time period (shift window 4A) is described as "motion matrix Bu", and the motion matrix B for the u-th time period for the v-th subject 1B is described as "motion matrix Bv_u".

The joint angle calculator 402 calculates the angle at the last time of the current window, that is, the angle at the right end of each of the left hip joint, right hip joint, left knee joint, and right knee joint (#106).

It can be said that the angle of the left hip joint is the angle between the waist and the left thigh, and the angle of the right hip joint is the angle between the waist and the right thigh. Therefore, the joint angle calculator 402 calculates the angle formed by the segments of the hip and the left thigh based on the quaternion at the last time of the current window, as the angle of the left hip joint. For example, the angle of the left hip joint at the last time in the first time period is calculated based on quaternions Q1_99 and Q2_99. Similarly, as the angle of the right hip joint, the angle formed by both segments is calculated based on the quaternions of the waist and the right thigh at the last time of the current window.

Also, it can be said that the angle of the left knee joint is the angle between the left thigh and the left shin, and the angle of the right knee joint is the angle between the right thigh and the right shin. Therefore, the joint angle calculator 402 calculates the angle formed by the segments of the left thigh and the left shin based on the quaternion at the last time of the current window, as the angle of the left knee joint. As the angle of the right knee joint, the angle between the segments of the right thigh and the right shin is calculated based on the quaternion at the last time of the current window.

The calculated angles of the left hip joint, right hip joint, left knee joint, and right knee joint are hereinafter referred to as "angle θHL", "angle θHR", "angle θKL", and "angle θKR". In particular, the angle of the u-th time period is described as "angle θHL_u", "angle θHR_u", "angle θKL_u", and "angle θKR_u", and each angle of the u-th time period of v-th subject 1B is described as "angle θHL_v_u”, “angle θHR_v_u”, “angle θKL_v_u”, and “angle θKR_v_u”.

The data set registration unit 403 stores the attribute vector C acquired in step #101, the motion matrix Bu extracted by the shift window value extraction unit 401 in step #105, the angle θHL_u calculated by the joint angle calculation unit 402 in step #106, A data set consisting of the angle θHR_u, the angle θKL_u, and the angle θKR_u is stored as the data set 50 in the data set storage unit 404 (#107).

Then, shift window value extraction unit 401, joint angle calculation unit 402, and data set registration unit 403 shift the current window to the right by 0.2 seconds (Yes in #108, #109), and step #104 to step #104. The process of #107 is performed. However, if 100×6 accelerations and angular velocities are not aligned even after the shift, that is, if there is no continuation (No in #108), the collection of data sets from subject 1B1 ends.

Furthermore, the shift window value extraction unit 401, the joint angle calculation unit 402, and the data set registration unit 403 collect the data sets 50 from each subject 1B other than the subject 1B1 by the same method, and store them in the data set storage unit 404. (Yes in #110, #101 to #109).

Since the shift window 4A shifts by 0.2 seconds, about 3000 shift windows 4A appear during the walking time, ie, 10 minutes, for one subject 1B. Therefore, about 3000 data sets 50 are obtained from one subject 1B. Therefore, if there are 200 subjects 1B, for example, approximately 600,000 data sets 50 are obtained.

[3. machine learning]
FIG. 7 is a diagram illustrating an example of the flow of machine learning processing. FIG. 8 is a diagram showing an example of the neural network 60. As shown in FIG.

The machine learning unit 405 (see FIG. 3) performs machine learning based on the data set 50 stored in the data set storage unit 404 to obtain the angles of the left hip joint, the right hip joint, the left knee joint, and the right knee joint. Generate a trained model to infer The machine learning method will be described below with reference to FIGS. 7 and 8. FIG.

　The machine learning phase mainly includes a training step, a verification step, and a test step, as shown in FIG.

A neural network 60 as shown in FIG. 8 is prepared in advance in the machine learning unit 405 . Neural network 60 is composed of first network 61 , second network 62 and third network 63 .

The first network 61 is an LSTM (Long Short-term Memory) neural network that calculates and outputs a feature matrix that indicates the features of the motion matrix B extracted by the shift window 4A.

The third network 63 is a network consisting of one or more connected layers and outputs a row vector indicating the characteristics of the attribute vector C. In this embodiment, it is composed of the bonding layer 63A. The coupling layer 63A has 32×4 weighting factors and 32 biasing factors, and when an attribute vector C is input, it outputs a 32×1 row vector indicating the characteristics of the attribute vector C.

The second network 62 is a network consisting of a plurality of fully-connected layers, and in this embodiment is composed of four fully-connected layers 62A-62D. The feature matrix output from the first network 61 is expanded one-dimensionally (256×1) and input to the fully connected layer 62A. Output values of all units of the fully connected layer 62A are input to each unit (node) of the fully connected layer 62B. The output values (a 128×1 row vector) from fully connected layer 62B are merged with the output values from connected layer 63A into a 160×1 row vector. All values of the 160×1 row vector are then input to each unit of the fully connected layer 62C. That is, the output values of all units of the joint layer 63A and the output values of all units of the joint layer 62B are input to each unit of the joint layer 62C. Further, the output values of all units of the fully connected layer 62C are input to each unit of the fully connected layer 62D. The angles ψHL, ψHR, ψKL, and ψKR of the left hip joint, right hip joint, left knee joint, and right knee joint are output from each of the four units of the fully connected layer 62D.

A part of the dataset 50 stored in the dataset storage unit 404 is selected as training data and used for learning of the neural network 60. A portion of the remaining data set 50 is selected as validation data and used for tuning the neural network 60 . The rest are then selected as test data and used for the final evaluation of neural network 60 . For example, 87% of the dataset 50 is selected as training data, 12% as validation data, and 1% as test data.

As shown in FIG. 7, the machine learning unit 405 repeatedly executes the training step and the verification step alternately multiple times. In the training step, the neural network 60 is trained based on the data set 50 selected as training data. The verification may be performed by the holdout method, the cross-validation method, or any other known method.

The machine learning unit 405 trains the neural network 60 as follows, for example, using a certain data set 50a. The motion matrix B included in the data set 50a is input to the first network 61 as input data (explanatory variable data), the attribute vector C is input to the third network 63 as input data, the first network 61, the Arithmetic processing of each layer of the second network 62 and the third network 63 is performed. As a result, angles ψHL, ψHR, ψKL, and ψKR of the left hip joint, right hip joint, left knee joint, and right knee joint are calculated and output from the fully connected layer 62D.

Then, the machine learning unit 405 uses the data of the angles θHL, θHR, θKL, and the angle θKR included in the data set 50a as teacher data (correct data), the first network 61, the second network 62, and Adjust each parameter (eg weighting factor or bias value) of the third network 63 . That is, each parameter is adjusted so that the difference between the angle ψHL and the angle θHL, the difference between the angle ψHR and the angle θHR, the difference between the angle ψKL and the angle θKL, and the difference between the angle ψKR and the angle θKR are reduced.

The machine learning unit 405 trains the neural network 60 using the other data set 50 selected as training data in a similar manner.

In the verification step, the machine learning unit 405 verifies the output values (estimated values) of the trained neural network 60 using the data set 50 selected as verification data, and tunes the hyperparameters of this neural network 60.

In the second training step, the machine learning unit 405 trains the neural network 60 tuned in the first verification step by the method described above, and in the second verification step, trains the neural network 60 trained in the second training step. Tune network 60 in the manner described above. From the third time onwards, the processing of the training step and the verification step is performed in the same way.

A neural network 60 trained and tuned by repeating such a set of training steps and verification steps (that is, epochs) is hereinafter referred to as a "learned neural network 67". Then, in the test step, the final evaluation of the trained neural network 67 is performed using the data set 50 selected as test data.

When the trained neural network 67 has a certain accuracy, the machine learning unit 405 stores the trained neural network 67 as a trained model in the trained model storage unit 406. If it does not have a constant accuracy, then each step can be continued with another new data set 50 until a constant accuracy is obtained. Alternatively, the width of the shift window 4A may be changed, the data set 50 may be collected again, and the machine learning may be redone.

[4. inference〕
FIG. 9 is a diagram showing an example of the trained neural network 67. As shown in FIG.

When the machine learning is completed, the angles of the left hip joint, the right hip joint, the left knee joint, and the right knee joint when the inference target person 1A is walking are mainly obtained by the learned neural network 67 and one inertial sensor 3. can guess. Hereinafter, the inertial sensor 3 will be referred to as an "inertial sensor 38" to distinguish it from the inertial sensors 31-37. Also, it is assumed that the inertial sensor 38 is given "8" as an identification number.

Like the inertial sensor 37, the inertial sensor 38 is attached to the waist belt in a predetermined posture. The guess subject 1A wears this belt so that the inertial sensor 38 is arranged at a specific position on the waist of the guess subject 1A. As a result, the inertial sensor 38 is fixed at a specific position on the waist of the inference target person 1A. This specific position is the same as the specific position during the preparation of the dataset.

The inference target person 1A inputs his or her own age, gender, weight, and height attributes to the learning inference device 2. Hereinafter, the vector representing the input attribute is described as "attribute vector C'".

When the inferred subject 1A starts walking, the inertial sensor 38 measures the acceleration and angular velocity of each of the three axes at a sampling rate of 100 Hz. Then, the inertial sensor 38 or the sensor application 41 obtains the hip acceleration and angular velocity at each time (every 1/100th of a second) in the movement coordinate system, and inputs them to the joint angle inferring unit 407 (see FIG. 3). .

Then, the joint angle inference unit 407 performs inference processing. The procedure of this processing will be described below with reference to FIG.

The joint angle inferring unit 407 inputs the motion matrix B′ representing the hip acceleration and angular velocity at the time when the joint angle is to be inferred and the hip acceleration and angular velocity for the most recent second to the first network 61 of the trained neural network 67. At the same time, the attribute vector C' is input to the third network 63. The motion matrix B' is a 100×6 matrix, and the motion matrix B' representing the acceleration and angular velocity at time t and the acceleration and angular velocity for the most recent second is

is.

Then, the joint angle inference unit 407 calculates the angles ψHL_t, ψHR_t, and ψKL_t from the fully connected layer 62D by performing calculations on the layers constituting the first network 61, the second network 62, and the third network 63, respectively. , and ψKR_t. These angles are the angles of the left hip joint, right hip joint, left knee joint, and right knee joint of the guess subject 1A at time t.

The inference result output unit 408 outputs the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t obtained by the joint angle inference unit 407 at each time t. For example, each angle is displayed on the display 27 as a line graph. Alternatively, table data indicating each angle is generated and transmitted to an external device.

Alternatively, the inference result output unit 408 generates an animation of a pedestrian in which the left hip joint, right hip joint, left knee joint, and right knee joint change to angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t, respectively, as follows, It may be displayed on the display 27 .

The inference result output unit 408 prepares a three-dimensional model of the pedestrian's human body. While the pedestrian is walking, the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t are obtained by the inertial sensor 38 or the like at each time t. change the three-dimensional model to Then, an image of the human body is generated by rendering the changed three-dimensional model and displayed on the display 27 . This reproduces the animation.

According to the inertial sensor 38, the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t are obtained at a sampling rate of 100 Hz. may be changed.

In addition, the inference result output unit 408 may calculate the center of gravity of the inference target person 1A at each time t. The center of gravity may be calculated by a known method. For example, since the software of the MTw Awinda system described above has a function of calculating the center of gravity, the software may be used to calculate the center of gravity.

Alternatively, the center of gravity may be calculated using AI technology, for example, as follows. Various walking postures of a human having an average body shape are captured in advance, and the angle and center of gravity of each part (left hip joint, right hip joint, left knee joint, and right knee joint) in each posture are obtained. A learned model for the center of gravity is generated in advance by performing machine learning using the angle of each part as explanatory variable data and the center of gravity as teacher data.

Then, the inference result output unit 408 calculates the center of gravity at each time t by inputting the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t into this learned model while the inference target person 1A is walking.

Alternatively, a human with an average body shape is made to take various postures in advance, the posture is measured by motion capture, and the center of gravity in each posture is obtained. A learned model for the center of gravity is generated in advance by performing machine learning using each posture as explanatory variable data and the center of gravity as teacher data.

Then, the inference result output unit 408 calculates the center of gravity at each time t by inputting the posture of the three-dimensional model for the animation described above into this learned model.

The center of gravity is defined as the value of the position coordinates of the foot on the ground on the reference coordinate system. However, when both feet are on the ground, the foot that landed first should be used as the reference. Also, an animation of a pedestrian may be displayed such that the position of the center of gravity is indicated by a marker such as a point.

The inference result output unit 408 may further detect and output variations based on changes in the center of gravity and posture during walking.

[5. Overall Processing Flow and Effect of the Present Embodiment]
FIG. 10 is a flowchart for explaining an example of the overall processing flow of the joint angle estimation program 40. As shown in FIG.

Next, the overall processing flow of the learning inference device 2 will be described with reference to a flowchart.

The learning inference device 2 executes AI construction processing and joint angle inference processing based on the joint angle estimation program 40 according to the procedure shown in FIG.

The learning inference device 2 generates and stores a dataset 50 in the dataset preparation phase (#11 in FIG. 10). These processing methods are as described with reference to FIG. A trained neural network 67 is generated and stored by performing machine learning based on the data set 50 (#12, #13). These processing methods are as described with reference to FIGS. 7 and 8. FIG.

Then, when the learning inference device 2 acquires the hip acceleration, angular velocity, and attributes of the inference target person 1A (#14, #15), the learning inference device 2, based on these information and the learned neural network 67, determines whether the inference target person 1A is walking. Angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t of the left hip joint, right hip joint, left knee joint, and right knee joint are estimated (#16) and output (#17). These processing methods are as described with reference to FIG. Furthermore, an animation showing the state of the inference target person 1A may be displayed based on the inference result, or the center of gravity may be calculated and the position thereof may be shown along with the animation.

According to this embodiment, the data required to generate a trained model for estimating the four joint angles can be obtained by using the same type of seven inertial sensors 3 as sensors. As a result, it is possible to generate a learned model more efficiently than before, and to identify the angles of the joints of an individual such as a human being while walking. Furthermore, only one inertial sensor 3 may be attached to the inference target person 1A during inference. Therefore, it is possible to specify the joint angle while reducing the burden on the inferred subject 1A more than before.

[6. Modification]
FIG. 11 is a diagram showing examples of MAE in each of the first to third patterns. FIG. 12 is a diagram showing an example of MAE when the inertial sensor 38 is fixed to each part. FIG. 13 is a diagram showing an example of a control method for the robotic prosthesis 71. FIG.

In this embodiment, the learning inference device 2 performs machine learning using only the hip acceleration and angular velocity of the seven segments as input data. It may be used as input data for machine learning. In this case, instead of the motion matrix B shown in FIG. 8, the following 100×18 matrix in which data for the waist, left leg, and right leg are arranged in columns may be used as input data.

In the inference, the inertial sensor 38 is not only attached to a specific position on the waist of the inference target person 1A, but also one inertial sensor 3 is attached to each specific position of the left leg and right leg, and the learning inference device 2 , the acceleration and angular velocity of the waist, left leg, and right leg are determined by these three inertial sensors 3, respectively. By inputting the 100×18 matrix of these accelerations and angular velocities into the first network 61 (see FIG. 9) of the trained neural network 67, the angle of each of the four joints of the subject 1A is estimated.

In this embodiment, the learning inference device 2 uses four attributes of age, gender, weight, and height as input data for machine learning. good. Alternatively, other attributes may be used. Alternatively, machine learning may be performed without using the attribute vector C. In this case, the third network 63 is not provided in the neural network 60 (see FIG. 8). Also, each unit of the fully connected layer 62D of the second network 62 receives only the output value from the fully connected layer 62C.

In this embodiment, an LSTM neural network is used as the first network 61 (see FIGS. 8 and 9), but other forms of neural networks may be used. For example, CNN (Convolutional Neural Network), BLSTM (Bidirectional LSTM), or WNN (Wavelet Neural Network) may be used. When using CNN, the motion matrix B is converted into a grayscale image in machine learning and input to the first network 61 . Similarly, the motion matrix B' is converted to a grayscale image and input to the first network 61 in inference. Instead of the grayscale image, a line graph image such as that shown in the shift window 4A of FIG. 6A may be used.

Here, the accuracy of inference is examined based on the MAE (Mean Absolute Error) of inference for each of the following three patterns (1) to (3).

(1) First pattern In the first pattern, seven inertial sensors 3 (31 to 37) acquire data on the waist, left and right thighs, left and right shins, and left and right legs, and perform machine learning. This is a pattern in which waist data is acquired by the inertial sensor 38 and inferred. Attribute vector C is not used.

When experiments were conducted with the first pattern using WNN, LSTM, BLSTM, and CNN neural networks, the results shown in Fig. 11(A) were obtained. According to this result, although the MAE of each of the four neurals differs depending on the object of inference, the MAE generally falls within the range of 6-9. Overall, however, BLSTM appears to have the highest accuracy.

(2) Second Pattern The second pattern is a modification of the first pattern to use attribute vectors (age, gender, weight, and height) in both machine learning and inference. Experiments with the second pattern using each of the four neural networks yielded the results shown in FIG. 11(B). According to this result, the total value of MAE is lower than in the case of the first pattern, and it can be seen that a certain effect can be obtained by using the attribute vector.

(3) Third Pattern The third pattern is a modification of the first pattern in which inertial sensors 3 on the left and right feet are added to perform inference. When experimenting with the third pattern using each of the four neural networks, the results shown in FIG. 11(C) were obtained. According to this result, it can be seen that all MAEs are lower than in the case of the first pattern.

According to these experiments, it can be seen that the more types of data, the higher the accuracy, but the more types of data, the more time it takes to prepare inference. Therefore, the pattern to be used should be properly used according to the environment or purpose.

In this embodiment, the learning inference device 2 performs machine learning using the hip acceleration and angular velocity as input data. Machine learning may be performed using the acceleration and angular velocity of any one of as input data. For example, when using the acceleration and angular velocity of the left thigh as input data, instead of the motion matrix B shown in FIG. 8, the following 100×6 matrix may be used as input data.

Then, during the inference, the inertial sensor 38 is fixed at a specific position of the left thigh instead of the specific position of the waist of the inferred subject 1A, and the learning inference device 2 uses the inertial sensor 38 to detect the acceleration of the waist and the Find the angular velocity. By inputting the 100×6 matrix of these accelerations and angular velocities into the first network 61 (see FIG. 9) of the learned neural network 67, the angle of each of the four joints of the subject 1A is estimated.

FIG. 12 shows the MAE when machine learning and inference are performed by BLSTM with the inertial sensors 38 fixed to the waist, thighs, shins, and feet, respectively. Among the MAEs shown in FIG. 12, the MAE corresponding to the waist is the same as the MAE corresponding to BLSTM among the MAEs shown in FIG. 11(A).

According to the MAE shown in FIG. 12, it can be seen that fixing the inertial sensor 38 to the thigh, shin, or foot provides better reasoning than fixing it to the waist. Best fixed to the shin. However, since it is easier to fix it to the waist than to fix it to the shin, it is possible to use different fixing destinations depending on the environment or purpose.

By the way, it is sometimes difficult for subject 1B to stand so that the angle of his left hip joint is zero degrees. Therefore, the learning inference device 2 sets the actual angle of the left hip joint at the time of calibration as the reference angle, and sets the difference between the reference angle and the actual angle of the left hip joint during walking as the angle of the left hip joint in the data set 50. may be as shown. The same is true for the right hip joint, left knee joint, and right knee joint.

In this embodiment, the learning inference device 2 generated a versatile trained neural network 67 by performing machine learning using the data set 50 collected from multiple subjects 1B. However, the trained neural network 67 dedicated to this subject 1B may be generated by collecting the data set 50 from one specific subject 1B and performing machine learning. In this case also, there is no need to provide the third network 63 in the neural network 60, and only the output values from the fully connected layer 62C are input to each unit of the fully connected layer 62D of the second network 62.

In the present embodiment, the case of generating a neural network (learned model) for inferring the hip joint angle and knee joint angle has been described as an example. It is also possible to generate a neural network of

For example, a neural network may be generated to infer the angles of the left and right ankle joints (ankle joints). In this case, in the data set preparation phase, the learning inference device 2 acquires the angle of the left ankle joint as correct data based on the quaternions of the left shin and the left leg respectively, machine learning by acquiring the angle of the right ankle joint as correct data.

Alternatively, a neural network may be generated to infer the angle of the open legs in the front-back direction (angle of both legs). In this case, in the data set preparation phase, the learning inference device 2 acquires the forward and backward leg angles based on the quaternions of the left and right thighs as correct data, and performs machine learning. good.

Alternatively, a neural network may be generated to infer the anteroposterior angles of the left and right shoulder joints. In this case, one inertia sensor 3 is fixed to each of the left and right upper arms, the angle of the left shoulder joint is acquired as correct data based on the quaternions of the waist and the left upper arm, respectively, and based on the quaternions of the waist and the right upper arm. Then, the angle of the right shoulder joint is obtained as correct data, and machine learning is performed.

The learning reasoning device 2 can be used in various situations such as medical care, sports, and entertainment. For example, the learning reasoning device 2 can be used for patient rehabilitation as follows.

The staff of the medical institution fixes the inertial sensor 38 to a specific position on the patient's waist, and inputs the patient's age, gender, weight, and height attribute values to the learning inference device 2 . When the patient starts walking, the acceleration and angular velocity of the waist at each time (for example, every 1/100th of a second) in the movement coordinate system are obtained and input to the learning inference device 2 .

Then, the learning inference device 2 estimates the angles HL_t, ψHR_t, ψKL_t, and ψKR_t of the patient's left hip joint, right hip joint, left knee joint, and right knee joint at each time t ( inference). Furthermore, the center of gravity is calculated. Then, an animation is displayed showing how the patient walks. At this time, changes in the position of the center of gravity are also reproduced. Furthermore, the variation in posture and center of gravity during walking is inspected and output.

The staff advises the patient on the preferred way to walk while watching the animation and variations in posture and center of gravity together with the patient. The learning reasoning device 2 may also display an animation representing an ideal gait next to the animation of the patient.

A unique ID is assigned to each patient, and the patient's information (name, address, contact information, face photo, chart, information on artificial limbs or prosthetic limbs, rehabilitation history, signature of consent to disclaimer, or Code information for preventing falsification of such information, etc.) may be associated with the ID and managed in the learning inference device 2 . By referring to such information, the staff can support rehabilitation more efficiently and reliably than before.

The angle inferred by the learning inference device 2 may be used to control the prosthetic leg. Here, as shown in FIG. 13, the control method will be described by taking as an example the case where the guess subject 1A2 uses the robot prosthetic leg 71 for the left leg.

The robot prosthetic leg 71 corresponds to the area below the left knee of a human to the toe of the left foot, and has a joint in a portion corresponding to the ankle. The joint has approximately the same range of motion as the ankle joint of a human left leg, and is extended and bent by a motor and a controller that controls the motor.

The inertial sensor 38 and the robot prosthetic leg 71 are fixed in advance at respective predetermined positions of the person to be guessed 1A2, and the age, sex, weight, and height attributes of the person to be guessed 1A2 are input to the learning and reasoning device 2.

When the inference target person 1A2 starts walking, the acceleration and angular velocity of the waist at each time in the movement coordinate system are obtained and input to the learning inference device 2. The learning inference device 2 estimates the angle of each joint at each time t by the method illustrated in FIG.

Then, the controller of the robot prosthesis 71 controls the motor based on the transmitted angle (command angle). For example, the motor is controlled so that the joint of the prosthetic leg 71 of the robot forms the commanded angle. By driving the joints of the robot prosthesis 71 in this way, natural walking becomes possible.

In the present embodiment, as shown in FIGS. 6A and 6B, the learning inference device 2 uses the most recent data including the time when the joint angle is formed as input data paired with the joint angle (correct data). Machine learning was performed using the acceleration and angular velocity at each time for 1 second, but machine learning may be performed using the acceleration and angular velocity at each time in the immediately following time zone, or both immediately before and after Machine learning may be performed using the acceleration and angular velocity at each time in the time period.

That is, for example, when the time when the joint angle is formed is "T",

matrix may be used as input data. In this case, the acceleration and angular velocity at each time in the immediately following time zone are similarly input to the learned neural network 67 during inference.

or

matrix may be used as input data. Note that α+β=100. In this case, the acceleration and angular velocity at each time in both the immediately preceding and immediately succeeding time zones are similarly input to the learned neural network 67 during inference.

In this embodiment, as shown in FIG. 6, the learning inference device 2 uses a shift window 4A with a time width of 1 second and a shift width of 0.2 seconds to acquire the data set 50. was used, other time widths or shift widths of shift windows may be used. For example, a shift window with a duration of 1.2 seconds and a shift width of 0.5 seconds may be used. Also, in this case, in the inference, a matrix (matrix of acceleration and angular velocity) corresponding to the shift width is used as the motion matrix B'. That is, when the time width of shift window 4A is 1.2 seconds, a matrix of 120×6 is used.

In the present embodiment, the inertial sensors 31 and 38 are attached to a dedicated belt in a predetermined posture, and the belt is worn at a specific position on the waist of the subject 1B or the inferred subject 1A. Inertial sensors 31, 38 are fixed in a specific posture. However, the inertial sensors 31 and 38 may be fixed slightly shifted or tilted from a specific position. Therefore, in each of machine learning and inference, calibration may be performed so that the positions and orientations of the inertial sensors 31 and 38 in the reference state are regarded as the initial positions.

In the present embodiment, the learning inference device 2 acquires the angles of the left hip joint, the right hip joint, the left knee joint, and the right knee joint based on the positions and orientations of the segments obtained by the inertial sensors 31-35. , may be obtained in other ways. For example, a marker may be attached to each segment, the position and orientation of each segment may be specified by photographing the marker with one or more cameras, and each angle may be calculated. That is, it may be calculated by so-called optical motion capture. Alternatively, one or more depth cameras may identify the three-dimensional position of each segment and calculate each angle. Alternatively, each angle may be obtained by a mounting tool as described in Patent Document 2.

In this embodiment, as the learned neural network 67, a neural network for inferring joint angles during walking is generated. Neural networks can also be generated to infer angles during other actions such as walking, skiing, and so on.

In this embodiment, a neural network for inferring the angles of human joints is generated as the trained neural network 67. However, animals having bodies and limbs other than humans, such as dogs, cats, and horses, may also be used. A neural network can also be generated to infer the joint angles of an individual.

In the present embodiment, each function shown in FIG. 3 is provided in the learning inference device 2, but may be distributed and provided in a plurality of devices. For example, the shift window value extraction unit 401, the joint angle calculation unit 402, the data set registration unit 403, and the data set storage unit 404 are provided in the first computer, the machine learning unit 405 is provided in the second computer, and the trained model A storage unit 406, a joint angle inference unit 407, and an inference result output unit 408 are provided in the third computer. A first computer, a second computer, and a third computer are connected by a communication line such as the Internet, a LAN (Local Area Network), or a public line. The first computer, the second computer, and the third computer perform the following processes based on the first computer program, the second computer program, and the third computer program, respectively.

The first computer transmits the data set 50 stored in the data set storage unit 404 to the second computer along with a machine learning command. Then, the second computer generates a trained neural network 67 by performing machine learning using the received data set 50, and transmits it to the third computer. The third computer stores the trained neural network 67 in the trained model storage unit 406 . Then, the learned neural network 67 is used to estimate the four joint angles of the person 1A to be estimated.

In addition, the joint angle estimating system 1 and the learning inference device 2 as a whole or the configuration of each part, the contents of the processing, the order of the processing, the configuration of the data set, the range of the shift window 4A, etc. can be changed as appropriate in line with the spirit of the present invention. be able to.

1 Joint angle estimation system 1B Subject (learning target individual)
2 Learning inference device (individual angle learning system, individual angle estimation device)
31 inertial sensor (first inertial sensor)
37 inertial sensor (second inertial sensor)
401 shift window value extraction unit (first acquisition unit)
402 joint angle calculation unit (acquisition means, second acquisition means)
405 machine learning unit (learning means)
407 joint angle inference unit (inference means)
50 data sets (input data, correct data)
67 Trained Neural Network (Trained Model)

Claims (12)

1. a first inertial sensor attached to a predetermined part of a learning target individual to obtain acceleration and angular velocity of the predetermined part;
   Acquisition means for acquiring the angle of the joint of the learning target individual as a correct angle;
   learning means for generating a trained model by performing machine learning using the acceleration and angular velocity of a predetermined part of the learning target individual as input data and using the correct angle as correct answer data;
   a second inertial sensor attached to a predetermined portion of the inference target individual to determine the acceleration and angular velocity of the predetermined portion;
   estimating means for estimating joint angles of the inference target individual by inputting the acceleration and angular velocity of a predetermined part of the inference target individual into the learned model;
   A joint angle learning and estimating system comprising:
2. a first acquisition means for acquiring acceleration and angular velocity of a predetermined part of a learning target individual that is a learning target;
   a second acquiring means for acquiring the angle of the joint of the learning target individual as a correct angle;
   learning means for generating a learned model by performing machine learning using the acceleration and angular velocity of the predetermined part as input data and using the correct angle as correct data;
   A joint angle learning system characterized by comprising:
3. The learning means performs the machine learning using, as the input data, the acceleration and angular velocity of the predetermined part in a time period of a predetermined length immediately before or after the time when the correct angle is formed. generate a finished model,
   The joint angle learning system according to claim 2.
4. The second obtaining means obtains an angle formed by a predetermined part of the learning target individual and each of a plurality of predetermined segments among segments forming a limb of the learning target individual by calculating acceleration and angular velocity of the predetermined part. and obtaining the correct angle by calculating based on the acceleration and angular velocity of each of the plurality of predetermined segments;
   The joint angle learning system according to claim 2 or 3.
5. the plurality of predetermined segments includes a left thigh and a right thigh;
   The second acquisition means acquires, as the correct angles, a left hip joint angle formed between the predetermined portion and the left thigh and a right hip joint angle formed between the predetermined portion and the right thigh, and
   The learning means generates the learned model by performing the machine learning using the left hip joint angle and the right hip joint angle as the correct data.
   The joint angle learning system according to claim 4.
6. the plurality of predetermined segments further includes a left shin and a right shin;
   The second obtaining means further obtains the left knee joint angle formed by the left thigh and the left shin and the right knee joint angle formed by the right thigh and the right shin, as well as the acceleration and angular velocity of the left thigh. Acquire as the correct angle based on the acceleration and angular velocity of the right thigh,
   The learning means generates the learned model by performing the machine learning using the left knee joint angle and the right knee joint angle as the correct data.
   The joint angle learning system according to claim 5.
7. The learning means generates the learned model by performing the machine learning using the age, sex, weight, or height of the learning target individual as the input data.
   The joint angle learning system according to any one of claims 2 to 6.
8. The predetermined part is any one of the torso, waist, left thigh, right thigh, left shin, right shin, left leg, and right leg.
   The joint angle learning system according to any one of claims 2 to 7.
9. By inputting the acceleration and angular velocity of a predetermined part of the inference target individual to be inferred into the learned model generated by the joint angle learning system according to any one of claims 2 to 7, the inference target individual guessing means for guessing the joint angles of
   A joint angle estimating device comprising:
10. Acquiring the acceleration and angular velocity of a predetermined part of the learning target individual, which is the object of learning, using a first inertial sensor attached to the predetermined part,
    Acquiring the angle of the joint of the learning target individual as a correct angle,
    generating a learned model by causing a computer to perform machine learning using the acceleration and angular velocity of a predetermined part of the learning target individual as input data and using the correct angle as correct data;
    A joint angle learning method characterized by:
11. to the computer,
    executing a first acquisition process for acquiring the acceleration and angular velocity of a predetermined part of the learning target individual that is the target of learning;
    executing a second acquisition process for acquiring the angle of the joint of the learning target individual as a correct angle;
    causing a machine learning system to perform machine learning using the acceleration and angular velocity of the predetermined part as input data and using the correct angle as correct data to execute a learning process for generating a learned model;
    A computer program characterized by:
12. to the computer,
    executing a process of acquiring the acceleration and angular velocity of a predetermined part of the inference target individual that is the inference target;
    By inputting the acquired acceleration and angular velocity of a predetermined part of the inference target individual to the learned model generated by the joint angle learning system according to any one of claims 2 to 7, the joint of the inference target individual to perform the process of guessing the angle of
    A computer program characterized by:

Images