KR102517554B1 - Knee joint angle estimating apparatus and method - Google Patents

Abstract

A knee joint angle estimator and method for estimating the same are disclosed. An apparatus for estimating knee joint angle according to the present invention includes a sensor for detecting at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle during walking of a subject; markers respectively attached to the subject's knee, the upper side of the knee, and the lower side of the knee; and a joint angle estimation unit that connects each marker to measure a knee angle and estimates a knee joint angle corresponding to a shoe based on the measured knee angle and a detection signal detected by a sensor.

Description

Knee joint angle estimation device and its estimation method {KNEE JOINT ANGLE ESTIMATING APPARATUS AND METHOD}

The present invention relates to an apparatus for estimating a knee joint angle and a method for estimating the same, and more particularly, to a convolutional neural network (CNN) image-based method and a foot pressure mapping image for a gait cycle while using various shoes. It relates to a knee joint angle estimating device for estimating a knee joint angle and a method for estimating the same.

The use of footwear changes knee kinematics. The motion of the knee angle is more increased when wearing shoes compared to when walking barefoot. Blanchette demonstrated that knee flexion increased with increasing heel height. Kerrigan also demonstrated that the use of high-heeled shoes contributes to knee osteoarthritis.

In general, knee osteoarthritis is associated with factors such as aging, gender, overweight, knee injury, repetitive use of the joint, bone density, muscle weakness, and joint laxity. Additionally, the type of footwear may be another factor that accelerates the progression of knee osteoarthritis, primarily due to increased heel height.

Knee osteoarthritis usually causes endogenous limitations of the knee along with pain resulting in gait patterns. Monitoring of the knee joint angle can play an important role in biomechanically evaluating the function of the knee, and is helpful in both clinical and research studies of the knee. Therefore, the evaluation of joint angle can be defined as one of the three key variables used to monitor the body's joint function.

Joint angle measurement plays an important role in functional evaluation and rehabilitation of proprioception, which is important for rehabilitation of balance, posture, and motor control. In rehabilitation medicine, evaluation of joint angle is mainly performed through clinical measurement tools to investigate treatment effect monitoring, patient progress rate, and treatment guide intervention. Accurately measuring joint angle brings more advantages to the application of gait evaluation. One of the main approaches in the development of joint angle measurement systems is to obtain reliable joint angles with an uncomplicated system. Such an approach can be used to develop a continuous monitoring system beneficial for real-time gait monitoring. This could provide an inexpensive and convenient way for patients not to have to attend clinics or hospitals. As medical monitoring related to joint angles has increased, various studies have been conducted to evaluate joint angles. Importantly, new techniques of machine or deep learning learning can be integrated with sensor technology and applied to real-time monitoring systems that are beneficial for outdoor monitoring. Therefore, machine or deep learning learning with various source data is useful for evaluating lower limb joint angles, but there are not many attempts to do so.

Publication No. 10-2018-0089126 (published date: 2018.08.08)

The present invention was invented to estimate the angle of the knee joint in accordance with the above-mentioned needs of the times, and the gait cycle while using various shoes using a convolutional neural network (CNN) image-based and foot pressure mapping image. An object of the present invention is to provide a knee joint angle estimating device and method for estimating a knee joint angle for .

An apparatus for estimating knee joint angle according to an aspect of the present invention for achieving the above object includes a sensor for detecting at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle during walking of a subject; markers respectively attached to the subject's knee, an upper side of the knee, and a lower side of the knee; And a joint angle estimator for measuring a knee angle by connecting each of the markers and estimating a knee joint angle corresponding to a shoe based on the measured knee angle and the detection signal detected by the sensor. characterized by

The knee joint angle estimator described above may further include a camera installed at a set distance from the subject's position and capturing a set number of image frames per unit time for analyzing the subject's gait state. In this case, the joint angle estimation unit estimates the knee joint angle by synchronizing the image frame captured by the camera, the detection signal detected by the detection sensor, and the knee angle measured by each of the markers.

Here, the joint angle estimator may extract an image of a set gait cycle from an image frame for a set gait time taken by the camera, and use the extracted image as reference data of a Convolutional Neural Network (CNN) image-based model. .

In addition, the detection sensor is made of a flexible polymer insole with a thickness of 0.15 mm or less, and is composed of more than a set number of cells per unit area, and it is preferable to detect the detection position and average pressure based on each cell. do.

In addition, the joint angle estimator determines the knee joint angle corresponding to each of flat shoes with a heel height of 1.0 cm or less, sneakers with a heel height of 3.0 cm or less, and classic pump heels with a height of 9.0 cm or less. It is desirable to estimate

A method for estimating a knee joint angle according to an aspect of the present invention for achieving the above object is a method for estimating a knee joint angle corresponding to a shoe, which is performed by a knee joint angle estimating device, by using a sensor, detecting at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle during walking of the child; measuring a knee angle by connecting markers respectively attached to the subject's knee, an upper side of the knee, and a lower side of the knee; and estimating a knee joint angle corresponding to the shoe based on the measured knee angle and the detection signal detected by the sensor.

The above-described method for estimating the knee joint angle may include the steps of photographing a set number of image frames per unit time for analyzing the gait state of the subject using a camera installed at a set distance from the position of the subject; and synchronizing the image frame captured by the camera, the detection signal detected by the detection sensor, and the knee angle measured by each of the markers.

Here, in the step of estimating the knee joint angle, an image of a set gait cycle is extracted from an image frame for a set gait time taken by the camera, and the extracted image is used as reference data of a Convolutional Neural Network (CNN) image-based model. can be used as

In addition, the detection sensor is made of a flexible polymer insole with a thickness of 0.15 mm or less, and is composed of a set number or more cells per unit area, and detects a detection position and an average pressure based on each cell.

In addition, the step of estimating the knee joint angle corresponds to each of flat shoes with a heel height of 1.0 cm or less, sneakers with a heel height of 3.0 cm or less, and classic pump heels with a heel height of 9.0 cm or less. Estimate the knee joint angle.

According to the present invention, it is possible to estimate the knee joint angle for the gait cycle while using various shoes using a convolutional neural network (CNN) image based and foot pressure mapping image.

1 is a diagram schematically showing the configuration of a knee joint angle estimating device according to an embodiment of the present invention.
2 is a view showing an example of measuring a knee joint angle.
3 is a view showing examples of three types of shoes.
FIG. 4 is a diagram showing the characteristics of the three types of shoes shown in FIG. 3 .
Figure 5 is a flow diagram of a CNN algorithm with regressing foot pressure mapping images to estimate knee joint angle.
6 is a diagram illustrating an example of a foot pressure mapping image at each gait event of a gait cycle.
7 is a flowchart illustrating the acquisition of 100 foot pressure mapping images from GC images.
8 is a diagram showing an example of GC of input data including 100 images extracted from GC images.
9 is a diagram showing the proposed CNN image-based network architecture, including feature extraction and regression blocks.
10 is a diagram showing details of each element in a CNN image-based regression task under execution of a foot pressure mapping image for estimating a knee joint angle.
11 is a diagram showing 10 gait cycles of knee joint angles compared with CNN model predictions for three types of shoes and reference data.
Figure 12 shows the accuracy of prediction for the three test shoe types.
13 is a flowchart illustrating a method for estimating a knee joint angle according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described through exemplary drawings. In describing the reference numerals for the components of each drawing, the same numerals indicate the same components as much as possible, even if they are displayed on different drawings. In addition, in describing an embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function hinders understanding of the embodiment of the present invention, the detailed description will be omitted.

Also, terms such as first, second, A, B, (a), and (b) may be used to describe components of an embodiment of the present invention. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. When an element is described as being “connected,” “coupled to,” or “connected” to another element, the element may be directly connected, coupled, or connected to the other element, but not between the element and the other element. It should be understood that another component may be “connected”, “coupled” or “connected” between elements.

Long- and Short-Term Memory Neural Network and Time-Advanced Feature, which rely on surface electromyographic (EMG) data, are very accurate in predicting knee joint angle. However, EMG data is obtained by directly attaching a sensor to the skin, which has a problem in that it may cause discomfort to the subject.

When using an inertial measurement unit (IMU) with machine learning to estimate the ankle joint angle, orientation algorithms are needed. IMU technology requires determining the optimal position to obtain accurate data for the model. Instead of directly attaching the sensor to the skin or body, using input data from foot movements and a deep learning approach can play an important role in creating a model for estimating lower extremity joint angles. Therefore, many researchers have applied plantar pressure input data to a deep learning approach to estimate lower extremity joint angles.

As one application of plantar pressure input data, there is a study that collects ground reaction force input data from a sensor placed on an insole for an artificial neural network (ANN) model for estimating an ankle joint angle. They demonstrated that the ANN model can evaluate gait kinematics using ground reaction force data. To achieve higher accuracy for the use of deep learning algorithms to predict lower extremity joint angles, one of the other studies incorporated the Golden Section (GS) algorithm into a general regression neural network that relies on multi-source signals. They obtained better correlations compared to backpropagation neural networks using multi-source signals including surface electromyography (sEMG), hip joint angle, plantar pressure data, and GS as optimizers to minimize errors. efficiency and short computational time. Using foot pressure data and deep-learning root method, it is possible to build a model that can evaluate joint angles. The advantage of using a pressure system that collects foot pressure data is that it is a convenient system as all you have to do is place the sensor on the insole. That is, such a system is convenient to use as a pressure system because of less complex system settings, and data obtained therefrom can be configured in various forms and applied to various applications.

In general, foot pressure analysis can be applied to shoe design, sports performance analysis and injury prevention, balance control improvement, and disease diagnosis. In addition, the pressure information can identify and manage damage associated with various musculoskeletal, cortical and neurological diseases. Another study asserted that information on the pressure system could prove the relationship between foot pressure and lower extremity posture. The pressure system can generate a variety of data formats that can be applied to develop other deep learning approaches based on the data format. For example, a pressure system can consist of three types: a pressure distribution platform, an imaging technology with sophisticated image processing software, and an in-shoe system. In particular, the pressure system of Inshu System is flexible and can be installed in various shoes. Data obtained from the Inshu system includes numbers and images. Another study found that the coefficient of friction used increased when the shear stress increased during walking in high-heeled shoes. Therefore, the use of various shoes, especially high-heeled shoes, induces two types of stress: normal stress and shear stress. Commercial pressure systems can only measure normal stress. Accordingly, numerical foot pressure data generated by the pressure system may provide machine or deep learning results with low accuracy when estimating lower extremity joint angles during use of a high-heeled shoe.

Since the foot pressure system can store numerical and image data, the foot pressure mapping image can contain shear and normal stress information, which means that more information can improve the quality of training. Machine learning can be applied to important classification and regression tasks in the fields of gait analysis, medical and bioinformatics. The advantage of using machine learning is that it can be applied to small data sets, but it requires feature selection and extraction, which can increase computational time and complexity. To overcome this problem, deep learning can be suitable because it has the ability to learn properly with direct data input rather than a feature extraction procedure. Importantly, deep learning of convolutional neural networks (CNNs) has not yet been introduced to evaluate knee joint angles through image data. Therefore, the present invention estimates the knee joint angle for the gait cycle while using various shoes using a convolutional neural network (CNN) image based and foot pressure mapping image.

1 is a diagram schematically showing the configuration of a knee joint angle estimating device according to an embodiment of the present invention.

Referring to FIG. 1 , a knee joint angle estimation device 100 according to an embodiment of the present invention may include a sensor 102, a marker 104, a joint angle estimation unit 106, and a camera 108. .

The detection sensor 102 detects at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle during the subject's walking. At this time, the detection sensor 102 may be implemented as an F-Scan sensor mounted on the insole of the shoe.

The F-Scan sensor is a commercial product and can be defined as a clinical tool capable of detecting pressure information on the sole of the foot. In particular, the F-Scan sensor is accurate, reliable and cost effective. Here, the F-Scan sensor is made of an insole made of a flexible polymer material and can be implemented as a matrix system including an ink-based force sensing resistor having a thickness of 0.15 mm or less. That is, the F-Scan sensor has 960 individual pressure sensing positions and is composed of 4 cells per unit area (cm ² ). At this time, each cell measures the force, and the average pressure for each cell can be calculated using system software. In this case, sensing elements are arranged in rows and columns of the F-Scan sensor, and data can be displayed in two dimensions on a computer screen through system software. In addition, vertical ground stress, pressure, peak pressure, contact area, and gait cycle time data can be obtained through the system software corresponding to the F-Scan sensor, and such data can be saved as numerical values and images for further analysis. there is. By applying the F-Scan sensor to various types of shoes, it is possible to collect foot pressure data according to the type of shoes.

The marker 104 is attached to the subject's knee, the upper side of the knee, and the lower side of the knee, respectively. That is, in order to measure the angle of the knee joint in any posture of the subject, a plurality of markers 104 may be attached to the knee, the hip portion above the knee, and the ankle portion below the knee. At this time, the angle measurement software is used to connect the positions of each marker 104, and as shown in FIG. 2, the change in angle of the knee joint can be measured according to the movement of the subject while walking. .

The joint angle estimator 106 measures the knee angle by connecting each marker 104, and calculates the knee joint angle corresponding to the shoe based on the measured knee angle and the detection signal detected by the sensor 102. guess

The camera 108 is installed at a set distance from the subject's position, and captures a set number of image frames per unit time for analyzing the subject's gait state. At this time, the joint angle estimator 106 synchronizes the image frame captured by the camera 108, the detection signal detected by the detection sensor 102, and the knee angle measured by each marker 104 to synchronize the knee angle. Joint angles can be estimated. At this time, the joint angle estimation unit 106 synchronizes the video captured by the camera 108, the detection signal detected by the sensor 102, and the knee joint angle measured through the marker 104 in time. , It is possible to display the detection signal and the measurement signal of the knee joint angle together with the video signal photographed by the camera 108 in real time.

Meanwhile, the knee joint angle estimation according to the embodiment of the present invention estimated the knee joint angle for 6 healthy female participants. Each participant has experience using high heels and a treadmill, and is 20.7±0.8 years old, weighs 49.8±1.7 kg, and is 158.7±4.0 cm tall. In addition, participants with experience of flat feet, foot pain, leg injuries, and lower extremity surgery were excluded from the estimation of knee joint angle.

In addition, as the experimental equipment for estimating the angle of the knee joint according to the embodiment of the present invention, a treadmill, a smartphone (camera), a tripod, an F-Scan sensor, and three types of shoes were constructed. Here, for foot pressure mapping, the frame rate was set to 100 fps (frame per second) by F-Scan (Tekscan Inc., Boston, MA, USA), and each gait state of the subject was imaged using a smartphone camera. recorded with In this case, the camera recorded data at 30 frames per second at a distance of 2.5 m from the center of the treadmill and 1.0 m from the ground. In addition, markers 104 were placed on the hips, knees, and ankles of each subject, and the position of each marker 104 was automatically tracked using the angle tool of Kinovea software.

As shown in FIG. 3, the experimental protocol included walking on a treadmill with a comfortable gait chosen by the subject using three types of shoes: F1: flat shoes, F2: sneakers, and F3: classic pump heels. At this time, the order of using the shoes may be randomly adjusted. In addition, the main purpose of this study was informed at the beginning of the experiment, and the subjects were guided to walk with a comfortable gait of their own choosing to get used to the experimental environment. Subjects had to walk on a treadmill for 2 minutes in each shoe condition, and the shoe change interval was 2 minutes. 4 shows three types of shoes used in the knee joint angle measurement according to an embodiment of the present invention.

Figure 5 is a flow diagram of a CNN algorithm with regressing foot pressure mapping images to estimate knee joint angle. This can be done when walking with a comfortable gait in a variety of shoes.

1) Input data processing

Foot pressure mapping images collected in each experimental condition were measured using the F-Scan sensor and saved as fsx files through F-scan Research 7.0 (Tekscan Inc., Boston, MA, USA). Here, F-scan Research 7.0 is used to open the fsx file and save the file to an image file.

When force is applied to the F-scan sensor, the foot pressure mapping image appears as an image of the contact area. As shown in FIG. 6 , the gait cycle includes a stance phase (60% of the gait cycle) and a swing phase (40% of the gait cycle). In Eh 6, L is left foot, R is right foot, HS is heel strike, FF is flat foot, MSt is mid-stance, HO is heel off, TO is toe off, IS is initial swing, MSw is mid-swing indicates

The gait cycle (GC) is defined as the heel strike moment (heel contact with the ground) until the next heel strike during walking. In general, the gait cycle begins when the heel first strikes the ground. However, since only a small-sized mapping image appears, it is not easy to extract each gait cycle image. In the embodiment of the present invention, similar to the image size studied by Wafai Research, gait cycle data at the time of heel strike was cut, and 10 GC images were extracted from the mid-minute data, which is a stable analysis area. Each GC image was cropped with Kinovea 0.8.27 (Open and Free Software Foundation, Inc., Bostan, MA, USA). Since the GC image duration is not constant for each subject and experimental condition, 100 images were extracted from each GC image to generate input data of the same length. 7 is a flowchart illustrating the acquisition of 100 foot pressure mapping images from GC images.

First, frames per second (fps) of the GC image were extracted, and the number of frames of the GC image was counted. In addition, the duration of the GC image was calculated by dividing the total frames of the GC image by frames per second. In addition, a new period (Tf: timeframe) for extracting each frame (image) from the GC image was calculated using Equation 1. Each image is assigned a label (1-100) and stored in a folder. And, each folder is named according to subject and experiment condition. The function of generating images from images was performed through Python 3.7.

[Equation 1]

Here, T _f is defined as a new period used to extract each image (frame) from the GC image, and T _GC is defined as the total time of the GC image.

In an embodiment of the present invention, the subject must walk on the treadmill with a comfortable gait, and the treadmill can reduce the stride time. Therefore, the duration of each gait cycle data is short. If 100 images are generated from the gait cycle image, a slight change may appear on the image compared to each standing image in the other images. Foot pressure mapping information appears when the foot contacts the ground. During the swing phase, ground stress does not occur in the foot pressure mapping because the foot is lifted off the ground. 8 shows an example of GC of input data including 100 images extracted from GC images.

2) Reference data processing

The reference knee joint angle can be corrected through Kinovea software in video recording. The 2-minute video was cut to 1 minute by deleting the beginning and end, and the gait segmentation performance was performed through the middle 1 minute of the GC video. In Kinovea software, the angle tool is assigned to track the marker starting from the heel strike and continuing until the next heel strike, at which time the calibration is stopped and the data is saved as an Excel file. The same process may be repeated until 10 gait cycles are extracted from the video of the middle 1 minute. For knee joint angle estimation according to an embodiment of the present invention, 10 gait cycles of knee joint angle data for each subject and experimental condition were extracted and used as reference data for a CNN image-based model. Since the knee joint angles of the GCs were not of the same length, interp1 was applied to the 1D function of MATLAB to obtain 100 data points (DP) per GC.

3) Proposed CNN image-based model

To reduce the heavy matrix of the CNN model, the image data is shaped into 120x120x1 height, width, and gray scale, where the channel dimensions are each 1. Each pixel of the image data includes a value range from 0 to 255, but each pixel of the image data may be normalized to a small range of 0-1 to facilitate CNN training. Figure 9 shows the proposed CNN image-based network architecture, including feature extraction and regression blocks.

The performance of deep learning algorithms depends on two factors including architecture parameters and training parameters. Figure 10 shows the details of each element in the CNN image-based regression task under the execution of the foot pressure mapping image for estimating the knee joint angle.

In general, the performance of deep learning algorithms is influenced by selecting optimally tuned hyperparameters, including architectural parameters and training parameters used in the feature extraction and regression stages. For the feature extraction process, hyperparameters play an important role in optimizing the accuracy of the deep learning model, and the deep learning model should be selected before training the model. When choosing architecture parameters, it is necessary to consider the number of layers, number of filters, size of each filter and pooling size. In the embodiment of the present invention, the CNN image-based model uses 32 filters, 64 filters, and 64 filters for the Conv2D_1 layer, the Conv2D_2 layer, and the Conv2D_3 layer, respectively. Also, the size of each filter consists of 3x3 and 2x2 max pooling in all convolutions. At this time, dropouts with probability levels of 10%, 10%, and 25% were selected after maximum pooling in the Conv2D_1 layer, the Conv2D_2 layer, and the Conv2D_3 layer, respectively. The activation function is a non-linear transformation that plays an essential role in defining how the input weight is transformed, which can be used in the next layer of neurons. In addition, the outputs of Conv2D_1, Conv2D_2, Conv2D_3, and density were converted through activation functions of Sigmoid, Relu, Relu, and Elu. Linear activation was selected because the estimation of the knee joint angle according to the embodiment of the present invention operates on the CNN image-based model of the regression task. Thus, the output layer is supplied by applying a linear activation function to the sum output of the dense layer (see Fig. 9). For the training parameters, the batch size is 50,500 epochs and the RMSpro optimization with learning of 0.007 is set.

Six subjects walked on a treadmill wearing three types of shoes. GC includes 100 images, and 10 gait cycles for each experimental condition were extracted. Samples were randomly split into 4-1-1 subjects with respect to 12000-3000-3000 images respectively for training, validation and testing.

The results of the predicted knee angle obtained by applying the test image of the foot pressure mapping to the CNN model were 1) 10 points of the moving average filter, 2) a digital filter with a cutoff frequency of 10 Hz, and 3) the predicted result to obtain the knee joint angle. Un-level three-step preprocessing is required.

4) Performance analysis

Knee joint angle estimation according to an embodiment of the present invention tests the performance of CNN image-based models using Mean Absolute Error (MAE), Mean Relative Error (MRE), and correlation coefficient (R) can do.

MAE is defined to observe the error between the estimated value and the actual value without considering its direction. The MRE is calculated to evaluate the goodness of fit of the model during training. The R-value shows how well the regression line represents the predicted value and how well the independent variable is adjusted for variation in the model. The three indicators mentioned above are:

[Equation 2]

[Equation 3]

[Equation 4]

는 실제 값의 평균, 그리고

는 예측 값의 평균이다.where y _i is the actual value, x _i is the predicted value, N is the number of samples,

is the average of the actual values, and

is the average of the predicted values.

Estimation of the knee joint angle according to an embodiment of the present invention uses three matrices to evaluate the performance of the CNN image-based model integrated with the foot pressure mapping image. Figure 11 shows 10 gait cycles of knee joint angles compared with CNN model predictions and reference data for three types of shoes. The CNN image base can accurately measure the knee joint angle and shows a similar pattern between the estimated knee joint angle and the actual knee joint angle. Accuracy for the three test shoe types is shown in FIG. 12 . The MAE (deg) of the shoe was 9.8, 8.9, and 7.2 for F1, F2, and F3, respectively. Also, referring to FIG. 12, the MRE (%) of the shoes were 7.0, 6.0, and 5.0 for the three shoe types F1, F2, and F3, respectively. In addition, the R-values (%) of the CNN image-based implementation for estimating the knee joint angles of F1, F2, and F3 were 72.0, 80.0, and 75.0, respectively. Through this, it can be seen that similar prediction results are shown for the three types of shoes, and therefore, the CNN image base shows excellent adaptability to different types of shoes.

To estimate the knee joint angle according to an embodiment of the present invention, the performance of CNN image-based models was investigated using foot pressure mapping images when walking with a comfortable gait wearing various types of shoes. Although trained without image pattern data in the swing phase (see Fig. 8), the CNN image base can predict the knee joint angle with an MRE of less than 10% (see Fig. 12), and for the three types of shoes, the GC's reference data and A similar knee joint angle pattern can be provided (see FIG. 11). Since CNN image-based networks are highly dependent on input image patterns, large amplitude differences during the swing phase may lead to poor prediction.

Joint angles can be evaluated through numerical data obtained from different sensors using machine learning or deep learning algorithms. In addition, joint angles could be obtained using image data for constructing a CNN model.

Knee joint angle estimation according to an embodiment of the present invention focuses on using CNN and foot pressure images. When F1, F2, and F3 shoes were worn, MAE (deg) of 9.8, 8.9, and 7.2 angles of the left knee joint was measured using 100 GC images. According to the present invention, the MAE values were similar for all three test shoes. Therefore, the CNN image base is highly adaptable to various types of shoes. In the embodiment of the present invention, 100 images were extracted from GC data. In this case, as more images are used, a lower MAE may be indicated.

In outdoor or home monitoring, it is desirable to minimize complicated system settings, difficulty in finding sensor alignment positions, and inconvenience of attaching sensors to the skin or body joints required for the user to use the measurement system according to the embodiment of the present invention. . In the embodiment of the present invention, it is only necessary to insert the sensor into the insole.

In addition, the deep learning approach is an end-to-end process, which is data automatic feature extraction, and the input image data is directly applied to the deep learning model. Deep learning attempts to learn high-level features from data, and the implementation of feature extraction plays an essential role in biomedical applications in providing accurate performance that has been omitted from deep learning processes. This is a breakthrough for deep learning over machine learning, and can be considered more suitable for medical applications.

13 is a flowchart illustrating a method for estimating a knee joint angle according to an embodiment of the present invention. The knee joint angle estimating method according to an embodiment of the present invention may be performed by the knee joint estimating device 100 shown in FIG. 1 .

1 to 13 , the detection sensor 102 detects at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle when the subject is walking (S102). At this time, the detection sensor 102 may be implemented as an F-Scan sensor mounted on the insole of the shoe.

The F-Scan sensor is a commercial product and can be defined as a clinical tool capable of detecting pressure information on the sole of the foot. In particular, the F-Scan sensor is accurate, reliable and cost effective. Here, the F-Scan sensor is made of an insole made of a flexible polymer material and can be implemented as a matrix system including an ink-based force sensing resistor having a thickness of 0.15 mm or less. That is, the F-Scan sensor has 960 individual pressure sensing positions and is composed of 4 cells per unit area (cm ² ). At this time, each cell measures the force, and the average pressure for each cell can be calculated using system software. In this case, sensing elements are arranged in rows and columns of the F-Scan sensor, and data can be displayed in two dimensions on a computer screen through system software. In addition, vertical ground stress, pressure, peak pressure, contact area, and gait cycle time data can be obtained through the system software corresponding to the F-Scan sensor, and such data can be saved as numerical values and images for further analysis. there is. By applying the F-Scan sensor to various types of shoes, it is possible to collect foot pressure data according to the type of shoes.

The marker 104 is attached to the subject's knee, the upper side of the knee, and the lower side of the knee, respectively. That is, in order to measure the angle of the knee joint in any posture of the subject, a plurality of markers 104 may be attached to the knee, the hip portion above the knee, and the ankle portion below the knee. At this time, the angle measurement software is used to connect the positions of each marker 104, and as shown in FIG. 2, the change in angle of the knee joint can be measured according to the movement of the subject while walking. (S104).

The camera 108 is installed at a set distance from the subject's position, and captures a set number of image frames per unit time for analyzing the subject's gait state (S106). At this time, the joint angle estimator 106 synchronizes the image frame captured by the camera 108, the detection signal detected by the detection sensor 102, and the knee angle measured by each marker 104 to synchronize the knee angle. Joint angles can be estimated. At this time, the joint angle estimation unit 106 synchronizes the video captured by the camera 108, the detection signal detected by the sensor 102, and the knee joint angle measured through the marker 104 in time. , It is possible to display the detection signal and the measurement signal of the knee joint angle together with the video signal photographed by the camera 108 in real time.

The knee joint angle estimating device 100 may synchronize an image frame captured by the camera 108, a detection signal detected by the sensor 102, and a knee angle measured by each marker 104 ( S108).

In addition, as described above, the knee joint angle estimator 100 extracts an image of a set gait cycle from an image frame for a set gait time photographed by the camera 108, and converts the extracted image to a convolutional neural network (CNN). ), it is possible to estimate the knee joint angle corresponding to each shoe by using it as the reference data of the image-based model (S110).

Embodiments according to the present invention have been described above, but these are merely examples, and those skilled in the art will understand that various modifications and embodiments of equivalent range are possible therefrom. Therefore, the protection scope of the present invention should be defined by the following claims as well as those equivalent thereto.

Claims (10)

a sensor for detecting at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle when the subject is walking;
markers respectively attached to the subject's knee, an upper side of the knee, and a lower side of the knee;
a joint angle estimator for measuring a knee angle by connecting each of the markers and estimating a knee joint angle corresponding to a shoe based on the measured knee angle and a detection signal detected by the sensor; and
a camera installed at a set distance from the subject's position and photographing a set number of image frames per unit time for analyzing the subject's walking state;
including,
The joint angle estimation unit estimates a knee joint angle by synchronizing an image frame photographed by the camera, a detection signal detected by the detection sensor, and a knee angle measured by each marker, and photographed by the camera. A knee joint angle estimator characterized by extracting an image of a set gait cycle from an image frame for a set gait time and using the extracted image as reference data for a convolutional neural network (CNN) image-based model. delete delete According to claim 1,
The detection sensor is made of a flexible polymer insole with a thickness of 0.15 mm or less, is composed of a set number or more cells per unit area, and detects the detection position and average pressure based on each cell. Characterized in that Knee joint angle estimator. According to claim 1,
The joint angle estimation unit estimates knee joint angles corresponding to flat shoes with a heel height of 1.0 cm or less, sneakers with a heel height of 3.0 cm or less, and classic pump heels with a height of 9.0 cm or less. Knee joint angle estimator, characterized in that. In the knee joint angle estimating method corresponding to the shoe, performed by the knee joint angle estimating device,
detecting at least one of vertical ground pressure, pressure, peak pressure, contact area, and gait cycle when the subject walks by using a sensor;
measuring a knee angle by connecting markers respectively attached to the subject's knee, an upper side of the knee, and a lower side of the knee;
photographing a set number of image frames per unit time for analyzing a gait state of the subject using a camera installed at a set distance from the position of the subject;
synchronizing an image frame photographed by the camera, a detection signal detected by the detection sensor, and a knee angle measured by each of the markers; and
estimating a knee joint angle corresponding to the shoe based on the measured knee angle and the detection signal detected by the sensor;
Including,
The step of estimating the knee joint angle extracts an image of a set gait cycle from an image frame for a set gait time taken by the camera, and uses the extracted image as reference data of a Convolutional Neural Network (CNN) image-based model. Knee joint angle estimation method, characterized in that. delete delete According to claim 6,
The detection sensor is made of a flexible polymer insole with a thickness of 0.15 mm or less, is composed of a set number or more cells per unit area, and detects the detection position and average pressure based on each cell. Characterized in that Knee joint angle estimation method. According to claim 6,
The step of estimating the knee joint angle corresponds to each of flat shoes with a heel height of 1.0 cm or less, sneakers with a heel height of 3.0 cm or less, and classic pump heels with a height of 9.0 cm or less. Knee joint angle estimation method, characterized in that for estimating the angle.

Images