KR20230101464A - Smart Devices Based Multisensory Approach for Complex Human Activity Recognition System and the method thereof - Google Patents

Abstract

A multi-sensory access system based on a smart device according to an embodiment of the present invention includes a database for receiving and collecting sensor data measured from sensors of at least one smart device to detect a user's activity; a pre-processing unit which removes noise from the sensor data collected in the database and divides it into windows or segments; a feature extractor extracting features by increasing the feature vector size of the preprocessed sensor data; and a classifier for classifying physical activity by applying the extracted feature to a machine learning algorithm.

Description

Smart Devices Based Multisensory Approach for Complex Human Activity Recognition System and the method thereof}

The present invention relates to a multi-sensory access system based on a smart device and a method therefor, and in particular, complex human activity recognition is classified by applying sensor data measured using a sensor function built into a smart device to deep learning learning. It relates to a smart device-based multi-sensory access system and method capable of providing high accuracy and fast calculation.

Recently, sensor-based human activity recognition (HAR) is an emerging field of machine learning that has advantages compared to vision-based human activity recognition. However, vision-based HAR is not favored due to limited portability due to hardware installation, cost, data storage requirements and computation time.

In contrast, human activity recognition systems based on wearable sensors are used in smartphones, watches, and fitness bands with various built-in smart sensors. Sensors such as sensors, barometers, and the like are provided in smart devices.

Due to the sufficient functionality and easy availability of these sensors in a variety of ways, sensor-based human activity recognition systems have become more popular.

On the other hand, sensor-based activity recognition is more effective for observing less repetitive activities (e.g., hand/arm movements) on smart watches, and better results for observing repetitive activities (i.e., overall body movements) on smartphones. provides This means that sensor location also plays an important role in activity recognition.

Therefore, recently, studies that can obtain better results by using a smart phone and a smart watch sensor together are active. In other words, basic human activities (e.g. sitting, walking, running, going up and down stairs) are recognized as a smartphone (pocket location), but complex human activities (eating, smoking, talking, etc.) involve hand movements. Because of this, it can be easily recognized using a wrist wearable sensor.

For example, a complex activity may overlap with a basic activity such as smoking a cigarette while walking or sitting, walking or eating ice cream. These activities are recognized using multiple sensors, but a combination of wrist-worn sensors and smartphone sensors effectively recognizes the type of activity a person is performing.

However, although reliable recognition of complex human activities can present a new direction for HAR applications, there is a problem of poor recognition accuracy.

Korean Patent Application No. 10-2008-0068467

In order to solve the above problems, the present invention can provide human activity recognition classification with high accuracy and fast calculation by applying sensor data measured using a sensor function built into a smart device to deep learning learning for complex human activity recognition. Its purpose is to provide a smart device-based multi-sensory access system and method.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, a smart device-based multi-sensory access system according to an embodiment of the present invention uses sensor data measured from at least one sensor of a smart device to detect a user's activity. database that receives and collects input; a pre-processing unit which removes noise from the sensor data collected in the database and divides it into windows or segments; a feature extractor extracting features by increasing the feature vector size of the preprocessed sensor data; and a classifier for classifying physical activity by applying the extracted feature to a machine learning algorithm.

Here, in particular, the sensor data collected in the database is characterized in that it is sensor data measured from four three-axis sensors, accelerometers, gyroscopes, linear acceleration sensors, and magnetometers, each included in smart devices such as smart phones and smart watches. there is.

Here, in particular, the 12-dimensional data measured from each of the four 3-axis sensors in the smart phone and smart watch collected in the database labels each sample and fuses the data of the two devices in parallel. It has a characteristic.

Here, in particular, the feature vector of the feature extractor is characterized in that it is composed of a rotation feature, a time domain feature, and a frequency domain feature.

Here, in particular, the rotational characteristics are the pitch of rotation about the x-axis (*?*), the roll of rotation about the z-axis (φ), and the acceleration vector of instantaneous rotation (α) derived from the orientation of the device. ) is included in the calculation.

Here, in particular, the time domain feature is characterized in that dispersion of pitch σ ² θ _i , roll σ ² φ _i , and acceleration magnitude σ ² α _i is calculated using a window.

), 롤(fφ) 및 가속도 크기(fα)의 데이터를 계산하는 점에 그 특징이 있다.Here, in particular, the frequency domain feature is the pitch (f) using a fast Fourier transform (FFT)

), roll (fφ), and acceleration magnitude (fα) are calculated.

Here, in particular, the classifier is characterized in that it uses machine learning algorithms of Naive Bayes (NB), K-Nearest Neighbors (KNN), and Neural Network (NN). .

In addition, as a technical means for achieving the above-described technical problem, a smart device-based multi-sensory approach method according to an embodiment of the present invention measures data from at least one sensor of a smart device to detect a user's activity in a database. receiving and collecting the received sensor data; a preprocessing step of removing noise from the sensor data collected in the database and dividing it into windows or segments; extracting features by increasing the feature vector size of the preprocessed sensor data; and

It is characterized in that it includes the step of classifying the physical activity by applying the extracted feature to a machine learning algorithm.

Here, in particular, the sensor data collected in the database is characterized in that it is sensor data measured from four three-axis sensors, accelerometers, gyroscopes, linear acceleration sensors, and magnetometers, each included in smart devices such as smart phones and smart watches. there is.

Here, in particular, the 12-dimensional data measured from each of the four 3-axis sensors in the smart phone and smart watch collected in the database labels each sample and fuses the data of the two devices in parallel. It has a characteristic.

In particular, the feature vector is characterized in that it is composed of a rotation feature, a time domain feature, and a frequency domain feature.

), z축에 대한 회전의 롤(roll)(φ) 및 장치의 방향에서 파생되는 순간 회전의 가속도 벡터(α)를 포함하여 계산하는 점에 그 특징이 있다.Here, in particular, the rotational feature is the pitch of rotation about the x-axis (

), a roll of rotation about the z-axis (φ), and an acceleration vector of instantaneous rotation (α) derived from the direction of the device.

Here, in particular, the time domain feature is characterized in that dispersion of pitch σ ² θ _i , roll σ ² φ _i , and acceleration magnitude σ ² α _i is calculated using a window.

), 롤(fφ) 및 가속도 크기(fα)의 데이터를 계산하는 점에 그 특징이 있다.Here, in particular, the frequency domain feature is the pitch (f) using a fast Fourier transform (FFT)

), roll (fφ), and acceleration magnitude (fα) are calculated.

In particular, the machine learning algorithm is characterized in that it uses Naive Bayes (NB), K-Nearest Neighbors (KNN), and Neural Network (NN) algorithms.

According to any one of the above-described problem solving means of the present invention, complex human activity recognition is applied to deep learning learning using sensor data measured using a sensor function built into a smart device to perform human activity recognition classification with high accuracy and fast calculation. can be provided with

In addition, the present invention can provide correct habit coaching by accurately recognizing complex human activity recognition using several smart sensors to identify bad activities and providing tracking recognition feedback.

1 is a diagram schematically showing the configuration of a multi-sensory access system based on a smart device according to an embodiment of the present invention.
Figure 2 is a diagram showing a schematic architecture of the smart device-based multi-sensory access system of the present invention.
FIG. 3 is a diagram illustrating a multi-sensory approach learning process of FIG. 3;
4 is a diagram showing an example of a chaos matrix of the naive Bayes classifier of the present invention.
5 is a diagram showing an example of a chaos matrix of the K-nearest neighbor classifier of the present invention.
6 is a diagram showing an example of a chaos matrix of the neural network classifier of the present invention.
7 is a schematic flowchart of a smart device-based multi-sensory access method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. In order to clearly describe the present invention, parts irrelevant to the description in the drawings are omitted, and similar reference numerals are assigned to similar parts throughout the specification. In addition, while explaining with reference to the drawings, even if the configuration is indicated by the same name, the drawing number may vary depending on the drawing, and the drawing number is only described for convenience of explanation, and the concept, characteristic, function of each component is indicated by the corresponding drawing number. or the effect is not to be construed as limiting.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part is said to "include" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated, and one or more other components. It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In this specification, 'unit' or 'module' includes a unit realized by hardware or software, or a unit realized by using both, and one unit is realized by using two or more hardware may be, or two or more units may be realized by one hardware.

1 is a diagram schematically showing the configuration of a multi-sensory access system based on a smart device according to an embodiment of the present invention, and FIG. 2 shows a schematic architecture of the multi-sensory access system based on a smart device according to the present invention. FIG. 3 is a diagram illustrating a multi-sensory approach learning process of FIG. 3 .

As shown in FIG. 1, the smart device-based multi-sensory access system 100 according to an embodiment of the present invention includes a database 110, a pre-processing unit 120, a feature extraction unit 130, a classification unit ( 140) and a decision unit 150.

The database 110 receives, collects, and stores sensor data measured from at least one sensor of a smart device in order to detect a user's activity.

The sensor data collected in the database 110 consists of a data set of sensor data measured from four three-axis sensors, accelerometers, gyroscopes, linear acceleration sensors, and magnetometers, each included in smart devices such as smart phones and smart watches. .

In addition, the 12-dimensional data measured from each of the four 3-axis sensors in the smart phone and smart watch collected in the database 110 labels each sample and fuses the data of the two devices by connecting them in parallel. .

More specifically, the database 110 integrates 13 basic human activity data with publicly available data sets, and 13 complex activities and *** data sets of 26 basic and complex activities are formed and stored. Here, the data set can be divided into test and training data.

For example, the data set is an Android application called "Linear data collector V2" used for sensor data logging. It will collect data at a rate of 50 samples per second via accelerometers, gyroscopes, linear acceleration sensors and magnetometers for each human activity performed by multiple individuals of different ages.

Let the application easily interface with the sensor for data recording and collection, and before recording the sensor data, the application can in turn record the participant's name and ID. During activity, the Android device is placed in a specific body position, and data can be classified and collected by the application. That is, data can be collected as a comma-separated value (CSV) file of the smart device. At this time, 14 nested basic and complex activities are tested.

Thus, the final data set contains more than 25 human activities, each with a known label and timestamp, fifty samples (at 50 Hz of each sensor output) over 60 seconds of a variety of human body activities performed by multiple users. activity was collected.

Therefore, a total of (4 * 3 = 12) dimensional data is collected by each device composed of four 3-axis sensors. In particular, 12-dimensional data can be collected on each device, 50 samples per second for a single human activity. A big data set will overcome the misclassification of a sample of all participants, similarly complex human activity recognition systems, and will be able to recognize accurate activities.

In addition, 12-dimensional data from each device (smartphone, smart watch) can be labeled at a frequency of 50Hz, each sample of the data and time stamped.

At this time, complex human activity data is collected from smart phones and smart watches for better recognition, and the data from the two devices are connected in parallel to fuse.

Thus, a sample space dimension can be formed with 24-dimensional data for each human activity.

The pre-processing unit 120 performs a pre-processing process of removing noise from the sensor data collected in the database 110 and dividing it into windows or segments.

More specifically, as a preprocessing process of normalizing raw data measured by sensors built into smart phones and smartwatches, after fusing the sensor data, noise is removed from the preprocessed raw data and divided into windows or segments.

In other words, human body activity recognition is not directly classified using the raw data of the sensor, and the classification task obtains a feature vector with structural data changes obtained from various preprocessing and feature extraction techniques.

Each sensor generates three time series along the x-axis, y-axis, and z-axis.

The feature extraction unit 130 extracts features by increasing the feature vector size of the preprocessed sensor data.

After pre-processing in the pre-processing unit 120, the 3-dimensional (x, y, z) raw data obtained from the four sensors of each smart device includes a 12-dimensional vector. After that, fusion of the sensor and the data of the two devices makes the data 24-dimensional. After that, feature extraction increases the feature vector size to a maximum of 72-dimensional features, which means 9 features in each sensor of each device. In this case, the feature vector may be composed of rotation, time domain, and frequency domain features.

), z축에 대한 회전의 롤(roll)(φ) 및 장치의 방향에서 파생되는 순간 회전의 가속도 벡터(α)를 포함하여 계산하게 된다.Here, the rotational feature of the feature vector is the pitch of rotation about the x-axis (

), the roll of rotation about the z-axis (φ), and the acceleration vector of instantaneous rotation (α) derived from the orientation of the device.

These rotation characteristics are calculated as in Equations 1 to 3 below.

Equation [1]- [3]

Here, sx _i ; sy _i ; sz _i is x; y; z sensor, each device i = 1; 2; 3; 4 ie, smart phone, smart watch, etc.

For the time domain feature of the feature vector, the dispersion of pitch σ ² θ _i , roll σ ² φ _i , and acceleration magnitude σ ² α _i is calculated using a window.

These features are normalized using Matlab's mat2gray() command. That is, several statistical characteristics are derived over a defined period. Six features of each sensor are extracted using the window method. For a given input data Xt, a window of size k is calculated as:

Xt = [x(t); x(t - 1); x(t - 2);. . . x(t - k)]. The window is filtered using an average filter of size q using Matlab's fspecial( ) and imfilter( ) functions.

where q = floor

After obtaining the averaging window for a specific period (t), the following six features for each sensor are computed. That is, the pitch, roll, and acceleration magnitudes can be calculated with unbiased variance by Equation [4] as follows for the window.

Equation [4]

u일 수 있으며, 윈도우에 대한 입력 특성의 평균이고 n은 센서의 숫자이다. 특징 7-9는 윈도우에 대한 고속 푸리에 변환(FFT)의 구성 요소이고, 이전 단계는 각 센서의 9가지 특징을 모두 제공하므로 72가지 특징 세트가 있게 된다. 즉, 샘플을 얻을 때마다 특징 공간 정보는 다음과 같다.where β is θ, φ, α or

It can be u, the average of the input characteristics for the window and n is the number of sensors. Features 7-9 are the components of the Fast Fourier Transform (FFT) over the window, and the previous step provides all 9 features of each sensor, so there are 72 feature sets. That is, whenever a sample is obtained, the feature space information is as follows.

More specifically, the time series component, which is a time domain feature, is data s ₁ of four sensors of smartphone s ₁ ; s ₂ ; s ₃ ; s ₄ is collected, s ₁ is accelerometer sensor data, s ₂ is linear acceleration sensor data, s ₃ is gyroscope sensor data, and s ₄ is magnetometer data. Since each sensor contains an axis (x; y; z) of time-series (t) data, sensor data denoted by s ₁ s ₁ x; s ₁ y; s ₁ z displays the x-axis, y-axis, and z-axis data of the first sensor. Similarly, s ₂ represented by smartphone s ₂ ; s ₃ ; The s ₄ sensor data is ( s ₂ x; s ₂ y; s ₂ z ), ( s ₃ x; s ₃ y; s ₃ z ), and ( s ₄ x; s ₄ y; s ₄ z ) as follows: displayed

Equation [5]

X ₁ (t) =[ s ₁ (x , y , z) . . . s ₄ (x , y , z) ] ---------(5)

The vector dimension of X ₁ (t) is 12 for all samples of each human activity. Similarly, the time series data of a smartwatch sensor is represented by the following formula.

Equation [6]

X ₂ (t) = [ s ₁ (x , y , z)...s ₄ (x , y , z) ]-----(6)

After combining the sensor data using the two devices in time series, X(t) is obtained in the following equation [7].

Equation [7]

X (t) = X ₁ (t) : X ₂ (t)------------(7)

where X(t) is the 24-dimensional raw data from all samples of each human activity.

In addition, the calculated rotational feature V (t) ie, pitch ( θ ), roll ( φ ), and acceleration magnitude (α) of raw data are as shown in Equation [8] below.

Equation [8]

V ₁ (t) = [ s 1 (θ , , α) . . . s 8 (θ , , α) ]----------------(8)

The following 48 statistical features can be obtained using the window method over a defined short period. As shown in Equation [9], the dispersion of the pitch σ ² θ _i , the roll σ ² φ _i , and the magnitude of acceleration σ ² α _i is calculated.

Equation [9]

V ₂ (t) = [ σ ² (θ 1, 1, α 1 ) . . .σ ² (θ 8, 8, α 8 ) ]]----------(9)

), 롤(fφ) 및 가속도 크기(fα)의 데이터를 계산하게 된다. 즉, 고속 푸리에 변환(FFT)을 사용하여 윈도우 크기가 (3 × 3)에 대해 24개의 주파수 영역 특징 Vf 를 다음 식 [10]과 같이 계산된다. In addition, the frequency domain feature of the feature vector is obtained by using a fast Fourier transform (FFT) to obtain a pitch (f

), roll (fφ), and acceleration magnitude (fα) data are calculated. That is, 24 frequency domain features Vf for a window size of (3 × 3) are calculated as in the following equation [10] using fast Fourier transform (FFT).

Equation [10]

Vf = [ f (θ ₁ , ₁ , α ₁ ) . . . f (θ ₈ , ₈ , α ₈ ) ]----------------(10)

Table [1] below shows the contribution of sensors in feature extraction of sensors, and shows calculations for obtaining rotation features, time domain features, and frequency domain features by sensor data from each sensor of a smart phone and a smart watch. .

Finally, the final feature space is obtained for all sensor data, if Z is the total number of features using the following equations (10)-(12).

Therefore, the entire data set is divided into data for training and testing processes in a ratio of 8:2 and can be used for classification process.

4 is a diagram showing an example of a chaos matrix of the naive Bayes classifier of the present invention, FIG. 5 is a diagram showing an example of a chaos matrix of the K-nearest neighbor classifier of the present invention, and FIG. It is a diagram showing an example of a confusion matrix of a neural network classifier.

The classifier 140 applies the extracted features to a machine learning algorithm to classify various physical activities.

As explained earlier, smart phones and smart watches use four sensors in each device to detect human body activity. Nine features are derived from each sensor raw data. However, although various sensors have been used to identify human activities, each sensor-derived function plays an independent role in detecting and recognizing specific activities. The classification algorithm will recognize the 26 basic and complex activities listed previously. Two datasets, the pocket location of the smart phone and the wrist location of the smart watch, are considered and classified using various classification methods.

More specifically, the classification unit 140 uses Naive Bayes (NB), K-Nearest Neighbors (KNN), and Neural Network (NN) for various human physical activities. to be classified.

First, data is divided into two groups to perform classification using Naive Bayes (NB). 80% of the data is used for classifier training and the remaining 20% is used as test data. Let P denote the sensor reading of a single class (where J is the total number of classes), and xp denote the pth sensor reading of the training data set. The elements of xp are the 12 sensor readings (3 axes for 4 sensors).

Collect the entire training data and transform the data into a single vector. It is then used to calculate the frequency of each value and construct the probability of each value in the class. This will give p(xijy = Cj) for each value xi and each class Cj. It then takes all elements of the new reading xtesti to determine if the sensor reading belongs to class or b and is calculated as:

Equation [14] -[15]

Here, if (14) > (15), we can assume that the new reading belongs to the class. Otherwise the new reading will belong to class b. Classification can be easily extended to comparison of multiple classes by taking the maximum.

In addition, the classification applying the K-Nearest Neighbors (KNN) algorithm is as follows: [16].

Equation [16]

This provides the entire class compared with C _j and assigns the number of neighbors K to each number of human body activities for human activity recognition using KNN. Here, 26 activities are assigned as "k number Neighbor". For several human activity recognition systems, KNN is used as a multi-class classifier. At this time, the Euclidean distance between the data points is calculated. The distance between the two points is the Euclidean distance and is calculated as the following equation [17].

Equation [17]

Similarly, the above procedure is repeated for multiple points. Calculate the KNN Euclidean distance from the mean of each neighboring group. After re-assigning each data point to a new class with the minimum distance, the centroid of this group of neighbors is computed. The process is repeated until there are few fixed numbers left for the data points.

In addition, classification is performed by applying a Neural Network (NN) algorithm. Here, the parameters of the neural network (NN), namely the number of neurons and the number of hidden layers, can be experimentally selected until the minimum value of the cross-entropy error of the test data is reached. To overcome the limitations of the MATLAB GUI, simulations can be implemented using MATLAB's neural network toolkit along with custom coding.

For an artificial neural network, the following configuration can be used. Input features = 72, output classes = 26, two hidden layers [110, 75 neurons], Sigmoid Activation function, Conjugate gradient training function, Error Back propagation, 500 epochs. It can be analyzed using several classifiers including Neural Network-Nearest Neighbor, Naive Bayes, Ensemble method AdaBoost Decision Tree. Here, we can use the MATLAB learner GUI application for ensemble purposes.

The determination unit 150 may provide feedback to the user by determining which activity is the pre-stored activity for the human activity recognition classified by the classification unit.

Meanwhile, as shown in FIGS. 4 to 6, classification is performed by applying the aforementioned Naive Bayes (NB), K-Nearest Neighbors (KNN), and Neural Network (NN). shows the accuracy of

That is, the accuracy of the classifier on a percentage basis is calculated as: Percentage of correct classification = 100 * (1 - Cr)), and percentage of misclassification = 100 * Cr). Here, Cr is the confusion value representing the declassification rate, obtained from the MATLAB function confusion that returns false positive, false negative, true positive, and true negative rate information. The neural network has been shown to give an accuracy of 99.340162%.

7 is a diagram schematically illustrating a flow chart for a multi-sensory access method based on a smart device according to an embodiment of the present invention. Here, a detailed description of each step of the smart device-based multi-sensory approach method will be omitted with reference to FIGS. 1 to 6 described above.

As shown in FIG. 7 , in the smart device-based multi-sensory approach method according to an embodiment of the present invention, first, sensor data measured from at least one sensor of the smart device is input to the database to detect the user's activity. A receiving and collecting step is performed (S710). Here, the sensor data collected in the database is sensor data measured from four three-axis sensors included in each of smart devices, such as a smart phone and a smart watch, such as an accelerometer, a gyroscope, a linear acceleration sensor, and a magnetometer. The 12-dimensional data measured from each of the four 3-axis sensors in the smart phone and smart watch collected in the database assigns a label to each sample, and the data of the two devices are connected in parallel and fused.

Then, a preprocessing step of removing noise from the sensor data collected in the database and dividing it into windows or segments is performed (S720).

Next, a step of extracting features by increasing the feature vector size of the preprocessed sensor data is performed (S730). Here, the feature vector may be composed of a rotation feature, a time domain feature, and a frequency domain feature.

), z축에 대한 회전의 롤(roll)(φ) 및 장치의 방향에서 파생되는 순간 회전의 가속도 벡터(α)를 포함하여 계산하게 된다.The rotational feature is the pitch of rotation about the x-axis (

), the roll of rotation about the z-axis (φ), and the acceleration vector of instantaneous rotation (α) derived from the orientation of the device.

The time domain feature calculates the variance of pitch σ ² θ _i , roll σ ² φ _i , and acceleration magnitude σ ² α _i using a window.

), 롤(fφ) 및 가속도 크기(fα)의 데이터를 계산하게 된다.The frequency domain feature uses a fast Fourier transform (FFT) to determine the pitch (f

), roll (fφ), and acceleration magnitude (fα) data are calculated.

Subsequently, a step of classifying the physical activity by applying the extracted feature to a machine learning algorithm is performed (S740). Here, classification is performed using Naive Bayes (NB), K-Nearest Neighbors (KNN), and Neural Network (NN) algorithms as the machine learning algorithm.

Therefore, according to the present invention described above, for reliable recognition of various physical activities of human in daily life, many applications such as eHealth, human tracking for remote monitoring and recognition feedback, coaching, human machine interaction, bad habit driving It can be very useful for applications.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

110: database
120: pre-processing unit
130: feature extraction unit
140: feature classification unit
150: judgment unit

Claims (16)

A database for receiving and collecting sensor data measured from at least one sensor of a smart device in order to detect a user's activity;
a pre-processing unit which removes noise from the sensor data collected in the database and divides it into windows or segments;
a feature extractor extracting features by increasing the feature vector size of the preprocessed sensor data; and
A smart device-based multi-sensory access system including a classifier for classifying physical activities by applying the extracted features to a machine learning algorithm.
According to claim 1,
The sensor data collected in the database is based on a smart device, characterized in that the sensor data measured from the accelerometer, gyroscope, linear acceleration sensor, and magnetometer, which are four three-axis sensors included in smart devices, smart phones and smart watches, respectively. A multi-sensory access system.
According to claim 2,
The 12-dimensional data measured from each of the four three-axis sensors in the smart phone and smart watch collected in the database labels each sample, and connects the data of the two devices in parallel to fuse them Smart device, characterized in that based multi-sensory access system.
According to claim 1,
The feature vector of the feature extraction unit is a smart device-based multi-sensory access system, characterized in that consisting of rotational features, time domain features and frequency domain features.
According to claim 4,
The rotational feature is the pitch of rotation about the x-axis (

), a roll (φ) of rotation about the z-axis and an acceleration vector (α) of instantaneous rotation derived from the direction of the device, characterized in that for calculation.
According to claim 4,
The time domain feature calculates the variance of pitch σ ² θ _i , roll σ ² φ _i , and acceleration magnitude σ ² _α i using a window.
According to claim 4,
The frequency domain feature uses a fast Fourier transform (FFT) to determine the pitch (f

), a smart device-based multi-sensory access system, characterized in that for calculating data of roll (fφ) and acceleration magnitude (fα).
According to claim 1,
The classifier uses a machine learning algorithm of Naive Bayes (NB), K-Nearest Neighbors (KNN), and Neural Network (NN). sensory access system.
Collecting sensor data measured from at least one sensor of a smart device in order to detect an activity of a user in a database;
a preprocessing step of removing noise from the sensor data collected in the database and dividing it into windows or segments;
extracting features by increasing the feature vector size of the preprocessed sensor data; and
A smart device-based multi-sensory approach comprising classifying physical activity by applying the extracted feature to a machine learning algorithm.
According to claim 9,
The sensor data collected in the database is based on a smart device, characterized in that the sensor data measured from the accelerometer, gyroscope, linear acceleration sensor, and magnetometer, which are four three-axis sensors included in smart devices, smart phones and smart watches, respectively. A multi-sensory approach.
According to claim 10,
The 12-dimensional data measured from each of the four three-axis sensors in the smart phone and smart watch collected in the database labels each sample, and connects the data of the two devices in parallel to fuse them Smart device, characterized in that based multisensory approach.
According to claim 9,
The feature vector is a smart device-based multisensory approach method, characterized in that consisting of rotational features, time domain features and frequency domain features.
According to claim 12,
The rotational feature is the pitch of rotation about the x-axis (

), a roll (φ) of rotation about the z-axis and an acceleration vector (α) of instantaneous rotation derived from the direction of the device, characterized in that for calculation.
According to claim 12,
The time domain feature calculates the variance of pitch σ ² θ _i , roll σ ² φ _i , and acceleration magnitude σ ² _α i using a window.
According to claim 12,
The frequency domain feature uses a fast Fourier transform (FFT) to determine the pitch (f

), a multi-sensory approach based on smart devices, characterized in that for calculating data of roll (fφ) and acceleration magnitude (fα).
According to claim 9,
The machine learning algorithm uses a Naive Bayes (NB), K-Nearest Neighbors (KNN), and a Neural Network (NN) algorithm. approach.

Images