WO2020125965A1

WO2020125965A1 - Device and method for detecting user activity by parallelized classification

Info

Publication number: WO2020125965A1
Application number: PCT/EP2018/085809
Authority: WO
Inventors: Sami Mekki; Lin Wang; Daniel Roggen; Stefan Valentin
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2018-12-19
Filing date: 2018-12-19
Publication date: 2020-06-25

Abstract

A device, particularly a mobile device is provided, comprising a plurality of sensors. The mobile device is configured to, in a testing phase, obtain data from the plurality of sensors; apply a plurality of trained classifiers on the obtained data, wherein each trained classifier is associated to one of the sensors; estimate, in each of the plurality of trained classifiers, a probability for each of a plurality of classes of a transportation mode of the device; and determine a transportation mode of the device based on the probabilities estimated in each of the trained classifiers. This disclosure also presents a device, particularly a server device. The server device is configured to, in a training phase, obtain training data of another device, particularly mobile device, comprising a plurality of sensors; associate a classifier to each of the sensors; train each classifier using the obtained training data; and provide the trained classifiers to the another device.

Description

DEVICE AND METHOD FOR DETECTING USER ACTIVITY BY

PARALLELIZED CLASSIFICATION

TECHNICAL FIELD

The present disclosure relates generally to the field of mobile devices, particularly of determining a transportation mode of the devices. To this end, the disclosure proposes a device, e.g. a mobile device, which includes a plurality of sensors and, in a testing phase, is configured to determine the transportation mode. The disclosure also proposes a device such as a server device that, in a training phase, is configured to train a classifier.

BACKGROUND

The transportation mode is an important type of context information that denotes a user mobility status during travel, such as standing, walking, cycling, driving a car, taking a bus, a train, a subway, etc.

Conventional machine learning techniques are widely employed to recognize the transportation mode from multimodal sensor data, such as the accelerometer, the gyroscope, the magnetometer and the Global Positioning System (GPS), the light sensor, the proximity sensor, the camera (e.g., the image sensor), the battery heat, the Near Field Communication (NFC), the Bluetooth, the network modem, etc.

FIG. 13 schematically illustrates a conventional method 1300 for transportation mode recognition from a mobile phone’s sensor data.

In the conventional method 1300, the following steps are performed, in order to determine the transportation mode of the mobile phone 1301.

• (a) Obtaining the multimodal sensor data from the mobile phone (e.g., from the sensors of the mobile phone).

• (b) Segmenting the obtained data into short frames. • (c) Performing feature extraction at individual frames and obtaining a feature vector.

• (d) Mapping the feature vector by a classifier in one of the transportation classes.

• (e) Determining the transportation class of the mobile phone.

For example, the multimodal sensor data is initially segmented into frames with a sliding window. The data in each frame may be used, and a vector of features may be computed. These computed feature vectors are then processed by a classifier, which aims to recognize the transportation mode of the user.

The conventional devices and methods for determining the transportation mode are based on training the classifiers considering all of the available data modalities, as it is shown in FIG. 14.

FIG. 14 schematically illustrates a conventional method 1400 for training a classifier by using all of the available data modalities.

A normal classifier 1401 with M transportation classes and TV sensor modalities is used, and the f_k is obtained. The f_k is a feature vector at the k- th frame including the sub vectors from TV sensor modalities, i.e., the f_k =

The conventional method

1400 further determines the e™ e [0,1] which is the score of each class, and finally the c_k e [1, M] which is the output label may be obtained, e.g., by using the c_k = argma x(e™}.

m

However, the conventional devices and methods have the disadvantages that, for example, in practice, it may happen that, the availability of different sensor modalities varies with the device and the used environment.

Besides, training the classifier with data of all of the sensors (e.g., at the same time) may result in a long training time. For example, during the training phase, all of the possible combinations of the sensor modalities have to be tested in order to obtain the optimal trained classifiers.

The disadvantages of the conventional devices and methods for determining the transportation modes can be summarized as follows: • The classifiers are usually trained by combining all of the sensor modalities, which may lead to a long training phase.

• Not all of the sensors are available at the same time, for training the classifiers.

• Multiple classifiers have to be trained and stored, in order to cover all of the possible combinations of the sensor modalities.

Furthermore, in the conventional methods that employ multiple sensors modalities (e.g., the accelerometer, the GPS and the sound) for determining the transportation mode, it is not determined yet, how to fuse the multiple modalities if they are trained independently.

Moreover, conventional devices and methods are also known that perform the classification and the runtime reduction by the implementation of the machine learning algorithms. For example, the design of Support Vector Machine (SVM) on distributed memory systems, which is related to the implementation and optimization of the algorithm.

In addition, the use of the Graphics Processing Unit (GPUs) are suggested, for example, in order to, e.g., overcome the hardware implementation problem, accelerate the training phase in the SVM, and decrease the training time.

SUMMARY

In view of the above-mentioned problems and disadvantages, the present invention aims to improve the conventional devices and methods for determining the transportation mode.

An objective is in particular to reduce the time of a training phase. Furthermore, the number of classifier that need to be trained and stored, in order to cover all of the possible combinations of the sensor modalities, should be reduced. In addition, transportation mode determination should be fast and reliable. The objective is achieved by the embodiments provided in the enclosed independent claims. Advantageous implementations of these embodiments are further defined in the dependent claims.

In particular the present invention proposes a device and a method that may determine the transportation mode and the locomotion mode of the device (e.g., the user of the device such as a smartphone) from the multimodal smartphone sensor’s data, e.g., by processing each sensor modality independently and in parallel. For instance, the device and the method may perform a parallelization of the training phase for the selected classifiers.

The main advantages of the embodiments of the invention can be summarized as follows:

• Reducing the runtime of the training phase by parallelizing the training and the classification of each sensors independently.

• Performing a post-filtering that may provide the virtual probability that may allow exploiting the temporal correlation between consecutive frames.

• The data fusion may become an easier process. For example, once the virtual probability is computed and it made available for each of the sensor and for each of the classes of the transportation mode, the data fusion may be performed.

A first aspect of the invention provides a device, particularly mobile device, comprising a plurality of sensors, the device being configured to, in a testing phase obtain data from the plurality of sensors; apply a plurality of trained classifiers on the obtained data, wherein each trained classifier is associated to one of the sensors; estimate, in each of the plurality of trained classifiers, a probability for each of a plurality of classes of a transportation mode of the device; and determine a transportation mode of the device based on the probabilities estimated in each of the trained classifiers.

In some embodiments, during a training phase the classifiers may be trained (e.g., by using training data). Moreover, during the testing phase, the trained classifiers may be applied on the obtained data and the transportation mode may be (e.g., rapidly) determined. Moreover, the probability (also hereinafter referred as virtual probability) for each of a plurality of classes of the transportation mode of the device may be estimated.

The estimated (virtual) probability may keep tracking of the temporal correlation between the consecutive frames. Furthermore, once the probability (e.g., the virtual probability) is estimated, the transportation mode may be determined, trough data fusion, and by considering the estimated probabilities of each class in each classifier. In some embodiments, the transportation mode of the device may be determined, moreover, the user activity (i.e., the user of the device) may further be determined, e.g., based on the determined transportation mode.

In an implementation form of the first aspect, the device is further configured to, in the testing phase, estimate the probability for each class in a trained classifier based on the score of the class at the output of the trained classifier and a predefined parameter.

In a further implementation form of the first aspect, the predefined parameter is a predefined number of frames in the obtained data or a predefined time.

In a further implementation form of the first aspect, the device is further configured to, in the testing phase, perform a fusion of the probabilities estimated in each of the trained classifiers, in order to determine the transportation mode of the device.

For example, the probabilities may be estimated and made available for each sensor and for each class of the transportation mode. Afterward, the data fusion may be performed.

In a further implementation form of the first aspect, the device is further configured to perform the fusion of the probabilities by averaging or multiplying the estimated probabilities for each class obtained in the classifiers.

This is beneficial, since the runtime may be decreased.

In a further implementation form of the first aspect, the device is further configured to determine the transportation mode of the device based on the class having the highest score after performing the fusion of the probabilities. This is beneficial, since the transportation mode of the device may be determined.

In a further implementation form of the first aspect, the plurality of trained classifiers are preconfigured in the device.

This is beneficial, since the plurality of trained classifiers may be, for example, trained and preconfigured (i.e., in advance and before the testing phase). Moreover, during the testing phase the trained classifiers may be used and the transportation mode of the device may be determined.

In a further implementation form of the first aspect, the device is further configured to, in a training phase, train a classifier for each of the sensors by: obtaining training data; selecting a sensor; associating a classifier to the selected sensor; and training the associated classifier using the obtained training data, thereby obtaining the trained classifiers.

This is beneficial, since by parallelizing the training and the classification of each sensors, e.g., independently, the duration time of the training phase may be reduced.

In a further implementation form of the first aspect, the device is further configured to obtain the training data from the plurality of sensors of the device and/or obtain the training data from a plurality of sensors of another device.

This is beneficial, since for each sensor from the plurality of the sensors, the training data may be obtained from one or more devices. Moreover, the classifier for each sensor may be selected and trained in order to obtain the (e.g., the optimal) trained classifier for the given sensor.

In a further implementation form of the first aspect, the device is further configured to, in the training phase, train each classifier independently.

This is beneficial, since by training each classifier independently (e.g., and in parallel), the duration time of the training phase may be reduced. In a further implementation form of the first aspect, the device is further configured to, in the training phase: train at least two classifiers at the same time.

This is beneficial, since two or more of the classifiers may be trained in parallel, which may reduce the duration time of the training phase.

In a further implementation form of the first aspect, a classifier is based on one or more of:

• A k-nearest neighbors algorithm, KNN, classifier.

• A decision tree, DT, classifier.

• A naive Bayes, NB, classifier.

• A support vector machine, SVM, classifier.

• Or any other type of classifier

A second aspect of the invention provides a method for a device, particularly a mobile device, comprising a plurality of sensors, the method comprising, in a testing phase, obtaining data from the plurality of sensors; applying a plurality of trained classifiers on the obtained data, wherein each trained classifier is associated to one of the sensors; estimating, in each of the plurality of trained classifiers, a probability for each of a plurality of classes of a transportation mode of the device; and determining a transportation mode of the device based on the probabilities estimated in each of the trained classifiers.

In an implementation form of the second aspect, the method further comprising, in the testing phase, estimating the probability for each class in a trained classifier based on the score of the class at the output of the trained classifier and a predefined parameter.

In a further implementation form of the second aspect, the predefined parameter is a predefined number of frames in the obtained data or a predefined time. In a further implementation form of the second aspect, the method further comprising, in the testing phase, performing a fusion of the probabilities estimated in each of the trained classifiers, in order to determine the transportation mode of the device.

In a further implementation form of the second aspect, the method further comprising performing the fusion of the probabilities by averaging or multiplying the estimated probabilities for each class obtained in the classifiers.

In a further implementation form of the second aspect, the method further comprising determining the transportation mode of the device based on the class having the highest score after performing the fusion of the probabilities.

In a further implementation form of the second aspect, the plurality of trained classifiers are preconfigured in the device.

In a further implementation form of the second aspect, the method further comprising, in a training phase, training a classifier for each of the sensors by: obtaining training data; selecting a sensor; associating a classifier to the selected sensor; and training the associated classifier using the obtained training data, thereby obtaining the trained classifiers.

In a further implementation form of the second aspect, the method further comprising obtaining the training data from the plurality of sensors of the device and/or obtaining the training data from a plurality of sensors of the another device.

In a further implementation form of the second aspect, the method further comprising, in the training phase, training each classifier independently.

In a further implementation form of the second aspect, the method further comprising, in the training phase, training at least two classifiers at the same time.

In a further implementation form of the second aspect, a classifier is based on one or more of: • A k-nearest neighbors algorithm, KNN, classifier.

• A decision tree, DT, classifier.

• A naive Bayes, NB, classifier.

• A support vector machine, SVM, classifier.

• Or any other type of classifiers.

A third aspect of the invention provides a device, particularly server device, configured to, in a training phase, obtain training data of another device, particularly mobile device, comprising a plurality of sensors; associate a classifier to each of the sensors; train each classifier using the obtained training data; and provide the trained classifiers to the another device.

Moreover, in some embodiments, the duration time of the training phase may be reduced, for example, by exploiting the temporal correlation between consecutive frames, and by performing a post-filtering process which may provide a virtual probability for each class of the transportation mode and for each sensor.

In some embodiments, each relevant sensor modality of the device (e.g., the mobile device, the smartphone) may be trained independently with the optimal classifier. Moreover, during the training phase, the classifiers (e.g., two or more classifiers) may be processed in parallel which may reduce the overall run time of the training phase. The device may consider any new sensors or eliminate a considered sensor, without training the classifier from the beginning. In some embodiments, any sensors may be added or set aside, at any time of the runtime of the transportation mode.

A fourth aspect of the invention provides a method comprising, in a training phase, obtaining training data of a device, particularly mobile device, comprising a plurality of sensors; associating a classifier to each of the sensors; training each classifier using the obtained training data; and providing the trained classifiers to the device.

It has to be noted that all devices, elements, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.

BRIEF DESCRIPTION OF DRAWINGS

The above described aspects and implementation forms of the present invention will be explained in the following description of specific embodiments in relation to the enclosed drawings, in which

FIG. 1 schematically illustrates a device comprising a plurality of sensors for determining a transportation mode in a testing phase, according to an embodiment of the invention.

FIG. 2 schematically illustrates a device for training a classifier and providing the trained classifier to another device, in a training phase, according to an embodiment of the invention.

FIG. 3 schematically illustrates a procedure for determining the transportation mode of the device, according to various embodiment of the invention.

FIG. 4 schematically illustrates a window comprising a predefined number of frames, according to various embodiments of the invention.

FIG. 5 schematically illustrates a flowchart of a procedure for training the classifiers and determining the transportation mode of the device, according to various embodiment of the invention. FIG. 6 schematically illustrates a method for determining the transportation mode of the device by performing a parallel classification over each sensor independently, according to various embodiment of the invention.

FIG. 7 schematically illustrates an example of the transportation mode ground truth of the device, according to various embodiment of the invention.

FIG. 8 illustrates the classification results corresponding to the original score of each class with different modalities, according to various embodiment of the invention.

FIG. 9 illustrates exemplary estimated probabilities for three different classes of transportation mode, according to various embodiment of the invention.

FIG. 10a illustrates the probabilities data fusion, according to various embodiment of the invention. FIG. 10b illustrates determining the transportation mode of the device after performing the data fusion, according to various embodiment of the invention.

FIG. 11 schematically illustrates a method for determining a transportation mode in the testing phase, according to an embodiment of the invention.

FIG. 12 schematically illustrates a method for training a classifier and providing the trained classifier, in the training phase, according to an embodiment of the invention.

FIG. 13 schematically illustrates a conventional method for the transportation mode recognition from a mobile phone sensor’s data.

FIG. 14 schematically illustrates a conventional method for training a classifier by using all of the available data modalities. DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 schematically illustrates an embodiment of a device 100 comprising a plurality of sensors 101, 102 for determining a transportation mode 110, 120, 130, according to an embodiment of the invention.

The device 100 may be, for example, a mobile device that has a plurality of sensors 101,

102.

The plurality of the sensors may be, for example, a GPS sensor, an accelerometer sensor, a gyroscope sensor, etc.

The device 100 is configured to, in a testing phase, obtain data 111, 112 from the plurality of sensors 101, 102. For example, the data may be obtained directly from the plurality of the sensors of the mobile device.

The device 100 is further configured to apply a plurality of trained classifiers 121, 122 on the obtained data 111, 112, wherein each trained classifier 121, 122 is associated to one of the sensors 101, 102.

The device 100 is further configured to estimate, in each of the plurality of trained classifiers 121, 122, a probability for each of a plurality of classes of a transportation mode 110, 120, 130 of the device 100.

The device 100 is further configured to determine a transportation mode 110, 120, 130 of the device 100 based on the probabilities estimated in each of the trained classifiers 121, 122.

For example, the device 100 may estimate the probabilities for each of a plurality of classes of the transportation mode 110, 120, 130. Moreover, the temporal correlation between the consecutive frames may be tracked. Furthermore, once the probabilities are estimated, the device 100 may perform the data fusion, and the transportation mode 110, 120, 130 may be determined. FIG. 2 schematically illustrates an embodiment of a device 200 for training a classifier 221, 222 and providing the trained classifier 121, 122 to another device 100, in the training phase, according to an embodiment of the invention.

The device 200 may be, for example, a server device. In some embodiments, the device 100 of FIG. 1 and the device 200 of FIG. 2 may be the same device or based on an identical device, without limiting the present invention to a particular device.

The device 200 is configured to, in a training phase, obtain training data 211, 212 of another device 100, particularly mobile device, comprising a plurality of sensors 101, 102.

The device 200 is further configured to associate a classifier 221, 222 to each of the sensors 101, 102.

The device 200 is further configured to train each classifier 221, 222 using the obtained training data 211, 212.

The device 200 is further configured to provide the trained classifiers 121, 122 to the another device 100.

Moreover, the device 200 may reduce the duration time of the training phase. For example, the device 200 may train each relevant sensor modality of the mobile device 100 independently. Moreover, the classifiers 221, 222 may be trained in parallel which may reduce the overall run time of the training phase. The device may allow to consider any new sensors or eliminate a considered sensor 101, 102.

FIG. 3 schematically illustrates a procedure 300 for determining the transportation mode 110, 120, 130 of the device 100 according to various embodiment of the invention. The procedure 300 may be (e.g., fully or partially) performed by the device 100 and/or the device 200, as described above.

In the procedure 300 of the FIG. 3, the f_{n k} is the k^th frame of the n^lh sensor, the _{n k} is the score of the m^lh class for the n^th sensor modality of the k^th frame, the P_{n k} is the probability (e.g., virtual probability) of the m^th class for the n^lh sensor modality of the k* frame, the p™ is the result of modality fusion, and the C_k is the output label (e.g., transportation mode of the device).

For each sensor 101, 102, a specific classifier (e.g., 221, 222) is selected and trained, in parallel. The classifier of the n^lh modality may be any type of classifier, for example, a k- nearest neighbours algorithm (KNN) classifier, a decision tree (DT) classifier, a naive Bayes (NB) classifier, a support vector machine (SVM) classifier, etc.

Moreover, at the

frame, the n^th classifier outputs a score e'"_/ e [0,1] for each candidate of transportation class from the plurality of classes of the transportation mode 110, 120, 130. Then the virtual probability of each class in each classifier may be computed, e.g., considering the scores in a window of L frames, where L is a pre-defined parameter, for example, it may be the number of frames in a 2-minute period.

Moreover, the probability p™_k (e.g., the virtual probability) may be defined as the average of score of each class over the L frames in the classifier n, according to Eq. (1) as follow:

Furthermore, after the computation of the virtual probabilities in each classifiers (for example, all of the virtual probabilities), a modality fusion may be performed, in order to compute the final joint probability /?'" of class m at the frame k .

In some embodiments, the final join probability /?'" may be computed, for example, as the average of N over the p™_k , and according to the Eq. (2) as follow:

In some embodiments, it may be assumed that the sensors 101, 102 are independently manufactured. Moreover, the final join probability p_k may be computed, using the Eq. (3) as follow:

N

m

P_k _QC P (3) n= 1 *

The output of data fusion block (i.e., the final join probability /?'" ) may allow to extract the transportation mode c_k of the device (e.g., the transportation mode of the user of the device, detecting the user activity, etc.) at the k* frame, for example, by finding the m^th index that maximizes, according to Eq. (4) as follow:

c_k = arg maxi/ ) , (4) m

The device 100 may determine the transportation mode, as described above.

FIG. 4 schematically illustrates a window 400 comprising a predefined number ( L ) of frames, according to various embodiments of the invention.

In the window 400 of L frames, the L is the pre-defined parameter. For example, in some embodiments, it may be the number of frames in a 2-minute period.

FIG. 5 schematically illustrates a flowchart of a procedure 500 for training the classifiers 221, 222 and determining the transportation mode 110, 120, 130 of the device 100, according to various embodiment of the invention. The procedure 500 may be (e.g., fully or partially) performed by the device 100 and/or the device 200, as described above. In some embodiments, the training phase 500a may be performed by the device 200 and the testing phase 500b may be performed by the device 100, as described above, without limiting the invention. Moreover, in some embodiments, the device 100 and/or the device 200 may (e.g., fully or partially) perform the training phase 500a and/or the testing phase 500b.

In the training phase 500a, the following steps may be performed, for example, by the device 200.

• At 501, the device 200 obtains training data 211, 212 from the smartphone’s sensors 101, 102.

• At 502, the device 200 selects sensors for the transportation mode detection.

• At 503, the device 200 define a classifier 221, 222 for each selected sensor 101,

102.

• At 504, the device 200 trains all of the classifiers 221, 222 at the same time or sequentially.

• At 505, the device 200 determines the classification model for each sensor 101, 102. For example, the device 200 may obtain the trained classifiers 121, 122 for the sensors 101, 102.

The device 200 may further provide the trained classifiers 121, 122 (i.e., specifically trained for the sensors 101, 102) to the device 100. Moreover, the device 100 may further determine the transportation mode in the testing phase.

In the testing phase 500b, the following steps may be performed, for example, by the device 100.

• At 506, the device 100 obtains data 111, 112 from the smartphone sensors 101,

102.

• At 507, the device 100 applies the classification model (e.g., the trained classifiers 121, 122) for each sensor 101, 102 obtained from the training phase 500a.

• At 508, the device 100 compute the virtual probability of each class in each classifier. • At 509, the device 100 uses the virtual probability for data fusion.

• At 510, the device 100 determines the transportation mode 110, 120, 130 of the device 100 (e.g., the user of the device 100).

The device 100 and/or the device 200 may determine the transportation mode and/or the locomotion of the devices (e.g., the users of the devices).

FIG. 6 schematically illustrates an embodiment of a method 600 for determining the transportation mode of the device 100 by performing a parallel classification over each sensor independently, according to various embodiment of the invention. The procedure 600 may be performed by the device 100 and/or the device 200.

At 601, the device 100 obtains the trained classifiers 121, 122. For example, each sensor modality of the smartphone is trained, independently (e.g., as it is illustrated in FIG. 3). The selected classifier for each sensor may be any type of the classifiers, for example, the KNN, the DT, the NB, the SVM, etc.

In the following, the device 100 and the method 600 are described based on the DT classifier in order to not be biased by the classifier performances, without limiting the invention to a specific classifier.

At 602, the device 100 computes the virtual probabilities. For example, after obtaining the score of each class at the output of each classifier 121, 122, the device 100 computes the virtual probability in a window of L frames. Moreover, the time correlation between consecutive frames may be tracked. The L_p is a pre-defined parameter, which may be the number of frames in a 2-minute period.

In some embodiments, for a class m of sensor n , the scores at frames [k - L_p + 1

may be obtained as [e™_{k-L +1},5 , e™_k] . Moreover, the virtual probability of class m , which may further be used for the post processing, may be computed, for example, by using the Eq. (1) described above.

At 603, the device 100 performs the modality fusion. The modality fusion may fuse the data from all sensors in order to detect the transportation mode of the user. The modality fusion module may compute the probability of each class per frame based on the virtual probabilities computed at step 602. The data fusion may be performed by for example, at least one of the following two approaches.

(1) The probability of class m at frame k may be obtained by taking the average of the virtual probabilities of class m of all sensors at frame k over the number of involved sensors, e.g., according to Eq. (2) as described above.

(2) In some embodiments, the sensors may be assumed that are independently manufactured. Moreover, the probability of class m at frame k may be proportional to the product of all virtual probabilities of class m of all sensors at frame k , e.g., according to Eq. (3) as described above.

Furthermore, the proportionality factor oc may be determined by using Eq. (5) as follow:

At 604, the device 100 determines the transportation mode.

For example, the transportation mode may be extracted from the computed probabilities and by selecting the class that provides the highest probability at the frame k , e.g., according to Eq. (4) as described above.

In the following, the device 100 is described based on being a smartphone that collects three sensors modality including the accelerometer, the gyroscope and the magnetometer.

FIG. 7 schematically illustrates an example of the transportation mode ground truth 700 of the device 100, according to various embodiment of the invention. The device 100 transportation mode ground truth 700 (for example, the user activity ground truth) as it is illustrated in FIG. 7 includes three different classes. The“X” axis in the FIG. 7 indicated with the frame index and represents the frame indexes from 6000 to 7500. The “Y” axis indicated with the “Class” and the three different classes are represented. The transportation mode having the class of 1 (class 1) is corresponding to the still, the class2 is corresponding to the car, and the class3 is corresponding to the train.

Figure 8 illustrates the classification results 800 corresponding to the original score of each class with different modalities, according to various embodiment of the invention.

Each sensor has been assigned a classifier, for example, a same classifier of the DT had been assigned to all of the sensors in order to show the impact of parallelization. The score output of each classifier for the different modalities is shown in FIG. 8. The rapid change of the device transportation mode (e.g., the user activity) determination at the output of each classifier may be filtered by the virtual probability block that may provide a more stable and with a slower variation, as it is depicted in the exemplary results 900 of FIG. 9. FIG. 9 illustrates the exemplary results 900 of estimated probabilities for three different classes of transportation mode, according to various embodiment of the invention. Moreover, these outputs may further be transferred to the modality fusion block in order to obtain the real probability of each class, for example, based on the three sensors (FIG. 10a).

The processing and the fusion result are illustrated in FIG. 10a and FIG 10b, respectively. FIG. 10a illustrates the virtual probability fusion result, according to various embodiment of the invention. The transportation mode of the device 100 (e.g., the user activity) may be extracted by taking the class that provides the highest probability at each frame as it is shown in FIG. 10b. FIG. 10b illustrates the transportation mode determination of the device 100 (e.g., the user activity detection), after performing the data fusion, according to various embodiment of the invention.

Table 1 shows the baselines accuracy and their corresponding confusion matrix for determining the transportation mode of the device (e.g., the user activity detection), without any data post-processing of the classifier output (e.g., no virtual probability is computed). Note that this confusion matrix is related to a specific experiment case and is provided as an example. The given numbers do not have to be exactly these values. The first three columns including the Accelerometer (Ace), the Gyroscope (Gyr), and the Magnetometer (Mag) show the results when the sensors are exploited, independently from each other, and without performing the modality fusion. The last column of table 1, specified with the“Acc+Gyr+Mag” shows the result of a normal classifier when the sensor modalities are considered at the same time, for example, as it is represented in FIG. 14.

Moreover, when all of the sensors are taken into account, the highest accuracy of the user activity detection may be obtained. The final result may be considered as the baseline performance for determining the transportation mode of the device (e.g., the user activity detection).

Table 1: Baseline results of the user activity detection without performing the post processing.

Table 2 shows the impact of performing post processing on the accuracy results. Moreover, it can be derived that the virtual probability that takes into account the temporal correlation between consecutive frames and is computed at the output of the classifier, may improve the detection performance (e.g., determining the transportation mode, detecting the user activity, etc.) of the baseline method.

Table 2: Results of user activity detection with post-processing.

Table 3 shows the detection performance obtained based on the data fusion of the sensors modalities and is computed after performing the post processing.

Table 3: Accuracy results with data fusion.

Using the data fusion of the sensors modalities being computed after performing the post processing, may provide comparable results with a classifier trained using the three modalities.

In Table 4-A the detection performance of a normal classifier is shown including the different sensors modalities. These results are comparable with the results being obtained based on the device and method discussed in this invention, in which each sensors is processed independently, then a data fusion is performed in order to obtain the result. The results are listed in Table 4-B.

Table 4- A: Accuracy of user detection with normal classifier method trained with all sensors each time and obtained with post processing.

Table 4-B: Accuracy of user detection with fusion of sensors results ( e.g., based on disclosed method of this invention). The performance in the both cases are comparable, i.e., the performance with a normal classifier trained with all sensors at the same time, and the performances based on a parallel training of each sensors, independently, followed by performing data fusion for the all sensors. In the device and method discussed in this invention, the classifiers for each of the sensors may be trained only once, and the results for a specific combination of sensors may be obtained through data fusion, i.e., performing the modality fusion. However, when the normal classification is applied (e.g., see FIG. 14), the classifiers are trained at the same time even when the included sensors change. Thus, the normal classification (the conventional methods) requires more time for training than the device and method of the invention. Table 5 shows the running time of the training phase for each sensor combination. The results provided in Table 5 represent a show case (i.e. an example) and should not be taken as unique references since, for example, the results may depend on the computer performances.

Table 5: Run time of the conventional classification for each sensors modalities. In the conventional methods, for a classification covering all of the seven combination of modalities, 91.5s is required (i.e., 7.4+7.7+7.8+26.3+13.3+15.2+13.8), which is the sum of the training time of all of the modalities combinations listed in Table 5.

Moreover, if the computer resources are provided, it may be assumed that the training phase may run in parallel and it may only last the maximum time of the training phase listed in table 5, which is 26.3s. The maximum time of the training phase may depend, for example, on the computer performances, and the values provided in Table 5 may depend on the computer used for the training. In other words, these values are not fixed or unique, but are examples.

The device and methods of the invention may allow to save time. For example, it may be only required to train the involved sensors independently, and without considering any combination during the training phase. Therefore, the total training time may take 22.9s (i.e., 7.7+7.4+7.8), when each sensor is trained, sequentially. In addition, it may take 7.8s, if all of the sensors are trained in parallel (for example, in a powerful machine, in a server device, etc.).

In some embodiments, the method may allow saving the time during the training phase, for example, around 70% of the run time may be saved. In some embodiments, different modality fusion procedures may be applied. For example, the data fusion for the sensors may be performed using different methods.

In some embodiments, the data fusion may be based on the average of the virtual probabilities of the involved sensors per class and per frame. In some embodiments, the data fusion may be based on computing the product of all virtual probabilities of the sensors involved in the user activity detection per class and per frame. Moreover, it may further be normalized over the sum of the all of the computed probabilities per frame. For instance, it may be equivalent to compute the probability of each class per frame by using

the transportation mode or the user activity detection, which is given by the Eq. (6), as follow:

In some embodiments, the data fusion may be performed by computing the joint probability for all of the sensors, when sensors dependencies are available.

In some embodiments, the sensors scalability may be performed. Moreover, the device and the method may be scalable to different sensor modalities. For example, the device and/or the method may be capable to include new sensor, without processing the data from the beginning. That means, only the data from the new sensor may be used in order to train the classifier which may also be different from the classifiers that are already used, for example, the trained classifiers applied on the obtained data for determining the transportation mode (e.g., the activity detection).

The classification result of the new sensor may further be integrated to the device and/or the method (e.g., for determining the transportation mode, the activity detection) by using the modality fusion, as it is described above. In some embodiments, it may be possible to remove any sensor modalities from the device and/or the method (e.g., for determining the transportation mode, the activity detection) without any further processing. For example, it may require only updating the equation used for the modality fusion which may save time in the data processing.

Note that, the described device and method are not limited to a specific classifier. Any classifier may be assigned for the sensor’s data. The classifier may further be trained based on a single sensor modality.

In some embodiments, the sensor scalability (addition or removing of the sensors) may be used.

FIG. 11 shows a method 1100 according to an embodiment of the invention for determining a transportation mode 110, 120, 130 in the testing phase. The method 1200 may be carried out by the device 100, as it described above.

The method 1100 comprises a step 1101 of obtaining data 111, 112 from the plurality of sensors 101, 102.

The method 1100 further comprises a step 1102 of applying a plurality of trained classifiers 121, 122 on the obtained data 111, 112, wherein each trained classifier 121, 122 is associated to one of the sensors 101, 102.

The method 1100 further comprises a step 1103 of estimating, in each of the plurality of trained classifiers 121, 122, a probability for each of a plurality of classes of a transportation mode 110, 120, 130 of the device 100.

The method 1100 further comprises a step 1104 of determining a transportation mode 110, 120, 130 of the device 100 based on the probabilities estimated in each of the trained classifiers 121, 122.

FIG. 12 shows a method 1200 according to an embodiment of the invention for training a classifier 211, 212 and providing the trained classifier 121, 122 to another device 100, in the training phase. The method 1300 may be carried out by the device 200, as it described above.

The method 1200 comprises a step 1201 of obtaining training data 211, 212 of a device 100, particularly mobile device, comprising a plurality of sensors 101, 102.

The method 1200 further comprises a step 1202 of associating a classifier 221, 222 to each of the sensors 101, 102.

The method 1200 further comprises a step 1203 of training each classifier 221, 222 using the obtained training data 211, 212.

The method 1200 further comprises a step 1204 of providing the trained classifiers 121, 122 to the device 100.

The present invention has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed invention, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word“comprising” does not exclude other elements or steps and the indefinite article“a” or“an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

1. Device (100), particularly mobile device, comprising a plurality of sensors (101, 102), the device (100) being configured to, in a testing phase:

obtain data (111, 112) from the plurality of sensors (101, 102);

apply a plurality of trained classifiers (121, 122) on the obtained data (111, 112), wherein each trained classifier (121, 122) is associated to one of the sensors (101, 102); estimate, in each of the plurality of trained classifiers (121, 122), a probability for each of a plurality of classes of a transportation mode (110, 120, 130) of the device (100); and

determine a transportation mode (110, 120, 130) of the device (100) based on the probabilities estimated in each of the trained classifiers (121, 122).

2. Device (100) according to claim 1, configured to, in the testing phase:

estimate the probability for each class in a trained classifier (121, 122) based on the score of the class at the output of the trained classifier (121, 122) and a predefined parameter.

3. Device (100) according to claim 2, wherein the predefined parameter is a predefined number of frames in the obtained data (111, 112) or a predefined time.

4. Device (100) according to any one of claims 1 to 3, configured to, in the testing phase:

perform a fusion of the probabilities estimated in each of the trained classifiers (121, 122), in order to determine the transportation mode (110, 120, 130) of the device (100).

5. Device (100) according to claim 4, configured to:

perform the fusion of the probabilities by averaging or multiplying the estimated probabilities for each class obtained in the classifiers (121, 122).

6. Device (100) according to any one of claims 4 or 5, configured to:

determine the transportation mode (110, 120, 130) of the device (100) based on the class having the highest score after performing the fusion of the probabilities.

7. Device (100) according to any one of claims 1 to 6, wherein the plurality of trained classifiers (121, 122) are preconfigured in the device (100).

8. Device (100) according to any one of claims 1 to 7, further configured to, in a training phase, train a classifier (221, 222) for each of the sensors (101, 102) by:

obtaining training data (211, 212);

selecting a sensor (101, 102);

associating a classifier (221, 222) to the selected sensor (101, 102); and training the associated classifier (221, 222) using the obtained training data (211, 212), thereby obtaining the trained classifiers (121, 122).

9. Device (100) according to claim 8, configured to:

obtain the training data (211, 212) from the plurality of sensors (101, 102) of the device (100) and/or obtain the training data (211, 212) from a plurality of sensors of the another device.

10. Device (100) according to any one of claims 8 or 9, configured to, in the training phase:

train each classifier (221, 222) independently.

11. Device (100) according to any one of claims 8 to 10, configured to, in the training phase:

train at least two classifiers (221, 222) at the same time.

12. Device (100) according to any one of claims 1 to 11, wherein a classifier is based on one or more of

a k-nearest neighbors algorithm, KNN, classifier;

a decision tree, DT, classifier;

a naive Bayes, NB, classifier;

a support vector machine, SYM, classifier.

13. Method (1100) for a device (100), particularly a mobile device, comprising a plurality of sensors (101, 102), the method (1100) comprising, in a testing phase:

obtaining (1101) data (111, 112) from the plurality of sensors (101, 102);

applying (1102) a plurality of trained classifiers (121, 122) on the obtained data (111, 112), wherein each trained classifier (121, 122) is associated to one of the sensors (101, 102);

estimating (1103), in each of the plurality of trained classifiers (121, 122), a probability for each of a plurality of classes of a transportation mode (110, 120, 130) of the device (100); and

determining (1104) a transportation mode (110, 120, 130) of the device (100) based on the probabilities estimated in each of the trained classifiers (121, 122).

14. Device (200), particularly server device, configured to, in a training phase:

obtain training data (211, 212) of another device (100), particularly mobile device, comprising a plurality of sensors (101, 102);

associate a classifier (221, 222) to each of the sensors (101, 102);

train each classifier (221, 222) using the obtained training data (211, 212); and provide the trained classifiers (121, 122) to the another device (100).

15. Method (1200) comprising, in a training phase:

obtaining (1201) training data (211, 212) of a device (100), particularly mobile device, comprising a plurality of sensors (101, 102);

associating (1202) a classifier (221, 222) to each of the sensors (101, 102);

training (1203) each classifier (221, 222) using the obtained training data (211, 212); and

providing (1204) the trained classifiers (121, 122) to the device (100).