WO2023016622A1

WO2023016622A1 - Method and apparatus for controlling an internet of things, iot, device

Info

Publication number: WO2023016622A1
Application number: PCT/EP2021/072124
Authority: WO
Inventors: Oleg Pogorelik
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2021-08-09
Filing date: 2021-08-09
Publication date: 2023-02-16

Abstract

A method of controlling an Internet of Things, IoT, device includes sensing a user gesture by an Inertial Measurement Unit, IMU (102, 602, 702). The user gesture has a control data encoded with a sequence of generic gesture units. The method includes performing a tremor-based user authentication by a tremor authenticator (104, 204, 604, 704) during the sensing of the user gesture. The method also includes determining the sequence of generic gesture units in the user gesture by a gesture decoder (106, 206, 606, 706) to decode the control data. The control data includes a command and an IoT device identification, ID. The method further includes sending the command and the IoT device ID to a control application (110, 210, 610A-N, 710) by a dispatcher (108, 208, 608, 708) in response to the tremor authenticator authenticating the user.

Description

METHOD AND APPARATUS FOR CONTROLLING AN INTERNET OF THINGS, IOT, DEVICE

TECHNICAL FIELD

The disclosure relates generally to controlling an Internet of Things, loT, device, and more particularly, the disclosure relates to a method and an apparatus for controlling the Internet of Things, loT, device.

BACKGROUND

Personal devices (e.g. smartphones) are typically used for controlling other devices and systems (e.g. loT devices/systems). For example, the smartphones may be used as a TV remote control, control of a door lock or an air conditioner, etc. The above-mentioned operations may require authentication and authorization from a user to perform. To control the operations of the other devices and systems (e.g. to unlock a device, to activate appropriate fimction/application on the device), the user may need to select a command and an object using the smartphone to control the other device. In case of a personal digital assistant, the user may have to use a key phrase and then have to say a command to control the other device. Typically, described sequences of actions are designed to ensure a proper security level, where the smartphone unlock or key phrase may serve a key role in user authentication and the rest of actions are applied for selecting an operation and a controllable object. In order to improve usability, existing operational systems support a predefined set of operations that can be performed without user authentication. However, the predefined set of operations are limited by access (e.g. for making a photo, for calling an emergency service, etc.) and functionality (e.g. one command may be coupled with a finger scan, a face unlock, etc.).

Known gesture recognition techniques are employed by the existing systems and applications to replace user interface (UI) driven operations. The known gesture recognition techniques are usually an application/a product specific and are applied for command emulation in shared systems or in systems where access control is enforced by other complementary systems (e.g. a gaming system). Further, this gesture-based authentication is inaccurate and weak.

Secure device operation that is performed using the smartphones/mobile devices typically start with an annoying controller unlock, a key phrase, a finger scan, etc. In known devices or products, several factors are combined to maintain a high level of authentication security and accuracy. Existing “gesture only” operations are limited by “no/low” security profile and functionality (i.e. hard mapped operation). Using gestures for authentication introduces high- security risks such as replay and impersonation. Further, the selection of the controllable object(s) is determined based on additional flows such as navigating through menus, audible intents, etc., which are complicated for non-technical users, error-prone and time consuming.

Therefore, there arises a need to address the aforementioned technical problem/drawbacks in controlling the other devices (e.g. an loT device) and systems.

SUMMARY

It is an object of the disclosure to provide a method of controlling an Internet of Things, loT, device, and an apparatus for controlling the Internet of Things, loT, device while avoiding one or more disadvantages of prior art approaches.

This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

The disclosure provides a method of controlling an Internet of Things, loT, device, and an apparatus for controlling the Internet of Things, loT, device.

According to a first aspect, there is provided a method of controlling an Internet of Things, loT, device. The method includes sensing a user gesture by an Inertial Measurement Unit, IMU. The user gesture has a control data encoded with a sequence of generic gesture units. The method includes performing a tremor-based user authentication by a tremor authenticator during the sensing of the user gesture. The method includes determining the sequence of generic gesture units in the user gesture by a gesture decoder to decode the control data. The control data includes a command and an loT device identification, ID. The method includes sending the command and the loT device ID to a control application by a dispatcher in response to the tremor authenticator authenticating the user.

The method provides better User Experience, UX, and enables fast and intuitive operations for controlling the loT device. The method improves an operation latency to 2-3 seconds, which is considerably shorter than a latency of 2-10 seconds that is normal in some state of the art techniques. The method combines the tremor-based user authentication with gesture recognition techniques to effectively control the loT device. The method replaces a gesture based authentication by the tremor-based user authentication and performs the tremor-based user authentication during gesture execution. The method eliminates the need for additional movements or use of an additional factor for authentication, thereby saves time. The method enables strong access control, as secure as a regular unlock authentication, which is hard to spoof. The method has a better privacy protection as an operator only knows the meaning of the command and is a spoof resilient. The method is scalable because of adding commands up to hundreds easily without the need to remember each command. As the method utilizes a gyroscope and an accelerometer, the method can support any device that runs IMU hardware such as a remote TV mouse, a headphone as well as typical phones and tablets.

Optionally, the IMU is arranged in a handheld device. Optionally, the method further includes initiating the sensing of the user gesture by detecting a flow trigger event by the IMU. The flow trigger event may include an aggressive gesture with an amplitude exceeding a pre-defined threshold.

Optionally, the sensing of the user gesture includes sampling gyroscope and/or accelerometer signals of the IMU. Optionally, the performing of the tremor-based user authentication and the determining of the sequence of generic gesture units are based at least partially on the same data samples.

Optionally, the control data further includes control parameters to be applied by the loT device, and the method further includes sending the control parameters to the control application if the user is authenticated. The control data may be encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture. According to a second aspect, there is provided an apparatus for controlling an Internet of Things, loT, device. The apparatus includes an Inertial Measurement Unit, IMU, a tremor authenticator, a gesture decoder, a dispatcher, and a control application. The Inertial Measurement Unit, IMU is configured for sensing a user gesture having a control data encoded with a sequence of generic gesture units. The tremor authenticator is configured for performing a tremor-based user authentication during the sensing of the user gesture. The gesture decoder is configured for determining the sequence of generic gesture units in the user gesture to decode the control data. The control data includes a command and an loT device identification, ID. The dispatcher is configured for sending the command and the loT device ID to the control application in response to the tremor authenticator authenticating the user.

The apparatus provides better User Experience, UX, and enables fast and intuitive operations for controlling the loT device. The apparatus improves an operation latency to 2-3 seconds which is very less when compared to 2-10 seconds as in the state of the art techniques. The apparatus combines the tremor-based user authentication with gesture recognition techniques to effectively control the loT device. The apparatus replaces a gesture based authentication by the tremor-based user authentication and performs the tremor-based user authentication during gesture execution. The apparatus eliminates the need for additional movements or use of an additional factor for authentication, thereby saves time. The apparatus enables strong access control, as secure as a regular unlock authentication, which is hard to spoof. The apparatus has a better privacy protection as an operator only knows the meaning of the command and is a spoof resilient. The apparatus is scalable because of adding commands up to hundreds easily without the need to remember each command. As the apparatus utilizes a gyroscope and an accelerometer, the apparatus can support any device that runs IMU hardware such as a remote TV mouse, a headphone as well as typical phones and tablets.

Optionally, the apparatus is a handheld device. Optionally, the apparatus further includes a trigger detector. The trigger detector may be configured for initiating the sensing of the user gesture in response to the IMU detecting a flow trigger event. The flow trigger event may include an aggressive gesture with an amplitude exceeding a pre-defined threshold. The IMU may be configured for sampling gyroscope and/or accelerometer signals to sense the user gesture. The tremor authenticator and the gesture decoder may be configured for using at least partially the same data samples. Optionally, the control data further includes control parameters to be applied by the loT device, and the dispatcher is configured for sending the control parameters to the control application if the user is authenticated. The control data may be encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture.

Therefore, in contradistinction to the existing solutions, the method provides better User Experience, UX, and enables fast and intuitive operations for controlling the loT device. The method improves the operation latency to 2-3 seconds which is very less when compared to 2- 10 seconds as in the state of the art techniques. The method combines the tremor-based user authentication with gesture recognition techniques to effectively control the loT device. The method replaces a gesture based authentication by the tremor-based user authentication and performs the tremor-based user authentication during gesture execution. The method eliminates the need for additional movements or use of an additional factor for authentication, thereby saves time. The method enables strong access control, as secure as a regular unlock authentication, which is hard to spoof. The method has a better privacy protection as an operator only knows the meaning of the command and is a spoof resilient. The method is scalable because of adding commands up to hundreds easily without the need to remember each command. As the method utilizes a gyroscope and an accelerometer, the method can support any device that runs IMU hardware such as a remote TV mouse, a headphone as well as typical phones and tablets.

These and other aspects of the disclosure will be apparent from and the implementation(s) described below.

BRIEF DESCRIPTION OF DRAWINGS

Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

FIG. l is a block diagram of an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure; FIG. 2 illustrates a handheld device that includes an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure;

FIG. 3 is a graphical representation that depcits an accelerometer/gesture pattern of a flap and shake of a handheld device in accordance with an implementation of the disclosure;

FIG. 4A illustrates a process flow for controlling an Internet of Things, loT, device using an apparatus in accordance with an implementation of the disclosure;

FIG. 4B is a graphical representation that depicts a gesture encoding model of an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure;

FIGS. 5A-5B are graphical representations that depict a sequence of generic gesture units of a user gesture sensed by a gyroscope and an accelerometer of an Inertial Measurement Unit, IMU in accordance with an implementation of the disclosure;

FIG. 6 illustrates an exploded view of an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure;

FIGS. 7A-7B are interaction diagrams that illustrate a method of controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure;

FIG. 8 is a flow diagram that illustrates a method of controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure; and

FIG. 9 is an illustration of an exemplary apparatus, a handheld device or a computer system in which the various architectures and functionalities of the various previous implementations may be implemented.

DETAILED DESCRIPTION OF THE DRAWINGS

Implementations of the disclosure provide a method of controlling an Internet of Things, loT, device and an apparatus for controlling the Internet of Things, loT, device. To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

Terms such as "a first", "a second", "a third", and "a fourth" (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms "include" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

FIG. 1 is a block diagram of an apparatus 100 for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure. The apparatus 100 includes an Inertial Measurement Unit, IMU 102, a tremor authenticator 104, a gesture decoder 106, a dispatcher 108, and a control application 110. The Inertial Measurement Unit, IMU 102 is configured for sensing a user gesture having a control data encoded with a sequence of generic gesture units. The tremor authenticator 104 is configured for performing a tremor-based user authentication during the sensing of the user gesture. The gesture decoder 106 is configured for determining the sequence of generic gesture units in the user gesture to decode the control data. The control data includes a command and an loT device identification, ID. The dispatcher 108 is configured for sending the command and the loT device ID to the control application 110 in response to the tremor authenticator 104 authenticating the user.

The apparatus 100 provides better User Experience, UX, and enables fast and intuitive operations for controlling the loT device. The apparatus 100 improves an operation latency to 2-3 seconds which is very less when compared to 2-10 seconds as in the state of the art techniques. The apparatus 100 combines the tremor-based user authentication with gesture recognition techniques to effectively control the loT device. The apparatus 100 replaces a gesture based authentication by the tremor-based user authentication and performs the tremorbased user authentication during gesture execution. The apparatus 100 eliminates the need for additional movements or use of an additional factor for authentication, thereby saves time. The apparatus 100 enables strong access control, as secure as a regular unlock authentication, which is hard to spoof. The apparatus 100 has a better privacy protection as an operator only knows the meaning of the command and is a spoof resilient. The apparatus 100 is scalable because of adding commands up to hundreds easily without the need to remember each command. As the apparatus 100 utilizes a gyroscope and an accelerometer, the apparatus 100 can support any device that runs IMU hardware such as a remote TV mouse, a headphone as well as typical phones and tablets.

Optionally, the apparatus 100 is a handheld device. The IMU 102 may be configured for sampling gyroscope and/or accelerometer signals to sense the user gesture. The tremor authenticator 104 and the gesture decoder 106 may be configured for using at least partially the same data samples.

Optionally, the control data further includes control parameters to be applied by the loT device, and the dispatcher 108 is configured for sending the control parameters to the control application 110 if the user is authenticated. The control data may be encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture.

FIG. 2 illustrates a handheld device 200 that includes an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure. The apparatus includes an Inertial Measurement Unit, IMU, a tremor authenticator 204, a gesture decoder 206, a dispatcher 208, and a control application 210. Optionally, the apparatus is the handheld device 200. The handheld device 200 may be a mobile phone, a remote TV mouse, a headphone or any IMU enabled device, etc. The Inertial Measurement Unit, IMU is configured for sensing a user gesture of a user holding the handheld device 200. The user gesture has a control data encoded with a sequence of generic gesture units. Optionally, the user gesture includes a distinguishable

X, Y, Z patterns of an accelerometer and/or a gyroscope. The X, Y, Z pattems/samples of the accelerometer and/or the gyroscope may be determined using the IMU. The distinguishable X,

Y, Z patterns may be implemented for flow triggering (e.g. a wake up), user authentication, a command and an loT device identification, ID (i.e. a controllable object identification). Optionally, the apparatus further includes a trigger detector 202. The trigger detector 202 is configured for initiating the sensing of the user gesture in response to the IMU detecting a flow trigger event (e.g. an excessive magnitude). The flow trigger event includes an aggressive gesture with an amplitude exceeding a pre-defined threshold. The IMU may be configured for sampling gyroscope and/or accelerometer signals to sense the user gesture. The tremor authenticator 204 and the gesture decoder 206 may be configured for using at least partially the same data samples. The accelerometer and/or the gyroscope signals (i.e. the X, Y, Z patterns/samples of the user gesture) may be streamed concurrently to the tremor authenticator 204 and the gesture decoder 206 to enable fast and strong user authentication. Optionally, the tremor authenticator 204 is configured for performing a tremor-based user authentication during the sensing of the user gesture using a user ID of the user. The gesture decoder 206 is configured for determining the sequence of generic gesture units in the user gesture to decode the control data. The control data includes the command and the loT device identification, ID. The dispatcher 208 is configured for sending the command and the loT device ID to the control application 210 in response to the tremor authenticator 204 authenticating the user.

Optionally, the user gesture is a reproducible hand or a body movement. During the training of the apparatus, each user gesture may be associated with an operation and optionally a controllable object (e.g. an loT device). For example, a single shake of the handheld device 200 may be associated with a first air conditioner and two shakes of the handheld device 200 may be associated with a second air conditioner, etc. Optionally, a movement of a hand or a body of the user is determined using the apparatus as follows. Furthermore, the user gesture may be associated with a building block for complex command construction that will be used in many commands as a generic component.

For example, the trigger detector 202 may specify a start of an authentication and gesture analysis. The gesture decoder 206 identifies an operation related to a gesture that is used for encoding the control data to control the Internet of Things, loT, device. The tremor authenticator 204 may record an authentication tremor fragment that has control data for reliable user authentication. Optionally, the above-mentioned components of the apparatus may overlap or may not overlap completely or partially. The overlapping may save a decoding time. However, in case of aggressive gestures, the apparatus may provide high false rejects during the authentication. Hence, the optimal combination of the user gesture may be obtained during the training of the apparatus.

FIG. 3 is a graphical representation that depicts a flap and shake accelerometer/gesture pattern of a handheld device 300 in accordance with an implementation of the disclosure. The graphical representation depicts the flap and shake accelerometer/gesture pattern (i.e. an X, Y, Z pattern) of the handheld device 300 of a user who knocks a front panel of the handheld device 300 two times to unlock a door. The X, Y, Z pattern of the accelerometer includes a flow trigger event, a flow trigger pre-defined threshold, a user gesture, a hold (i.e. the user holding the handheld device 300 without a flap or a shake), a command and an loT device identification, ID pattern. The X, Y, Z pattern of the accelerometer may be determined using an Inertial Measurement Unit, IMU of the handheld device 300. The X, Y, Z pattern of the accelerometer may be embedded to support the flow trigger event (i.e. a wake up), a tremor authentication, a command and an loT device identification, ID (i.e. a controllable object identification).

The flow trigger event may be created based on a detection of an exceptional condition. For example, an aggressive bump, flaps and knocks on the handheld device 300 are useful for the flow initiation. These movements may create sharp peaks in the X, Y, Z pattern of the accelerometer that is significantly different from regular usage patterns. Optionally, the flow trigger event helps to minimize false wakeups and saves a battery of the handheld device 300. The peaks axis and a polarity of the X, Y, Z pattern may enable an operation to control an Internet of Things, loT, device and the loT device identification.

FIG. 4A illustrates a process flow for controlling an Internet of Things, loT, device using an apparatus in accordance with an implementation of the disclosure. The apparatus provides a new user experience using a single user gesture to perform a number of operations such as a user authentication, a command classification and a selection of a controllable object/device. At a step 402, a trigger detector is configured for initiating the sensing of a user gesture in response to an Inertial Measurement Unit, IMU detecting a flow trigger event. At a step 404, a tremor authenticator is configured for performing a tremor-based user authentication during the sensing of the user gesture. At a step 406, an operation of an Internet of Things, loT, device is selected by the user using the apparatus (e.g. a handheld device of the user). At a step 408, the loT device/an object is selected (e.g. a tap, a knock to select an object, a portrait for sealing lights, a panoramic for side lights, etc.) to perform the selected operation. At a step 410, the operation is applied on the loT device/the object.

FIG. 4B is a graphical representation that depicts a gesture encoding model of an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure. The graphical representation depicts the gesture encoding model of the apparatus to switch on a light using a flap of an apparatus (e.g. a handheld device). Optionally, the graphical representation depicts X, Y, Z pattern of an accelerometer and/or gyroscope that includes a flow trigger event which is higher than a flow trigger pre-defined threshold, a user gesture/an operation gesture (e.g. a flap of the apparatus/ a handheld device to switch on the light), a tremor recording fragment for a tremor-based user authentication and an object selection. The X, Y, Z pattern of the accelerometer may be sensed using an Inertial Measurement Unit, IMU of the apparatus.

Optionally, the tremor based user authentication requires 1.5 - 2.5 seconds of holding the apparatus/handheld device in hands of the user. The tremor may be captured during hands movements of the user, so that the tremor-based user authentication may be performed during the sensing of the user gesture and following the above holding period. The apparatus may employ an appropriate algorithm that evaluates hand stability and skip inappropriate for authentication fragments.

In an embodiment, Triggering Event (T) is determined based on the the X, Y, Z pattern of the accelerometer as follows:

(T = ( abs(X) > X_max) | ( abs(Y) > Y_max) | ( abs(X) > X_max)

OR T= SQRT (X^A2+Y^A2+Z^A2) > Magnitude_max)

AND Duration Excessive S _ceri .es > Time_m m_nax_x,? where abs is an absolute value, SQRT is a square root, ^A2 is a square, Magnitude_max is a magnitude threshold, Duration _{Excessive Series} is ^a total duration of a gesture, and T ime_max is a maximum time threshold.

In another embodiment, the Trigerring Event (T) is determined as follows:

(T = ( abs(X) > X_max) | ( abs(Y) > Y_max) | ( abs(X) > X_max)

OR T= SQRT (X^A2+Y^A2+Z^A2) > Magnitude_max)

AND Duration E „xcessive S _ceri .es < Time_r m_niirm_i, where T ime_min is a minimum time threshold. The user gestures essentially have infinite number of variations. For example, knocking front panel of the handheld device two times may unlock the door #2, while knocking back panel of the handheld device once may lock door #1. Flapping hand with the handheld device upward may “switch on lights”, flapping in the opposite direction, may “switch lights off’, etc.

FIGS. 5A-5B are graphical representations that depict a sequence of generic gesture units of a user gesture sensed by a gyroscope and an accelerometer of an Inertial Measurement Unit, IMU in accordance with an implementation of the disclosure. FIG. 5A depicts the sequence of generic gesture units, GGU, of the user gesture of the gyroscope. FIG. 5B depicts the sequence of generic gesture units of the user gesture of the accelerometer. The Inertial Measurement Unit, IMU of an apparatus is configured for sensing the user gesture having a control data encoded with the sequence of generic gesture units. The control data may be encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture.

Optionally, any control data/gesture command, GC, may be presented as a sequence of one or more generic gesture units, GGU. GC = Sk=o GGU_k , where GGU = f (Tr, D, G, Tm, A, Fc). Where, Tr, D, G, Tm, A and Fc are gesture parameters of each generic gesture unit. Where, Tr is a trajectory, D is a direction of the user gesture, G is a grip of the user during the user gesture, Tm is a duration of the gesture, A is an amplitude of the user gesture and Fc is a Fluctuations of the user gesture.

Optionally, each gesture parameter has optimal applicability and range of values specified as shown in the below table:

Optionally, the user gestures and a GGU dictionary may be built from building blocks such as taps, knocks, shakes, grips, etc. which are learnt separately and may reduce training time and simplify learning process. A Command dictionary may be created by apparatus (i.e. generic and intuitive gestures may be a part of User Experience, UX design) or specified by the user using appropriate guidelines and gesture enrollment utility. The GGUs and the complete gestures may be generic and preprogrammed.

FIG. 6 illustrates an exploded view of an apparatus for controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure. The apparatus includes an Inertial Measurement Unit, IMU 602, a tremor authenticator 604, a gesture decoder 606, a dispatcher 608 and control applications 610A-N. The Inertial Measurement Unit, IMU 602 configured for sensing a user gesture having a control data encoded with a sequence of generic gesture units. The tremor authenticator 604 is configured for performing a tremor-based user authentication during the sensing of the user gesture. The gesture decoder 606 configured for determining the sequence of generic gesture units in the user gesture to decode the control data. The control data includes a command and an loT device identification, ID. The dispatcher 608 configured for sending the command and the loT device ID to the control applications 610A-N in response to the tremor authenticator 604 authenticating the user.

The IMU 602 may be configured for sampling gyroscope and/or accelerometer signals (e.g. X, Y, Z pattems/samples) to sense the user gesture. The tremor authenticator 604 and the gesture decoder 606 may be configured for using at least partially the same data samples. Optionally, the apparatus further includes an IMU mediation Engine 612 that is connected to the IMU 602 for flow initiation. Optionally, the user gesture has a distinguishable X, Y, Z patterns of an accelerometer and/or a gyroscope. The X, Y, Z patterns/samples of the accelerometer and/or the gyroscope may be determined using the IMU 602 of the apparatus. The X, Y, Z patterns may be embedded for flow triggering (e.g. a wake up), a user authentication, a command and an loT device identification, ID (i.e. a controllable object identification). The X, Y, Z patterns/samples of the user gesture may be received by a sampler 616 in the IMU mediation Engine 612 from the accelerometer and/or gyroscope of the IMU 602. Optionally, the apparatus further includes a trigger detector 614. The trigger detector 614 is configured for initiating the sensing of the user gesture in response to the IMU 602 detecting a flow trigger event. The flow trigger event includes an aggressive gesture with an amplitude exceeding a pre-defined threshold.

Optionally, the gesture decoder 606 is configured for retrieving the user gesture information/control data about the command and the loT device identification, ID (i.e. a controllable object). The command associated with a single loT device may not include the other device/object information. The gesture decoder 606 may manage sampling window and timing. The gesture decoder 606 further includes a command classifier 618, a gesture commands database, GC DB, 620, and an object classifier 622. The object classifier 622 may encode the loT device/object information into the user gesture and allow to retrieve the loT device/object ID. For example, a repetition of the pattern (e.g. knock, double knock, triple knock, etc.) during a predefined time period may be translated to a corresponding number of the light to be switched on. The command classifier 618 may encode the command into the user gesture and allow to retrieve the command. The gesture commands database 620 may include templates of the user gesture and generic gesture units that are used for decoding the user gesture. Optionally, the gesture decoder 606 includes a decoder feature extractor 624 that is configured for extracting features in a sequence of the generic gesture units in the user gesture to decode the control data.

Optionally, the tremor authenticator 604 includes a classifier 626 and a tremor feature extractor 628. The tremor authenticator 604 is configured for performing the tremor-based user authentication during the sensing of the user gesture using an authentication system 630. The tremor authenticator 604 may send the tremor-based user authentication information (e.g. approve/decline of the user authentication) to the dispatcher 608 to process the command. Optionally, the tremor feature extractor 628 is configured for extracting features from the user gesture for performing the tremor-based user authentication during the sensing of the user gesture.

The dispatcher 608 is configured for sending the command and the loT device ID to a control application 610A in response to the tremor authenticator 604 authenticating the user. Optionally, the dispatcher 608 manages forwarding flow of the command. Using information about commands supported by the different control applications 610A-N, the dispatcher 608 may activate an appropriate control application 610A, and forward its command to that control application 610A. The dispatcher 608 may process only the authenticated commands and the loT device ID. Optionally, the dispatcher 608 include an application dictionary 632 that is configured for providing information about the control applications 610A-N. Optionally, the tremor-based user authentication and the gesture decoding is performed concurrently. The tremor-based user authentication may take a time of about 2 to 2.5 seconds more than the gesture command. The dispatcher 608 may suspend the command till the dispatcher 608 gets an approval from the tremor authenticator 604.

FIGS. 7A-7B are interaction diagrams that illustrate a method of controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure. At a step 718, an Inertial Measurement Unit, IMU 702, includes a gyroscope and/or an accelerometer that senses X, Y, Z pattems/samples of a user gesture and sends the X, Y, Z pattems/samples of the user gesture to a trigger detector 712. The Inertial Measurement Unit, IMU 702, may be configured for sampling gyroscope and/or accelerometer signals (e.g. X, Y, Z pattems/samples) to sense the user gesture. At a step 720, the trigger detector 712 initiates the sensing of the user gesture if the IMU 702 detects a flow trigger event. If the IMU 702 does not detect a flow trigger event, it goes to step 722 for detecting the flow trigger event again.

At a step 724, the tremor authenticator 704 performs a tremor-based user authentication during the sensing of the user gesture and a gesture decoder 706 determines the sequence of generic gesture units in the user gesture to decode the control data concurrently using at least partially the same data samples. The control data includes a command and an loT device identification, ID. At a step 726, the gesture decoder 706 sends the command to a dispatcher 708. Optionally, the gesture decoder 706 includes a command classifier 714 to classify the command to identify a selectable object/an loT device. At a step 728, the gesture decoder 706 includes an object classifier 716 that is activated to identify an object/an loT device when the classified command requires a controllable object/IoT device specification.

At a step 730, if the selectable object (e.g. an loT device) is not identified, the gesture decoder 706 stops decoding of the control data and go to step 732. At the step 732, the trigger detector 712 searches for a new X, Y, Z pattern/sample from the gyroscope and/or the accelerometer of the IMU 702.

At a step 734, the tremor authenticator 704 sends the tremor-based user authentication information (e.g. approve/decline of the user authentication) to the dispatcher 708 to process the command. At a step 736, the gesture decoder 706 sends the selectable object (e.g. an loT device ID) to the dispatcher 708. At a step 738, the dispatcher 708 checks whether the user authentication is approved or declined for the command. At a step 740, if the user authentication is approved, the dispatcher 708 decodes the control data. At a step 742, if the decoding of the control data is completed, the dispatcher 708 forwards the control data with gesture parameters to a control application 710 for execution.

FIG. 8 is a flow diagram that illustrates a method of controlling an Internet of Things, loT, device in accordance with an implementation of the disclosure. At a step 802, a user gesture is sensed by an Inertial Measurement Unit, IMU, and the user gesture has a control data encoded with a sequence of generic gesture units. At a step 804, a tremor-based user authentication is performed by a tremor authenticator during the sensing of the user gesture. At a step 806, the sequence of generic gesture units in the user gesture is determined by a gesture decoder to decode the control data. The control data includes a command and an loT device identification, ID. At a step 808, the command and the loT device ID is sent to a control application by a dispatcher in response to the tremor authenticator authenticating the user.

The method provides better User Experience, UX, and enables fast and intuitive operations for controlling the loT device. The method improves an operation latency to 2-3 seconds which is very less when compared to 2-10 seconds as in the state of the art techniques. The method combines the tremor-based user authentication with gesture recognition techniques to effectively control the loT device. The method replaces a gesture based authentication by the tremor-based user authentication and performs the tremor-based user authentication during gesture execution. The method eliminates the need for additional movements or use of an additional factor for authentication, thereby saves time. The method enables strong access control, as secure as a regular unlock authentication, which is hard to spoof. The method has a better privacy protection as an operator only knows the meaning of the command and is a spoof resilient. The method is scalable because of adding commands up to hundreds easily without the need to remember each command. As the method utilizes a gyroscope and an accelerometer, the method can support any device that runs IMU hardware such as a remote TV mouse, a headphone as well as typical phones and tablets.

Optionally, the IMU is arranged in a handheld device. Optionally, the method further includes initiating the sensing of the user gesture by detecting a flow trigger event by the IMU. The flow trigger event may include an aggressive gesture with an amplitude exceeding a pre-defined threshold. Optionally, the sensing of the user gesture includes sampling gyroscope and/or accelerometer signals of the IMU. The performing of the tremor-based user authentication and the determining of the sequence of generic gesture units may be based at least partially on the same data samples.

Optionally, the control data further includes control parameters to be applied by the loT device, and the method further includes sending the control parameters to the control application if the user is authenticated. The control data may be encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture.

FIG. 9 is an illustration of an exemplary apparatus, a handheld device or a computer system 900 in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 900 includes at least one processor 904 that is connected to a bus 902, wherein the computer system 900 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), Hyper Transport, or any other bus or point-to- point communication protocol (s). The computer system 900 also includes a memory 906.

Control logic (software) and data are stored in the memory 906 which may take a form of random-access memory (RAM). In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on- chip operation, and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.

The computer system 900 may also include a secondary storage 910. The secondary storage 910 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory. The removable storage drive at least one of reads from and writes to a removable storage unit in a well-known manner.

Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 906 and the secondary storage 910. Such computer programs, when executed, enable the computer system 900 to perform various functions as described in the foregoing. The memory 906, the secondary storage 910, and any other storage are possible examples of computer-readable media.

In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 904, a graphics processor coupled to a communication interface 912, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 904 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system. For example, the computer system 900 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, an embedded system.

Furthermore, the computer system 900 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 900 may be coupled to a network (for example, a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an I/O interface 908.

It should be understood that the arrangement of components illustrated in the figures described are exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures.

In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.

Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

1. A method of controlling an Internet of Things, loT, device, the method comprising: sensing a user gesture by an Inertial Measurement Unit, IMU (102, 602, 702), the user gesture having a control data encoded with a sequence of generic gesture units, performing a tremor-based user authentication by a tremor authenticator (104, 204, 604, 704) during the sensing of the user gesture, determining the sequence of generic gesture units in the user gesture by a gesture decoder (106, 206, 606, 706) to decode the control data, wherein the control data comprises a command and an loT device identification, ID, and sending the command and the loT device ID to a control application (110, 210, 610A- N, 710) by a dispatcher (108, 208, 608, 708) in response to the tremor authenticator (104, 204, 604, 704) authenticating the user.

2. The method of claim 1, wherein the IMU (102, 602, 702) is arranged in a handheld device (200, 300).

3. The method of claim 1 or 2, further comprising: initiating the sensing of the user gesture by detecting a flow trigger event by the IMU (102, 602, 702), wherein the flow trigger event comprises an aggressive gesture with an amplitude exceeding a pre-defined threshold.

4. The method of claim 1, wherein the sensing of the user gesture comprises sampling gyroscope and/or accelerometer signals of the IMU (102, 602, 702), wherein the performing of the tremor-based user authentication and the determining of the sequence of generic gesture units are based at least partially on the same data samples.

5. The method of claim 1, wherein the control data further comprises control parameters to be applied by the loT device, and the method further comprises sending the control parameters to the control application (110, 210, 610A-N, 710) if the user is authenticated.

6. The method of any of claims 1 to 5, wherein the control data is encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture.

7. An apparatus (100) for controlling an Internet of Things, loT, device, the apparatus (100) comprising: an Inertial Measurement Unit, IMU (102, 602, 702), configured for sensing a user gesture having a control data encoded with a sequence of generic gesture units, a tremor authenticator (104, 204, 604, 704) configured for performing a tremor-based user authentication during the sensing of the user gesture, a gesture decoder (106, 206, 606, 706) configured for determining the sequence of generic gesture units in the user gesture to decode the control data, wherein the control data comprises a command and an loT device identification, ID, and a dispatcher (108, 208, 608, 708) configured for sending the command and the loT device ID to a control application (110, 210, 610A-N, 710) in response to the tremor authenticator (104, 204, 604, 704) authenticating the user.

8. The apparatus (100) of claim 7, wherein the apparatus (100) is a handheld device (200, 300).

9. The apparatus (100) of claim 7 or 8, further comprising a trigger detector (202, 614, 712) configured for initiating the sensing of the user gesture in response to the IMU (102, 602, 702) detecting a flow trigger event, wherein the flow trigger event comprises an aggressive gesture with an amplitude exceeding a pre-defined threshold.

10. The apparatus (100) of claim 7, wherein the IMU (102, 602, 702) is configured for sampling gyroscope and/or accelerometer signals to sense the user gesture, wherein the tremor authenticator (104, 204, 604, 704) and the gesture decoder (106, 206, 606, 706) are configured for using at least partially the same data samples.

11. The apparatus (100) of claim 7, wherein the control data further comprises control parameters to be applied by the loT device, and the dispatcher (108, 208, 608, 708) is configured for sending the control parameters to the control application (110, 210, 610A-N, 710) if the user is authenticated.

12. The apparatus (100) of any of claims 7 to 11, wherein the control data is encoded by one or more of the following parameters of each generic gesture unit in the sequence of generic gesture units: a trajectory of the gesture, a direction of the gesture, a duration of the gesture, an amplitude of the gesture and a grip of the user during the gesture.