WO2023197296A1

WO2023197296A1 - Hand-free method for operating a device with an operating interface

Info

Publication number: WO2023197296A1
Application number: PCT/CN2022/087057
Authority: WO
Inventors: Liang Han; Zhihong Guo
Original assignee: Orange SA
Current assignee: Orange SA
Priority date: 2022-04-15
Filing date: 2022-04-15
Publication date: 2023-10-19
Anticipated expiration: 2024-10-15

Abstract

The present invention relates to a method for operating a device (1) characterized in that it comprises the following steps of, performed by a processing unit (11) of the device: a) triggering on the device a blowing event detection mode, during which the occurrence of a blowing event is monitored, b) operating a specific action on the device when such a blowing event is detected by the device.

Description

Hand-free method for operating a device with an operating interface

TECHNICAL FIELD

The field of this invention is that of devices with an operating interface, such as mobile phones.

More particularly, the invention relates to a hand-free method for operating such a device.

BACKGROUND

With devices such as mobile phones, notifications of events such as an incoming call or reception of a SMS will raise sound, vibrations and/or light on the screen of the mobile phones. In some circumstances, such notification will bring interference to users. For instance, when a user focuses on one task on mobile phone, such as reading article, playing streaming video, playing games or any other long-time task, sometimes the mobile phone will pop up a notification (for instance because of a received message or a notification from a website) which will disturb the user.

Then, in order to remove this notification, the user must use another hand than the hand holding the phone, or alternatively move their finger for a long distance to remove the notification, which can be cumbersome especially when the phone’s screen is very large.

There is therefore a need for improving the user experience and for providing a more user-friendly way for operating such a device when the hands of the user are busy, in particular, but not limited to, for removing in a user-friendly way a notification displayed on the screen of a mobile terminal or otherwise outputted.

SUMMARY OF THE INVENTION

For these purposes, according to one aspect of the invention, a method is provided for operating a device characterized in that it comprises the following steps of, performed by a processing unit of the device:

(a) triggering on the device a blowing event detection mode, during which the occurrence of a blowing event is monitored,

(b) operating a specific action on the device when such a blowing event is detected by the device.

In a specific example, the method comprises a step of determining that a it comprises a step of determining that a notification is output through a user interface of the device, the blowing event detection mode being triggered by such a notification output.

It can also comprise a step of removing the output of the notification when such a blowing event is detected by the device.

Further, during said blowing event detection mode, the processing unit can monitor audio signals collected by the device to detect a blowing sound.

Said processing unit can in particular determine the power spectrum on audio signals and implement a recognition processing on said spectrum.

wherein during said blowing event detection mode, the processing unit monitors images captured by a camera of the device and processes said images to detect a specific behavior of the user indicative of the occurrence of a blowing event.

Also, during said blowing event detection mode, the processing unit can monitor images captured by a camera of the device and process said images to detect a specific behavior of the user indicative of the occurrence of a blowing event e.g. the processing unit can monitor images to a user’s face and, if a face is detected, to perform a face expression recognition confirming that the user is blowing on the device.

According to another aspect, it is proposed an electronic device comprising a user interface and a processing unit configured to implement the following steps:

a) triggering on the device a blowing event detection mode, during which the occurrence of a blowing event is monitored,

b) operating a specific action on the device when such a blowing event is detected by the device.

The invention further provides a computer program product, comprising code instructions for executing a method according to the first aspect for operating a device; and a computer-readable medium, on which is stored a computer program product comprising code instructions for executing a method according to the first aspect for operating a device.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of this invention will be apparent in the following detailed description of an illustrative embodiment thereof, which is to be read in connection with the accompanying drawings wherein:

- figure 1 illustrates an example of architecture in which the method according to the invention is performed;

- figures 2 to 4 illustrate the steps of various possible implementations for the invention; and

- figures 5a and 5b illustrate categorization of a normal expression vs. a pouting/blowing expression through the lines of the lips.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Architecture

The present invention relates to a method for operating a device 1 as represented by figure 1.

Said device comprises a processing unit 11, i.e. a CPU (one of mode processors) , a memory 12 (for example flash memory) and a user interface which typically includes a screen 13 (possibly touch sensitive) , a microphone 14 and as well as one or more loudspeakers 15.

The device 1 may also comprise other typical units such as a battery (not represented) and a camera 16.

Said device can be a mobile communication device which comprises a communication unit for connecting (in particular wirelessly) the device 1 to a network 20 (for example WiFi, Bluetooth, and preferably a mobile network, in particular a GSM/UMTS/LTE/5G network, see below) , etc.

Device 1 is typically a smartphone as illustrated on figure 1, or a tablet computer, a laptop, etc.

Notification

As illustrated on figure 2, processing unit 11 is configured to trigger a detection mode when a given event occurs (step A) .

In the example described in what follows, the event is the output of a notification by the user interface of device 1, and more particularly here the popping up of a notification on the screen 13 of device 1.

Said processing unit 11:

- either receives this notification from the communication unit (the notification is thus a call, an SMS, an e-mail, an alert from an application etc. ) ,

- or generates itself the notification and its output (for instance, if the battery level is below a given threshold such as 10%or 20%, the notification being a “low battery” notification) .

In both cases, the processing unit 11 outputs the notification on the spot in particular by raising sound through the loudspeakers, displaying a pop-up message on the screen of the user interface 13, or generating an alert through the vibration module and/or flashing light of a camera of the device.

Once it determines that a notification is output, the processing unit 11 triggers on the device a detection mode (step B) during which the occurrence of a blowing event is monitored. Such a detection mode can thus be called a “blowing event detection mode” .

Such a triggering can be systematic and launched for any notification output on device 1.

In another embodiment, the triggering only occurs in the case of notifications of a given category, for example only for the notifications received from the communication unit.

In other embodiments also, the triggering only occurs during a time period manually configured and/or scheduled by the user.

This detection mode can be active for a given period of time once it is triggered (detection mode duration T _ADM) .

The blowing event monitored here, during the blowing event detection mode, is a blowing event from a user interacting with device 1, in other words a voluntary blowing action performed by the user on device 1, in order to trigger the operation of a specific action by device 1.

When such a blowing event is detected (step C) , the processing unit 11 removes the output of the notification (step D) by the user interface of device 1.

Otherwise, the detection mode ends at the end of the detection mode duration T _ADM (step E) without that the processing unit 11 removes the output of the notification.

The detection mode duration T _ADM can be configured manually on the device. Said detection mode duration T _ADM can also be pre-programmed. In a given embodiment, said preprogrammed duration applies by default in the absence of any other manual configuration by a user. In another embodiment, the device does not allow any manual configuration and the preprogrammed duration applies systematically.

Blowing Detection

In a given embodiment (figure 3) , the blowing event detection is implemented using a microphone 14 of device 1.

In this embodiment, when the blowing event detection mode is triggered, the processing unit 11 switches on said microphone 14 and monitors the audio signals collected by the microphone 14, in order to detect a blowing sound which could be equated to a blowing event triggering the operation of a specific action on the device.

In a first possible implementation of this embodiment, the processing unit 11 is merely configured to detect a blowing sound, without further determination about the origin or the type of this blowing sound, and equates any detected blowing sound with a blowing event from a user wishing to trigger the operation of a specific action on device 1. Such an implementation is relatively simple to implement but mostly reliable in rather calm or silent environment.

In another implementation of this embodiment, the processing unit 11 determines the power spectrum of the sound which is detected by microphone 14 and implements a recognition processing on this power spectrum.

The analysis performed can be of the type described in:

“Recognition of Blowing Sound Types for Real-time Implementation in Mobile Devices” -Carbonneau and al. 2013, https: //www. etsmtl. ca/ETS/media/ImagesETS/Labo/LIVIA/Publications/2013/Ca rbonneau_NEWCAS_2013. pdf

In more details, the incoming audio streams from microphone 14 are first segmented by processing unit 11 into uniform slices of length Ts (step F) . Typical duration for said slices is Ts = 200 ms.

The various slices are then processed as follows, starting from the first slice.

A Discrete Fourier Transform (DFT) of the slice is computed (step G) .

The spectrum thus obtained is then used to obtain a number of signal features for the slice, such as band energy ratios, spectral centroid, peak amplitude frequency, periodicity evaluator.

These features are analyzed by a Bayes classifier (step H) to determine the sound type of each slice:

- blowing sounds made by someone trying to blow on the screen 13 or the microphone 14 (mainly corresponding to the cold sounds referred to in the above-mentioned publication) , vs.

- other sounds, including ambient noise or speech.

This other implementation is advantageous in that it makes it possible to discriminate recorded sounds between blowing sounds which are intended by a user to trigger the operation of a specific action on device 1 from other type of sounds which should not trigger the operation of a specific action on device 1.

It will be noted that such a process can be performed in real time on a device such as a mobile device (≤100 ms latency) .

If no blowing sound is detected for a given audio stream slice (test I) , the analysis is repeated on the next audio slice of the stream.

This is then repeated until either a blowing sound is detected and processing unit 11 removes the notification (step D) or until the end of detection mode duration T _ADM (step J) .

A tracking algorithm can also be implemented by processing unit 11 to take into account previous decisions of the classifier and improve the recognition performances. The performances could also be improved by training the system for a specific user at the beginning of a session.

Facial expression recognition

In specific embodiments, when the blowing event detection mode has been triggered or after having detected a blowing sound during this blowing event detection mode, the front camera 16 of device 1 can be used to detect whether the user is blowing or not on device 1, in order to trigger the operation of a specific action by this device 1.

Such a blowing event detection through the front camera 16 can be performed on its own (i.e. without simultaneously monitoring the occurrence of a blowing sound by means of a microphone) in order to detect a specific behavior of the user indicative of the occurrence of a blowing event, and may start as soon as the blowing event detection mode has been triggered.

More advantageously, the blowing event detection through the front camera 16 can be performed in addition to the monitoring of the occurrence of a blowing sound by means of microphone 14, in order to detect a specific behavior of the user confirming the occurrence of a blowing event as detected by means of microphone 14. In that case, blowing event detection through the front camera 16 may start as soon as the blowing event detection mode has been triggered or after that a blowing sound has been detected first by means of microphone 14 during the blowing event detection mode, as illustrated by figure 4.

To this end, processing unit 11 can switch on the front camera 16 for a given period of time (step K) to detect whether the user is blowing or not on the images captured.

The images captured by camera 16 are processed by processing unit 11 to detect a user’s face and, if a face is detected, to perform a face expression recognition (step L) in order to detect e.g. that the user is pouting.

Face expression recognition algorithms are today classically implemented on mobile devices, for example in the context of media content categorization (see e.g. US9918128) .

With such a face expression recognition algorithm, landmarks for the user's face are extracted from the face image, e.g. characteristic lines of the mouth.

The curvature of the lines of the lips are analyzed and categorized (see e.g. differences on the lines of the lips for a non spouting face and a spouting face as illustrated on figures 5a and 5b) .

One category for which a model can be trained by classified dataset of facial expression is the pouting of the user, which is here considered as confirming that the user is intentionally blowing on device 1.

When such a pouting expression is detected (step M) , the blowing event is confirmed and the mobile terminal removes the notification (step D) .

Device and computer program

Device 1 and its processing unit 11 are adapted for carrying out the method for operating as previously described.

In particular, device 1 is adapted to run a computer program product, comprising code instructions for executing (in particular with a processing unit 11 of the device 1) the previously described method as well as a computer-readable medium (in particular a memory 12 of the device 1) , on which is stored a computer program product comprising code instructions for executing said method. In particular, this computer program product may correspond to a dedicated software application to be installed on said device 1.

Further embodiments

Other ways to confirm a blowing event can be further contemplated.

In particular, other behaviors of the user can be used to confirm the blowing event.

For example, the user can shake his/her head to confirm that this notification is useless and its output can be removed.

When such a head movement is detected, processing unit 11 removes the notification from the user interface of device 1.

Further, the method described can be used for many scenarii other than that of the removal of pop-up notification on a communicating device.

A specific example is that of the browsing a long web page : when a web page is displayed on the screen 13 of device 1, and in particular when it is detected that the web page is so long that only a part of this web page can be displayed on this screen 13, the blow event detection mode can be triggered.

In that situation, the user can blow on device 1 in order to scroll through the web page, and thus release their hand.

Claims

A method for operating a device (1) characterized in that it comprises the following steps of, performed by a processing unit (11) of the device:

a) triggering on the device a blowing event detection mode, during which the occurrence of a blowing event is monitored,

b) operating a specific action on the device when such a blowing event is detected by the device.
A method according to claim 1, wherein it comprises a step of determining that a notification is output through a user interface of the device (1) , the blowing event detection mode being triggered by such a notification output.
A method according to claim 2, wherein it comprises a step of removing the output of the notification when such a blowing event is detected by the device.
A method according to claim 1, wherein during said blowing event detection mode, the processing unit monitors audio signals collected by the device to detect a blowing sound.
A method according to claim 4, wherein in order to detect said blowing sound, said processing unit determines the power spectrum on audio signals and implements a recognition processing on said spectrum.
A method according to any one of claims 1 to 5, wherein during said blowing event detection mode, the processing unit monitors images captured by a camera of the device and processes said images to detect a specific behavior of the user indicative of the occurrence of a blowing event.
A method according to claim 6, wherein the processing unit monitors images to a user’s face and, if a face is detected, to perform a face expression recognition confirming that the user is blowing on the device.
An electronic device (1) comprising a processing unit (11) configured to implement the following steps:

(a) triggering on the device a blowing event detection mode, during which the occurrence of a blowing event is monitored,

(b) operating a specific action on the device when such a blowing event is detected by the device.
A computer program product, comprising code instructions for executing a method according to any one of claims 1 to 7 for operating a device (1) .
A computer-readable medium, on which is stored a computer program product comprising code instructions for executing a method according to any one of claims 1 to 7 for operating a device (1) .