CN106708254B - Detector - Google Patents
Detector Download PDFInfo
- Publication number
- CN106708254B CN106708254B CN201610675089.0A CN201610675089A CN106708254B CN 106708254 B CN106708254 B CN 106708254B CN 201610675089 A CN201610675089 A CN 201610675089A CN 106708254 B CN106708254 B CN 106708254B
- Authority
- CN
- China
- Prior art keywords
- gesture
- ultrasonic
- signal
- detector
- ultrasonic signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
- G01S15/02—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems using reflection of acoustic waves
- G01S15/50—Systems of measurement, based on relative movement of the target
- G01S15/58—Velocity or trajectory determination systems; Sense-of-movement determination systems
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
- G01S15/88—Sonar systems specially adapted for specific applications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/52—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S15/00
- G01S7/523—Details of pulse systems
- G01S7/526—Receivers
- G01S7/527—Extracting wanted echo signals
- G01S7/5273—Extracting wanted echo signals using digital techniques
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/52—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S15/00
- G01S7/539—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S15/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/26—Power supply means, e.g. regulation thereof
- G06F1/32—Means for saving power
- G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
- G06F1/3234—Power saving characterised by the action undertaken
- G06F1/3287—Power saving characterised by the action undertaken by switching off individual functional units in the computer system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/97—Determining parameters from multiple pictures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10132—Ultrasound image
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Acoustics & Sound (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Multimedia (AREA)
- Psychiatry (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- User Interface Of Digital Computer (AREA)
- Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
- Telephone Function (AREA)
Abstract
A user input detector for a mobile device is described, comprising an ultrasonic demodulator having an input for receiving an ultrasonic signal reflected from a user and an output for outputting a demodulated ultrasonic signal; a gesture processor comprising a temporal frequency processing module configured to generate a time-varying ultrasound image spectrum from the demodulated ultrasound signal; an image feature extraction module configured to extract micro-Doppler features from the time-varying ultrasound image spectrum; a feature selection module configured to select and compress the extracted micro-Doppler features; and a gesture detection module configured to compare the selected micro-doppler feature to a set of known features and output a detected gesture based on the comparison.
Description
Technical Field
The present invention relates to a detector for user input to a mobile device.
Background
Smart mobile devices, such as mobile phones, have various user input methods for controlling the device, in particular, clicking a screen or a button or voice recognition. Air gestures have long been considered a very attractive alternative because of their unique intuitiveness, richness, and convenience. However, in addition to the fact that image sensors are typically not low power components, airborne gesture recognition techniques, such as camera-based recognition of complex gestures, also require processing of successive video frames. Camera-based solutions may require good lighting to function properly and may be sensitive to light interference.
Furthermore, other systems such as optical sensors, near field sensors, and capacitive sensors may not be able to recognize complex gestures, whether predetermined or custom.
Disclosure of Invention
Various aspects are defined in the appended claims. In a first aspect, a detector for user input to a mobile device is defined, the detector comprising: an ultrasonic demodulator having an input for receiving an ultrasonic signal reflected from a user and an output for outputting a demodulated ultrasonic signal; a gesture processor, the gesture processor comprising: a time-frequency processing module configured to generate a time-varying ultrasound image spectrum from the demodulated ultrasound signal; an image feature extraction module configured to extract micro-Doppler features from the time-varying ultrasound image spectrum; a feature selection module configured to select and compress the extracted micro-Doppler features; and a gesture detection module configured to compare the selected micro-doppler signature to a known set of signatures and output a detected gesture based on the comparison.
The detector allows complex air gestures to be recognized using ultrasound and used to control a mobile device such as a mobile phone or smart phone. Demodulation of the input signal allows extraction of gesture information contained or carried in a narrowband signal of a few kilohertz without the need to sample the full bandwidth of the signal when converting the signal between the analog and digital domains. By demodulating this signal, power consumption for gesture detection may be reduced and the mid-air gesture is made an "always on" feature when gesture detection is included in a smart device, such as a smartphone.
Extracting micro-doppler features may allow for the detection of compound motion with ultrasound rather than merely detecting, for example, the direction and speed of hand motion. Since the sub-parts of the human hand/arm have different motion characteristics when making complex gestures, different components can be used to generate user-specific signatures and recognize more complex gestures.
A number of image features from the reflected signal may be extracted for additional processing. Thus, much more information associated with gestures may be obtained, including but not limited to: sequencing of hand movements; timing of ordered hand movements; maximum speed and relative speed of movement; acceleration of hand movement; duration of hand movement; and a repeating pattern of hand movements.
In an embodiment of the detector, the ultrasonic demodulator may comprise a switch arranged between the ultrasonic signal input and the demodulated ultrasonic signal output, and wherein the ultrasonic demodulator is operable to switchably couple the ultrasonic signal to the demodulated ultrasonic signal output.
The switch may operate at a frequency similar to or the same as the transmitted ultrasonic signal.
In an embodiment of the detector, the ultrasonic demodulator may comprise a low pass filter arranged between the switch and the demodulated ultrasonic signal output.
In an embodiment, the detector may comprise a down-sampler coupled to the demodulated ultrasonic signal output.
In an embodiment, the detector may comprise a circular buffer arranged between the output of the ultrasonic demodulator and the gesture processor.
In an embodiment, the detector according to any of the preceding claims may comprise a Fast Fourier Transform (FFT) module arranged between the down-sampler and the gesture processor. The FFT module may be implemented in low bit width circuitry, e.g., between 4 and 8 bits, which results in a simpler module with lower power consumption.
In an embodiment, the detector may additionally comprise an activation controller arranged between the ultrasonic demodulator and the gesture detection module, wherein the activation controller is operable to determine a candidate gesture by comparing the demodulated ultrasonic signal to a predetermined threshold, and to enable the gesture processor in response to the candidate gesture being detected.
By only activating the gesture processor when a possible candidate gesture has been detected, activating the controller may reduce the power consumption of the detector. This may allow the gesture detector to operate in an "always on" mode.
Embodiments of the detector may be included in a mobile device and further configured to execute a user command in response to a detected gesture. Example user commands may include, but are not limited to, commands to mute and/or unmute, lock and/or unlock the mobile device, or change the volume of audio playback. A mobile phone including a detector may be capable of responding to user gestures up to 1 meter or more from the mobile phone. A mobile phone comprising a detector may be controllable, for example, when the mobile phone is in a pocket of a user of the phone.
In an embodiment, a mobile device may include a speaker coupled to an ultrasonic signal generator and a microphone coupled to a detector, wherein the mobile device is configured to emit an ultrasonic signal and detect a reflected ultrasonic signal.
When incorporated into a mobile device, existing speakers and microphones may be used in combination with detectors to implement a system for gesture detection that may be used to control the mobile device. The system may operate in a standby mode of operation and in an "always on" mode when other user input devices or user interfaces of the smartphone, such as a touch screen, are disabled. In embodiments, a speaker or other acoustic transducer may be used both to emit ultrasonic signals and to detect any reflected response. The transmitted ultrasonic signal may be a continuous single frequency tone or a single frequency pulse with a silent interval in between. Embodiments may detect complex air gestures with a single speaker-microphone pair.
Embodiments of mobile devices may include mobile or smart phones, tablet computers, PDAs, portable audio players, and wearable devices.
In a second aspect, a method of controlling a mobile device is described, the method comprising: transmitting an ultrasonic signal; demodulating a reflected ultrasonic signal from a user; generating a time-varying ultrasound image spectrum from the demodulated ultrasound signal; extracting micro Doppler characteristics from the time-varying ultrasonic image spectrum; selecting and compressing the extracted micro-doppler features; comparing the selected micro-doppler signature to a set of known signatures; and outputting the detected gesture based on the comparison.
In an embodiment, demodulating the reflected ultrasonic signal includes switching the reflected ultrasonic signal.
In an embodiment, demodulating the reflected ultrasound signal further comprises low pass filtering the reflected ultrasound signal.
In an embodiment, the step of generating a time-varying ultrasound image spectrum occurs only when a candidate gesture has been detected.
In an embodiment, the reflected ultrasound signal may be down-sampled.
Drawings
In the drawings and description, like reference numerals refer to like features. Embodiments of the invention will now be described in detail, by way of example only, with reference to the accompanying drawings, in which:
fig. 1 shows a diagram of (a) a transmitter and a receiver for receiving reflected ultrasonic signals from a user, and (b) a gesture detector for user input, according to an embodiment.
Fig. 2 shows a detector for user input according to an embodiment.
Fig. 3 shows a detector for user input according to an embodiment.
FIG. 4 shows example detector responses to different user inputs.
Fig. 5 shows a mobile device according to an embodiment.
Fig. 6 illustrates a method of operating a mobile device according to an embodiment.
Detailed Description
Figure 1A illustrates the doppler shift resulting from reflecting the transmitted signal from the first transducer 112 back to the target 100 with the receiver of the second transducer 114. For a separate transducer, the Doppler equation is
Wherein c is the speed of sound, fdIs Doppler shift, fsIs the frequency of the transmitted ultrasonic frequency, v is the velocity of the target, cos θtIs the cosine of the angle of the transmitted signal relative to the target, cos θrWhich is the cosine of the angle of the received signal relative to the target, as shown in fig. 1B. For collocated transducers, the angle of incidence of the transmitted and reflected light is approximately equal, i.e., θτ=θρθ, and equation 1 above can be simplified as:
with the transducers juxtaposed and incident at substantially zero angle, the maximum Doppler velocity for a gesture may typically be 3m/s, and the speed of sound at room temperature is typically 345 m/s. The ratio of the Doppler frequency compared to the transmitted frequency (fd/fs) should typically be in the range-0.5% < fd/fs < 0.5%. In this case, the bandwidth of interest around the transmitted tone may be very small, typically less than 1 KHz. The doppler shift frequency caused by the user gesture can be processed at a lower frequency, thereby reducing power consumption.
Fig. 1B shows a gesture detector 1000. The ultrasonic signal may be emitted by a co-located ultrasonic emitter (not shown). The reflected ultrasonic signal may be received by the demodulator 110. The output of the demodulator 110 may be connected to the time frequency processor 102. An output of the temporal frequency processor 102 may be connected to an image feature extractor 104. An output of the image feature extractor 104 may be connected to a feature selector 106. An output of the feature selector 106 may be connected to a gesture detection module 108. The temporal frequency processor 102, the image feature extractor 104, the feature selector 106, and the gesture detection module 108 may be a gesture processor 120.
In operation, the demodulator 110 may demodulate the ultrasonic signal. The demodulated signal may contain low frequency information, e.g., below 1kHz, which may be generated due to user gestures and reject higher frequency signals. By demodulating the signal, the remainder of the processing can be performed at a much lower frequency, which can reduce power consumption. The demodulated ultrasonic signal may be processed by a time-frequency processor 102, which time-frequency processor 102 generates a frequency spectrum using an FFT or other correlation algorithm. By generating time-varying signal spectra and comparing the time-varying signal spectra, more complex gestures may be identified than based on the comparison of the signal spectra at a particular point in time.
The micro-doppler feature extraction module 104 may generate a large set of features from the time-varying signal spectrum. These characteristics may include, for example, peak location, peak amplitude, and shape of doppler induced frequency leakage. The shape of the doppler leakage may refer to, for example, a three-dimensional shape of the signal spectrum having a time x-axis, a frequency y-axis, and an amplitude z-axis. The features generated by the micro-doppler feature extractor 104 may be selected and compressed by single value and principal component analysis and fragmentation. The skilled person will also appreciate that other image feature selection and compression techniques may be used. The selected and compressed micro-doppler feature set can reduce the complexity of subsequent gesture detection steps while still allowing complex gestures to be detected. Typical complex gestures may include, but are not limited to: flipping the hand in various modes, waving the hand in various modes, and moving the fingers in various modes. A simple gesture may be an up or down motion. The gesture detection module 108 may determine the gesture using, for example, a machine learning model such as a Support Vector Machine (SVM), nearest neighbor model, or decision tree. The skilled person will appreciate that other machine learning models may be used. The output of the gesture detection module 108 may be a user command for the mobile device.
The gesture detector 1000 may detect complex user input gestures with relatively low average power consumption, which may be, for example, less than 1 milliwatt. The peak power consumption may be less than 5 milliwatts. When incorporated into a mobile device, such as a smartphone or wearable device, the gesture detector 1000 may be incorporated into a user interface that detects gestures from a user of the mobile device and causes user commands depending on the detected gestures.
The components of the gesture detector 1000 may be implemented in hardware, software, or a combination of hardware and software. For example, demodulator 110 may be implemented as a hardware or software switch that operates at the frequency of the transmitted tone, which may be a frequency greater than 20KHz, for example. Typically, frequencies of 24KHz, 32KHz or 40KHz may be used. The gesture detection processor 120 may be implemented as logic hardware or software running on a digital signal processor.
Fig. 2 shows a gesture detector 2000. An ultrasonic signal input, which may include gesture information, may be received by demodulator 200. The demodulator 200 may have a low power passive switch 202 that may consume several microwatts of power. Depending on the transmitted tone, the low power passive switch 202 may switch at a frequency greater than 20KHz, typically 24KHz, 32KHz, or 40 KHz. The output of the low power passive switch 202 may be connected to a low pass filter 204. The low pass filter 200 may be a low order filter, such as a first or second order filter. The bandwidth of the output of demodulator 200 may be in the range of 2KHz to 8 KHz. The output of the demodulator 200 may be connected to an analog-to-digital converter 206. The analog-to-digital converter 206 may sample the output of the demodulator at a sampling frequency greater than 4 kHz. The digital output of the analog/digital converter may be connected to a further low pass filter 208, and the low pass filter 208 may be a cascaded integrator-comb (CIC) filter. The output of the further low pass filter 208 may be connected to a down sampler 210. The down sampler 210 may down sample the signal and reduce the bandwidth from, for example, 8KHz to a frequency below 1 KHz. The output of the downsampler 210 may be connected to an activation module 212. The activation module 212 may be comprised of a Fast Fourier Transform (FFT) module 214, and the Fast Fourier Transform (FFT) module 214 may be, for example, an 8-point fast Fourier transform operating at a frequency below 0.1 KHz. Alternatively, the fast fourier transform module may implement a 16-point or 32-point FFT. The activation module 212 may be comprised of an activation control module 216 connected to an output of the fast fourier transform module 214. The output of the downsampler 210 may be connected to a circular buffer 218. An output of the circular buffer 218 may be connected to the gesture processor 120.
In operation, the gesture detector 2000 may be in a first mode of operation whereby the gesture processor 120 is disabled. The ultrasonic signal may be received by the demodulator 200 and demodulated and filtered by the passive switch 202 and the low pass filter 204. The demodulated signal may be digitized by an analog/digital converter 206 and subsequently further low-pass filtered by a further low-pass filter 208. The digitized signal is then down sampled by a down sampler 210. The down-sampled signal is converted to the frequency domain by FFT 214. Then if a candidate gesture is detected, the activation control module 216 may determine whether the input signal includes gesture information and activate or enable the gesture processor 120. A candidate gesture may be present in the received input signal if the sidebands of the signal have a strong energy level. The term "strong energy level" may be considered to refer to a signal energy or amplitude that is greater than 10% of the energy or amplitude of the transmitted tone. The sidebands of the received input signal may correspond to the demodulated signal. The circular buffer 218 may temporarily store the downsampled data while the activation module 212 determines whether a candidate gesture has been detected. If no candidate gesture is detected, the data in the buffer should be overwritten and not otherwise processed. If a candidate gesture has been detected, the data in the buffer may be processed by gesture processor 120. By only activating gesture processor 120 when a candidate gesture has been detected, power consumption of gesture detector 2000 may be reduced.
Fig. 3 shows a gesture detector 3000. A digitized ultrasonic signal input, which may include gesture information, may be received by demodulator 300. The demodulator 300 may have a relatively low power switch 302, which relatively low power switch 302 may be a software switch that consumes less than 1 milliwatt. Depending on the originally transmitted ultrasonic tone frequency, the low power switch 302 may switch at a frequency greater than 20KHz, such as 24KHz, 32KHz, or 40 KHz. The output of the low power switch 302 may be connected to a low pass filter 304. The low pass filter 304 may be a low order filter, such as a first or second order filter. The bandwidth of the output of demodulator 300 may be in the range of 2KHz to 8 KHz. The output of demodulator 300 may be connected to a down-sampler 306.
The down sampler 306 may down sample the signal and reduce the bandwidth from, for example, 8KHz to a frequency below 1 KHz. The output of the downsampler 306 may be connected to an activation block 312. The activation module 312 may be comprised of a fast fourier transform module 310, and the fast fourier transform module 310 may be an 8-point fast fourier transform operating at a frequency below 0.1 KHz. Alternatively, the fast fourier transform module may implement a 16-point or 32-point FFT. The activation module 312 may be comprised of an activation control module 314 coupled to the output of the fast fourier transform module 310. The output of the downsampler 306 may be connected to a circular buffer 308. An output of the circular buffer 308 may be connected to the gesture processor 120. Gesture detector 3000 may be implemented as a software module executable on a digital signal processor.
In operation, gesture detector 3000 may be in a first mode of operation, whereby gesture processor 120 is disabled. The digitized ultrasonic signal may be received by the demodulator 200 and demodulated and filtered by the passive switch 302 and the low pass filter 304. The demodulated signal may be downsampled by a downsampler 306. The downsampled signal may be converted to the frequency domain by FFT 310. The activation control module 314 may then determine whether the input signal includes candidate gesture information. For example, candidate gesture information may be determined by comparing the energy level or amplitude of the demodulated signal corresponding to the sidebands of the digitized ultrasonic signal to the energy level or amplitude of the transmitted tone.
If the amplitude of the demodulated signal is greater than a predetermined threshold, a candidate gesture can be identified. The predetermined threshold may be a value, for example, greater than 10% of the amplitude or energy of the transmitted ultrasonic tone. If a candidate gesture is detected, the gesture processor 120 may be activated or enabled. The circular buffer 308 may temporarily store the downsampled data while the activation module 312 determines whether a candidate gesture has been detected. If no candidate gesture is detected, the data in the buffer should be overwritten and not otherwise processed. If a candidate gesture has been detected, the data in the buffer may be processed by gesture processor 120.
Fig. 4 shows an example of a time-varying spectrum for different gestures 4000 that may be detected by the gesture detectors 1000, 2000, or 3000. In these figures, the x-axis represents time and the y-axis represents frequency. The image intensity at a particular coordinate (x, y) may represent the energy level of frequency y at time x. The time-varying spectrum 4000 can be considered a two-dimensional representation of a three-dimensional doppler leakage shape. The doppler frequency above the dashed reference line 402 may indicate a shift in the positive direction, which may be a shift in the direction toward the gesture detector 2000. The doppler frequency below the dashed reference line 402 may indicate a frequency shift in a negative direction that corresponds to a shift away from the gesture detector 2000.
FIG. 4A illustrates an example response 400 caused by a hand first approaching the gesture detector 2000 and then exiting the gesture detector 2000. As the hand or object moves toward the gesture detector 2000, the spectral response shape 400 consists primarily of positive doppler frequencies with a positive frequency peak 404. As the hand or target moves away from the gesture detector 2000, the response consists primarily of a negative doppler frequency with a negative peak frequency 406.
FIG. 4B illustrates a response 410 over time to a user performing a swipe gesture. Initially, the response shape 410 consists primarily of a positive doppler frequency with a peak 412 followed by a negative offset frequency with a negative peak 414. The response shape 410 then consists essentially of another set of positive doppler frequencies with an additional peak 416 followed by another set of negative doppler frequencies with an additional negative peak 418.
FIG. 4C shows a response shape 420 resulting from a user performing a hand-flipping gesture. The response shows similar amounts of positive and negative doppler frequencies at the same point in time with a positive peak 422 and a negative peak 424.
FIG. 4D illustrates a response 430 resulting from a user gesture in which the user's hand first leaves the gesture detector 2000 and then approaches the gesture detector 2000.
The spectral response shape 430 consists essentially of negative doppler frequencies with negative frequency peaks 434 as the hand or target moves away from the gesture detector 2000, and then the spectral response shape 430 consists essentially of positive doppler frequencies with positive peak frequencies 432 as the hand or target moves toward the gesture detector 2000.
The skilled person will appreciate that other gestures may have different spectra with characteristic shape characteristics that may be stored and compared to the spectrum of the received ultrasonic signal to determine the type of user gesture.
Fig. 5 shows a mobile phone 5000 comprising a gesture detector 2000. The microphone 500 may be connected to the gesture detector 2000. An output of the gesture detector 2000 may be connected to the controller 502. The microphone 500 may be directly connected to the controller 502. An output of the controller 502 may be connected to an ultrasonic transmitter 504. An output of the audio processor 506 may be connected to a mixer 508.
The output of the controller may be connected to an audio processor 506. The output of the ultrasonic generator 504 may be connected to a mixer 508. The output of the mixer 508 may be connected to a speaker 510. In operation, the ultrasonic generator 504 may generate an ultrasonic signal that may be emitted by the speaker 510 via the mixer 508. The ultrasonic signal may be a continuous tone or may be pulsed. The frequency of the ultrasonic signal may be 20KHz or higher.
The reflected ultrasonic signal may be detected by the microphone 500 and the microphone's response to the reflected signal may be received by the gesture detector 2000. Gesture detector 2000 may process the received ultrasonic signals to determine whether a user input gesture has been detected. If the gesture detector 2000 has detected a user gesture, the gesture detector 2000 may output data to the controller 502 indicating which gesture has been detected. The detected gesture data may be processed by the controller 502 to trigger a user command.
The controller 502 may process detected user input commands and process detected user input commands to cause various behaviors of the mobile device. Example commands may be to unlock/lock the device, increase or decrease volume, take a picture using a camera feature (not shown) of the mobile phone 5000.
The microphone 500 may also detect voice or other audio inputs and route those voice or other audio inputs directly to the controller 502. The speaker 510 may also output voice or other audio content from the controller 502 that may be processed by the audio processor 506 and mixed with any ultrasonic signals generated by the ultrasonic generator 504. The speaker 510 and microphone 500 may be used for gesture detection and in normal operation, for example, for making or receiving calls or for playing music. The gesture detection may operate in parallel with other modes of operation of the mobile phone 5000.
Fig. 6 shows a method of operating a mobile device 6000. In step 600, an ultrasonic signal may be transmitted from the mobile device. The transmitted ultrasonic signal may be reflected from an object, which may be, for example, a hand of a user of the mobile device. In step 602, the reflected ultrasonic signal may be demodulated. In step 604, the demodulator signal may be processed to generate a time-varying ultrasound image spectrum. In step 606, micro-Doppler features may be extracted from the time-varying ultrasound image spectrum.
The micro-doppler signature may also be selected and these signatures may be compressed. The features may be selected and compressed, for example, by single value and principal component analysis and fragmentation. In step 608, the selective micro-doppler signature that may be compressed may be compared to signatures from a known set of signatures.
The selected image features may correspond to features extracted from a plurality of predetermined gestures (e.g., clapping, moving a hand toward or away from the mobile device, a gliding motion, a waving motion), or some other predetermined type of gesture. From the comparison to the known feature set, a user gesture may be determined, and in step 610, the mobile device may execute a user command in response to the comparison in step 608.
The method 6000 may enable control of a mobile device by analyzing a user gesture without touching the mobile device. The method 6000 may allow controlling a mobile device, such as a mobile phone, by: in step 600, a component of the mobile device (e.g., a speaker) is used to transmit an ultrasonic signal, and in step 602, a microphone is used to receive a reflected ultrasonic signal. These components may be shared with other functions of the mobile device.
A user input detector for a mobile device is described herein, the user input detector having an ultrasonic demodulator with an input for receiving an ultrasonic signal reflected from a user and an output for outputting a demodulated ultrasonic signal; a gesture processor, comprising: a time-frequency processing module configured to generate a time-varying ultrasound image spectrum from the demodulated ultrasound signal; an image feature extraction module configured to extract micro-Doppler features from the time-varying ultrasound image spectrum; a feature selection module configured to select and compress the extracted micro-Doppler features; and a gesture detection module configured to compare the selected micro-doppler feature to a set of known features and output a detected gesture based on the comparison. The user input detector may be incorporated into a mobile phone, for example, to provide an always-on low-power control mechanism for the mobile phone by recognizing user gestures and executing control commands in response to those user gestures.
Although the appended claims are directed to particular combinations of features, it should be understood that the scope of the disclosure of the present invention also includes any novel feature or any novel combination of features disclosed herein either explicitly or implicitly or any generalisation thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the present invention.
Features which are described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination.
The applicants hereby give notice that new claims may be formulated to such features and/or combinations of such features during the prosecution of the present patent application or of any further patent applications derived therefrom.
For the sake of completeness it is also stated that the term "comprising" does not exclude other elements or steps, the terms "a" and "an" do not exclude a plurality, a single processor or other unit may fulfil the functions of several means recited in the claims and reference signs in the claims shall not be construed as limiting the scope of the claims.
Claims (13)
1. A detector for user input to a mobile device, the detector comprising:
an ultrasonic demodulator having an input for receiving an ultrasonic signal reflected from a user and an output for outputting a demodulated ultrasonic signal;
a gesture processor, comprising:
a temporal frequency processing module configured to generate a time-varying ultrasound image spectrum from the demodulated ultrasound signal;
an image feature extraction module configured to extract micro-Doppler features from the time-varying ultrasound image spectrum;
a feature selection module configured to select and compress the extracted micro-Doppler features;
a gesture detection module configured to compare the selected micro-Doppler feature to a set of known features and output a detected gesture based on the comparison; and
an activation controller disposed between the ultrasonic demodulator and the gesture detection module, wherein the activation controller is operable to determine a candidate gesture by comparing an energy level or amplitude of a demodulated signal corresponding to a sideband of a digitized ultrasonic signal to a predetermined threshold, and enable the gesture processor in response to detecting a candidate gesture.
2. The detector of claim 1, wherein the ultrasonic demodulator comprises a switch disposed between the ultrasonic signal input and the demodulated ultrasonic signal output, and wherein the ultrasonic demodulator is operable to switchably couple the ultrasonic signal to the demodulated ultrasonic signal output.
3. The detector of claim 2, wherein the ultrasonic demodulator additionally comprises a low pass filter disposed between the switch and the demodulated ultrasonic signal output.
4. The detector of claim 3, further comprising a down sampler coupled to the demodulated ultrasonic signal output.
5. A detector according to any preceding claim, additionally comprising a circular buffer arranged between the output of the ultrasonic demodulator and the gesture processor.
6. The detector of claim 4, further comprising an FFT module disposed between the downsampler and the gesture processor.
7. A mobile device comprising a detector according to any preceding claim and additionally configured to perform a user command in response to the detected gesture.
8. The mobile device of claim 7, further comprising a speaker coupled to the ultrasonic signal generator and a microphone coupled to the detector, wherein the mobile device is configured to transmit ultrasonic signals and detect reflected ultrasonic signals.
9. The mobile device of claim 7, comprising one of a mobile phone, a tablet computer, and a wearable device.
10. A method of controlling a mobile device, the method comprising:
transmitting an ultrasonic signal;
demodulating a reflected ultrasonic signal from a user;
determining a candidate gesture by comparing an energy level or amplitude of a demodulated signal corresponding to a sideband of the digitized ultrasonic signal to a predetermined threshold;
in response to detecting the candidate gesture:
generating a time-varying ultrasound image spectrum from the demodulated ultrasound signal;
extracting micro Doppler features from the time-varying ultrasonic image spectrum;
selecting and compressing the extracted micro-doppler features;
comparing the selected micro-Doppler signature to a set of known signatures; and is
Outputting the detected gesture based on the comparison.
11. The method of claim 10, wherein demodulating the reflected ultrasonic signal comprises switching the reflected ultrasonic signal.
12. The method of claim 11, wherein demodulating the reflected ultrasound signal further comprises low pass filtering the reflected ultrasound signal.
13. The method of any one of claims 10 to 12, comprising down-sampling the reflected ultrasound signal.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP15181574.3A EP3133474B1 (en) | 2015-08-19 | 2015-08-19 | Gesture detector using ultrasound |
EP15181574.3 | 2015-08-19 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106708254A CN106708254A (en) | 2017-05-24 |
CN106708254B true CN106708254B (en) | 2021-06-29 |
Family
ID=54010875
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610675089.0A Active CN106708254B (en) | 2015-08-19 | 2016-08-16 | Detector |
Country Status (3)
Country | Link |
---|---|
US (1) | US9958950B2 (en) |
EP (1) | EP3133474B1 (en) |
CN (1) | CN106708254B (en) |
Families Citing this family (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3521853B1 (en) * | 2015-04-30 | 2021-02-17 | Google LLC | Rf-based micro-motion tracking for gesture tracking and recognition |
US10488937B2 (en) * | 2015-08-27 | 2019-11-26 | Verily Life Sciences, LLC | Doppler ultrasound probe for noninvasive tracking of tendon motion |
CN105938399B (en) * | 2015-12-04 | 2019-04-12 | 深圳大学 | The text input recognition methods of smart machine based on acoustics |
US11295119B2 (en) | 2017-06-30 | 2022-04-05 | The Johns Hopkins University | Systems and method for action recognition using micro-doppler signatures and recurrent neural networks |
CN107479771B (en) * | 2017-07-21 | 2020-10-23 | 浙江天益塑业有限公司 | Control method, terminal and storage medium |
NO347923B1 (en) * | 2017-09-15 | 2024-05-13 | Elliptic Laboratories Asa | User Authentication Control |
WO2019126625A1 (en) * | 2017-12-22 | 2019-06-27 | Butterfly Network, Inc. | Methods and apparatuses for identifying gestures based on ultrasound data |
CN108196257A (en) * | 2018-03-13 | 2018-06-22 | 深圳市枫芒科技有限公司 | Ultrasonic type article detection device and detection method |
US11630518B2 (en) | 2018-03-19 | 2023-04-18 | King Abdullah University Of Science And Technology | Ultrasound based air-writing system and method |
CN108696271B (en) * | 2018-05-28 | 2022-02-25 | 珠海慧联科技有限公司 | Sliding gesture recognition method and system based on capacitive touch keys |
DE102018124385B4 (en) * | 2018-10-02 | 2023-02-16 | Infineon Technologies Ag | display control devices and methods |
CN109597312B (en) * | 2018-11-26 | 2022-03-01 | 北京小米移动软件有限公司 | Sound box control method and device |
CN110045819B (en) * | 2019-03-01 | 2021-07-09 | 华为技术有限公司 | Gesture processing method and device |
US11442550B2 (en) * | 2019-05-06 | 2022-09-13 | Samsung Electronics Co., Ltd. | Methods for gesture recognition and control |
CN110389325B (en) * | 2019-07-17 | 2022-11-04 | 中北大学 | Radar micro Doppler signal extraction method of rotor unmanned aerial vehicle |
CN110474694A (en) * | 2019-07-17 | 2019-11-19 | 西北工业大学 | A kind of underwater sound wake-up circuit |
CN110796767A (en) * | 2019-10-31 | 2020-02-14 | 的卢技术有限公司 | Method and system for realizing key-free automobile key control based on gesture recognition |
WO2021130738A1 (en) * | 2019-12-23 | 2021-07-01 | Sonicedge Ltd | Sound generation device and applications |
CN113050788A (en) * | 2019-12-26 | 2021-06-29 | 华为技术有限公司 | Sound playing control method and device |
US11804077B2 (en) * | 2021-04-01 | 2023-10-31 | KaiKuTek Inc. | Generic gesture detecting method and generic gesture detecting device |
CN113655881A (en) * | 2021-08-16 | 2021-11-16 | 苏州触达信息技术有限公司 | Shortcut activation system and method based on sound wave control |
CN114236492B (en) * | 2022-02-23 | 2022-06-07 | 南京一淳科技有限公司 | Millimeter wave radar micro gesture recognition method |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100202656A1 (en) | 2009-02-09 | 2010-08-12 | Bhiksha Raj Ramakrishnan | Ultrasonic Doppler System and Method for Gesture Recognition |
US8907929B2 (en) * | 2010-06-29 | 2014-12-09 | Qualcomm Incorporated | Touchless sensing and gesture recognition using continuous wave ultrasound signals |
GB201013117D0 (en) * | 2010-08-04 | 2010-09-22 | Elliptic Laboratories As | Control of electronic devices |
US8749485B2 (en) * | 2011-12-20 | 2014-06-10 | Microsoft Corporation | User control gesture detection |
US9389690B2 (en) * | 2012-03-01 | 2016-07-12 | Qualcomm Incorporated | Gesture detection based on information from multiple types of sensors |
US9785217B2 (en) * | 2012-09-28 | 2017-10-10 | Synaptics Incorporated | System and method for low power input object detection and interaction |
DE102012022321A1 (en) | 2012-11-15 | 2014-05-15 | GM Global Technology Operations LLC (n. d. Ges. d. Staates Delaware) | Method for operating a vehicle and vehicle |
WO2014142468A1 (en) * | 2013-03-13 | 2014-09-18 | Samsung Electronics Co., Ltd. | Method of providing copy image and ultrasound apparatus therefor |
EP3019963A4 (en) * | 2013-07-12 | 2017-02-08 | University of Iowa Research Foundation | Augmentative and alternative communication |
EP2852057B1 (en) * | 2013-09-20 | 2018-05-16 | Nxp B.V. | An audio and ultrasound signal processing circuit and an ultrasound signal processing circuit, and associated methods |
WO2015054419A1 (en) * | 2013-10-08 | 2015-04-16 | University Of Washington Through Its Center For Commercialization | Devices, systems, and methods for controlling devices using gestures |
CN103793059A (en) * | 2014-02-14 | 2014-05-14 | 浙江大学 | Gesture recovery and recognition method based on time domain Doppler effect |
US9354709B1 (en) * | 2014-06-17 | 2016-05-31 | Amazon Technologies, Inc. | Tilt gesture detection |
CN104463119B (en) * | 2014-12-05 | 2017-10-31 | 苏州触达信息技术有限公司 | Combined type gesture identification equipment and its control method based on ultrasound with vision |
-
2015
- 2015-08-19 EP EP15181574.3A patent/EP3133474B1/en active Active
-
2016
- 2016-08-01 US US15/224,860 patent/US9958950B2/en active Active
- 2016-08-16 CN CN201610675089.0A patent/CN106708254B/en active Active
Also Published As
Publication number | Publication date |
---|---|
US20170052596A1 (en) | 2017-02-23 |
EP3133474A1 (en) | 2017-02-22 |
CN106708254A (en) | 2017-05-24 |
EP3133474B1 (en) | 2019-03-27 |
US9958950B2 (en) | 2018-05-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106708254B (en) | Detector | |
EP2911149B1 (en) | Determination of an operational directive based at least in part on a spatial audio property | |
US11023755B2 (en) | Detection of liveness | |
US20210027791A1 (en) | Detection of liveness | |
CN108603931B (en) | Proximity detection | |
EP2820536B1 (en) | Gesture detection based on information from multiple types of sensors | |
JP5789270B2 (en) | User interface providing apparatus and method using surface acoustic signal, and device having user interface | |
US10353495B2 (en) | Personalized operation of a mobile device using sensor signatures | |
CN110780741B (en) | Model training method, application running method, device, medium and electronic equipment | |
US20160247520A1 (en) | Electronic apparatus, method, and program | |
US10551973B2 (en) | Method of controlling a mobile device | |
CN110519450B (en) | Ultrasonic processing method, ultrasonic processing device, electronic device, and computer-readable medium | |
CN108766457A (en) | Acoustic signal processing method, device, electronic equipment and storage medium | |
CN106095203B (en) | Sensing touches the calculating device and method that sound is inputted as user gesture | |
US9772815B1 (en) | Personalized operation of a mobile device using acoustic and non-acoustic information | |
CN109032345A (en) | Apparatus control method, device, equipment, server-side and storage medium | |
JP2018528537A (en) | System and method for double knuckle touchscreen control | |
CN114943242A (en) | Event detection method and device, electronic equipment and storage medium | |
GB2516075A (en) | Sensor input recognition | |
Cao et al. | IPand: accurate gesture input with ambient acoustic sensing on hand | |
JP2014099073A (en) | Electronic apparatus, control method of the same, and program | |
GB2553040A (en) | Sensor input recognition | |
Tung | Acoustic sensing: Mobile applications and frameworks | |
Gong | Enhancing touch interactions with passive finger acoustics | |
JP2004046391A (en) | Writing implement |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |