EP3974951A1 - Method and device for controlling apparatus, electronic apparatus, and storage medium - Google Patents
Method and device for controlling apparatus, electronic apparatus, and storage medium Download PDFInfo
- Publication number
- EP3974951A1 EP3974951A1 EP20845961.0A EP20845961A EP3974951A1 EP 3974951 A1 EP3974951 A1 EP 3974951A1 EP 20845961 A EP20845961 A EP 20845961A EP 3974951 A1 EP3974951 A1 EP 3974951A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- electronic device
- recognition result
- ultrasonic
- motion
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 90
- 238000012795 verification Methods 0.000 claims abstract description 9
- 230000004044 response Effects 0.000 claims description 32
- 230000001133 acceleration Effects 0.000 claims description 20
- 230000005484 gravity Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 description 18
- 230000006870 function Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 4
- 241000255925 Diptera Species 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0416—Control or interface arrangements specially adapted for digitisers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
- G06F2203/04108—Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
Definitions
- This disclosure relates to the technical field of terminals, and in particular to a device control method, a device control apparatus, an electronic device, and a storage medium.
- a device control method, a device control apparatus, an electronic device, and a storage medium are provided in the present disclosure to improve the above problems.
- a device control method is provided in the present disclosure.
- the method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver, and the method includes the following.
- Operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated.
- a motion parameter of the electronic device is obtained.
- the recognition result is verified based on the motion parameter, and the electronic device is controlled based on a verification result.
- a device control apparatus is provided in the present disclosure.
- the apparatus is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver.
- the apparatus includes a gesture recognition unit, a posture obtaining unit, and a control unit.
- the gesture recognition unit is configured to perform operating gesture recognition through ultrasonic waves, and generate a recognition result.
- the posture obtaining unit is configured to obtain a motion parameter of the electronic device.
- the control unit is configured to verify the recognition result based on the motion parameter, and control the electronic device based on a verification result.
- an electronic device in the present disclosure.
- the electronic device includes an ultrasonic emitter, an ultrasonic receiver, a processor, and a memory.
- One or more programs are stored in the memory and are configured to be executed by the processor to realize the method in the first aspect.
- a computer-readable storage medium configured to store program codes which, when operated by a processor, are operable with a computer to perform the method in the first aspect.
- the electronic device emits ultrasonic waves through an ultrasonic emitter (such as an earpiece, a speaker, a special ultrasonic emitter, etc.). A part of the ultrasonic waves directly reach an ultrasonic receiver (a pick-up) through air (such as path 1 in FIG. 1 ), and the other part of the ultrasonic waves are reflected by an object to reach the ultrasonic receiver through air (such as path 2 in FIG. 1 ).
- an ultrasonic emitter such as an earpiece, a speaker, a special ultrasonic emitter, etc.
- the ultrasonic receiver picks up superimposed signals of direct and reflected ultrasonic waves, and the superimposed signals are converted into audio signals by an analog to digital (A/D) converter. Audio data are processed through an algorithm to obtain a motion state of the object relative to the electronic device, so as to instruct a display screen of the electronic device to be in a screen-on state or a screen-off state.
- A/D analog to digital
- an ultrasonic signal with a fixed frequency can be emitted through a built-in ultrasonic emitter of the electronic device. It can be understood that, a part of the ultrasonic waves emitted by the ultrasonic device directly reach the ultrasonic receiver through the air, and the other part of the ultrasonic waves are reflected by the object to reach the ultrasonic receiver through the air.
- the ultrasonic receiver picks up the superimposed signals of direct and reflected ultrasonic waves, and the superimposed signals are converted into the audio signals by the A/D converter.
- the object may include a human face, a human body, etc. For example, as illustrated in FIG.
- the ultrasonic signal with the fixed frequency is emitted through a built-in earpiece, speaker or special ultrasonic emitter of the electronic device.
- the part of the ultrasonic waves directly reach the pick-up through the air, and the other part of the ultrasonic waves are reflected by the object to reach the pick-up through the air.
- the pick-up picks up the superimposed signals of direct and reflected ultrasonic waves, and the superimposed signals are converted into the audio signals by the A/D converter.
- the electronic device can convert the obtained ultrasonic waves into the audio data through the A/D converter, transmit the audio data to an audio digital signal processor (ADSP), and process the audio data through the ADSP.
- ADSP audio digital signal processor
- the sensing data can be transmitted to a communication digital signal processor (CDSP) and be processed by the CDSP, so as to obtain posture information of the electronic device.
- CDSP communication digital signal processor
- the ADSP can process obtained audio data to obtain a motion state of the object relative to the electronic device.
- the ADSP can process the posture information of the electronic device transmitted from the CDSP, in other words, the ADSP can control display contents on a screen of the electronic device according to the obtained motion state and obtained the posture information of the electronic device. For example, while the electronic device is displaying a short video, the ADSP can control the short video to switch. While the electronic device is displaying a text content, the ADSP can control the screen to turn a page and other operations.
- the electronic device includes the ultrasonic emitter and the ultrasonic receiver.
- the ultrasonic emitter moves relative to an obstacle, in other words, the electronic device moves relative to the obstacle, such that the ultrasonic receiver also moves relative to the obstacle.
- a user's hand usually moves from an end of the electronic device with the ultrasonic emitter to the other end of the electronic device with the ultrasonic emitter.
- the electronic device itself will not move or only shake slightly.
- the electronic device will still recognize changes of ultrasonic frequencies based on ultrasonic signals received by the ultrasonic receiver, such that the electronic device will misrecognize that the user is controlling electronic device.
- a device control method, a device control apparatus, an electronic device, and a storage medium are provided in the present disclosure to improve the above problems.
- the method provided in implementations of the present disclosure can use a motion parameter of the electronic device, to verify a result of operating gesture recognition obtained by ultrasonic waves, such that accuracy of the gesture recognition through ultrasonic waves can be further improved.
- a device control method is provided in implementations of the present disclosure.
- the method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver, and the method includes the following.
- S110 operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated.
- the electronic device provided in the implementations of the present disclosure can have multiple ultrasonic emission ways.
- the electronic device can keep periodical ultrasonic signal emission after the electronic device is powered on.
- the electronic device can emit ultrasonic signals after entering a certain target scenario.
- some routine operations may be conducted on the electronic device. For example, several common applications are launched, or the electronic device is triggered into some operating modes, for example, a meeting mode or an outdoor mode, etc.
- the electronic device is provided with a function of the operating gesture recognition based on the ultrasonic waves, the user can control the electronic device which is powered on through a non-touch operating gesture.
- display of the electronic device may be controlled.
- the electronic device is controlled to turn a page to display a novel or a short video of a next page.
- the electronic device is provided with the function of the operating gesture recognition based on the ultrasonic waves, the novel and the short video displayed on the electronic device can be switched through the non-touch operating gesture.
- the electronic device detects that specific applications start to run, the electronic device can determine that the specific applications have entered a specific scene, and then the ultrasonic signals are triggered to be emitted.
- the operating gesture recognition based on the ultrasonic waves may increase a certain amount of power consumption, therefore, the user can configure some applications with a function of ultrasonic recognition operating gesture when some applications are running.
- the aforementioned specific applications i.e., applications which are allowed to enable the operating gesture recognition based on the ultrasonic waves
- the electronic device can be provided with a storage region for storing a list-based configuration file in a storage space corresponding to a system program.
- Package names of the specific applications which are allowed to enable the operating gesture recognition based on the ultrasonic waves or other contents for application identification are stored in the list-based configuration file.
- the list-based configuration file is searched for a currently launched or foreground running application. If the currently launched or foreground running application is in the list-based configuration file, the ultrasonic signals can be emitted by the ultrasonic emitter.
- a name of a currently foreground running application can be obtained by executing a getRunningTasks method of an ActivityManager.
- the electronic device can also obtain a list of applications used by the user through a UsageStatsManager, and recognize a recently used application recorded in the list as the currently foreground running application.
- the electronic device can also monitor a change of a window focus through an Android's own accessibility function, and take a package name corresponding to a focus window as the currently foreground running application.
- applications in the configuration file can be controlled by the user.
- the electronic device can set a configuration interface, such that the user can control which applications are allowed to enable the operating gesture recognition based on the ultrasonic waves.
- FIG. 4 multiple applications and a state of a controlling control corresponding to each of the multiple applications are displayed in an interface. It can be understood that, if a state of a controlling control corresponding to an application is "off as illustrated in FIG. 4 , it indicates that the application is removed form or not added to the above list, in other words, the application will not be recorded in the aforementioned configuration file. If the state of the controlling control corresponding to the application is "on” as illustrated in FIG. 4 , it indicates that the application is added to the above list. For example, the state of the controlling control corresponding to an application named "application A" is "on", which indicates that "application A" will exist in the aforementioned list.
- the electronic device can set a variable, and mark the state of the controlling control through the value of the variable. For example, if the variable value is 0, it indicates that the controlling control is in a state which represents an application is not allowed to enable the operating gesture recognition based on the ultrasonic waves, and if the variable value is 1, it indicates that the controlling control is in a state which represents an application allows to enable the operating gesture recognition based on the ultrasonic waves.
- the operating gesture recognition can be performed through various methods of ultrasonic waves in implementations of the present disclosure.
- a time difference method can be used.
- the electronic device emits scanning signals in an ultrasonic frequency band at intervals through the ultrasonic receiver, and receives reflected and direct (i.e., non-stop) ultrasonic signals through the ultrasonic receiver.
- a relative distance between the obstacle and the electronic device is determined by comparing a time difference between different received ultrasonic signals through an algorithm, and a relative velocity is also calculated according to the relative distance.
- a relative motion state between the electronic device and the obstacle can be further determined according to the relative distance and the relative velocity, such that a corresponding gesture can be recognized according to the relative motion state.
- phase difference method can be used.
- the electronic device emits ultrasonic signals continuously through the ultrasonic emitter.
- a receiving end determines a phase difference when reflected ultrasonic waves reach the ultrasonic receiver.
- the relative distance between the obstacle and the electronic device is determined according to the phase difference, the relative velocity can also be calculated by the relative distance, and the relative motion state between the electronic device and the obstacle can be further determined according to the relative distance and the relative velocity, such that the corresponding gesture can be recognized according to the relative motion state.
- a method that a difference of the Doppler effect areas is an audio feature can be used.
- Different motion states of the obstacle relative to the electronic device can cause regular changes of doppler_dif.
- doppler_dif is a greater positive value.
- doppler_dif is a smaller negative value.
- doppler_dif is a value close to 0.
- the motion state of the obstacle relative to the electronic device can be determined by setting a positive threshold and a negative threshold.
- doppler_dif is greater than threshold 1, the motion state of the obstacle relative to the electronic device is determined as a "close" state.
- the motion state of the obstacle relative to the electronic device is determined as an "away” state.
- the motion state of the obstacle relative to the electronic device is determined as a " noraml “ state.
- the operating gesture is determined according to the " close “ state and the " normal " state.
- the motion parameter of the electronic device includes at least one of: a motion direction, an acceleration, and a motion distance.
- a user's hand in the process of ultrasonic control based on the operating gesture, usually moves from an end of the electronic device with the ultrasonic emitter to the other end of the electronic device with the ultrasonic emitter. During this process, the electronic device itself will not move or only shake slightly. As an implementation, if the user holds the electronic device to move the electronic device toward the object and then move the electronic device away from the object, the electronic device will still recognize changes of ultrasonic frequencies based on ultrasonic signals received by the ultrasonic receiver, such that the electronic device will misrecognize that the user is controlling electronic device.
- the electronic device can obtain whether or how the electronic device itself moves when the electronic device recognizes the operating gesture, so as to determine whether a recognized operating gesture is accurate.
- a device control method is provided in the present disclosure.
- the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can also be obtained. Then the recognition result is verified based on the motion parameter, and the electronic device is controlled based on the recognition result.
- the motion parameter of the electronic device can be used to verify the recognition result of operating gesture obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- a device control method is provided in implementations of the present disclosure.
- the method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver.
- the method includes the following.
- the electronic device can collect the motion parameter by a gyroscope.
- the electronic device when the user performs a non-touch operation by a hand according to a specified rule, the electronic device itself will not move or only shake slightly. For example, as illustrated in FIG. 6 , when the user performs a non-touch sliding operation by a hand in an arrow direction, the electronic device will recognize an operating gesture through ultrasonic signals emitted by the ultrasonic emitter and ultrasonic signals reflected by the user's hand. During this process, the electronic device itself will not move. As illustrated in FIG. 7 and FIG. 8 , the electronic device itself will emit the ultrasonic signals, and the electronic device will move toward the user and then move away from the user, such that the electronic device will also recognize an operating gesture during this process. However, the user does not perform gesture operations during this process. In addition, during this process illustrated in FIG. 7 and FIG. 8 , the electronic device itself is in the motion state.
- the electronic device can recognize whether the user is controlling the electronic device through a gesture currently
- whether the recognition result corresponds to the motion posture indicted by the motion parameter is detected as follows.
- the recognition result is determined to correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is a target gesture, and the motion parameter indicates that the electronic device is stationary.
- the recognition result is determined to not correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in a specified direction.
- a moving direction of the target gesture corresponds to the specified direction.
- the electronic device when the user actually wants to control the electronic device through the gesture, the electronic device is usually stationary. If the electronic device recognizes that the operating gesture through ultrasonic waves, and determines that a current motion parameter of the electronic device indicates that the electronic device is stationary, the electronic device can determine that the recognition result corresponds to the motion posture indicated by the motion parameter. On the contrary, if the electronic device recognizes that the operating gesture through ultrasonic waves, and determines that the current motion parameter of the electronic device indicates that the electronic device is moving in the specified direction, the electronic device can determine that the recognition result does not correspond to the motion posture indicated by the motion parameter, such that the electronic device can determine that the current gesture is a misrecognition gesture, and the electronic device will not respond.
- a moving direction here is the same as or opposite to a gesture moving direction recognized by the electronic device.
- the gyroscope disposed in the electronic device is a six-axis gyroscope, which can sense positive accelerations in three directions.
- the electronic device can configure a direction of a certain axis of the gyroscope to be consistent with an extending direction from an ultrasonic emitter at one end of the electronic device and another ultrasonic emitter at the other end of the electronic device.
- an earpiece 99 at the top of the electronic device is configured as the ultrasonic emitter
- a speaker 98 at the bottom of the electronic device is also configured as the ultrasonic emitter.
- an extending direction from the earpiece 99 to the speaker 98 in other words, a direction indicated by an arrow in FIG. 9
- the electronic device can collect an acceleration along the Y axis to be the motion parameter of the electronic device, such that whether the electronic device moves along the Y axis can be recognized. It can be understood that, as illustrated in FIG. 9 , the aforementioned specified direction is the direction indicated by the arrow in FIG. 9 .
- recognition results and motion parameters corresponding to the recognition results can be stored in the electronic device through a data table. After a recognition result and a motion parameter are obtained, whether the recognition result corresponds to the motion parameter can be detected by looking up the data table. It can be understood that, in this storing way of the data table, each of corresponding data uniquely identifies a recognition result and a motion parameter, which are same to the recognition result and the motion parameter in the data table, as a recognition result and a motion parameter corresponding to the recognition result.
- the electronic device can be controlled based on the recognized operating gesture. For example, when the electronic device determines that the electronic device is in a scene of short video browsing, the electronic device performs the operating gesture recognition through ultrasonic waves, therefore, on condition that the electronic device recognizes that a control instruction corresponding to the recognized operating gesture is indicative of switching to a next video and the electronic device determines the recognition result to be valid, the electronic device will switch a currently displayed short video to the next video.
- the electronic device when the electronic device determines that the electronic device is in a scene of novel reading, the electronic device performs the operating gesture recognition through ultrasonic waves, therefore, on condition that the electronic device recognizes that a control instruction corresponding to the recognized operating gesture is indicative of switching to a next page and the electronic device determines the recognition result to be valid, the electronic device will switch a currently displayed page to the next page.
- the recognition result is determined to be invalid in response to detecting that the recognition result does not correspond to the motion posture indicated by the motion parameter.
- the following is an illustration of how the electronic device detects whether the recognition result corresponds to the motion posture indicated by the motion parameter, and how to control the electronic device.
- the electronic device can be configured with a separate storage region for storing the recognition result.
- the electronic device when the electronic device recognizes the operating gesture, the electronic device can generate the recognition result based on the operating gesture, and then store the recognition result in the storage region. For example, if the electronic device recognizes that the user's gesture is to slide from the bottom of the electronic device to the top of the electronic device, a corresponding recognition result which is to switch to a previous page or a previous video can be generated, and the recognition result can be stored in the storage region. Then, when the motion parameter of the electronic device is obtained, the recognition result stored in the storage region is verified based on the motion parameter. If the recognition result is determined to be valid, the electronic device can be controlled based on the recognition result, and the recognition result stored in the storage region can be deleted. If the recognition result is determined to be invalid, the recognition result can be directly deleted from the storage region.
- the user's hand may shake accidentally back and forth between two ultrasonic emitters of electronic device. For example, if a mosquito flies on the screen of the electronic device, the user may drive the mosquito away, and in the process of driving the mosquito away, the user's hand will shake back and forth on the electronic device. In this case, the electronic device will continuously recognize multiple operating gestures, but the multiple operating gestures are not an operation effect actually wanted by the user.
- the electronic device can configure a generation time for each recognition result.
- the electronic device can obtain the generation time corresponding to a historically updated recognition result by taking the generation time corresponding to a latest updated recognition result as a starting point, and then obtain the number of recognition results within a specified time period starting from the generation time corresponding to the latest updated recognition result. If the number is greater than 1, it means that the recognition result within the specified time period may be generated by user's misoperation, and the recognition result is determined to be invalid. If the number is 1, the recognition result is taken as a subsequent recognition result to be verified with the motion parameter of the electronic device.
- a no-touch sliding speed of each user from the bottom of electronic device equipped with one ultrasonic emitter to the top of electronic device equipped with the other ultrasonic emitter is different. Therefore, when a process that the user slides from the bottom of electronic device to the top of electronic device is regarded as an operating gesture, a length of time taken by each user to complete the operating gesture may be different.
- the length of time taken by each user to complete the operating gesture can be taken as the aforementioned specified time period.
- the length of time taken by the user to complete the operating gesture can be collected periodically, and can also be collected through a configuration interface manually triggered by the user.
- a device control method is provided in the present disclosure.
- the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can be obtained. Then whether the recognition result corresponds to the motion posture indicated by the motion parameter is detected.
- the recognition result can be determined to be valid in response to detecting that the recognition result corresponds to the motion posture, and the electronic device can be controlled based on the recognition result.
- the motion parameter of the electronic device can be used to verify the recognition result of operating gesture obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- a device control method is provided in implementations of the present disclosure.
- the method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver, and the method includes the following.
- S310 operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated.
- the user when the user is watching contents, the user sometimes holds the electronic device in hands.
- the electronic device when the electronic device is a smart phone, the user may hold the electronic device with one hand to watch contents.
- the electronic device may move slightly.
- the user holds the smart phone with a left hand, and uses a right hand to slide from the bottom of the smart phone to the top of the smart phone without touching the smart phone.
- the user's left hand may instinctively move toward the user, which causes the smart phone to move toward the user.
- misjudgment may be caused.
- the motion parameter in the specified direction can be further detected.
- the specified direction in this implementation is the same as or opposite to the moving direction of the operating gesture. If the motion parameter indicates that the electronic device moves in a direction other than the specified direction, the electronic device can directly determine that the recognition result is valid.
- Y-axis direction defined in the foregoing is taken as an example.
- the ultrasonic emitter is provided, when the electronic device detects that the user's operating gesture is to slide from the bottom to the top of the electronic device (i.e., the Y-axis direction) without touching the electronic device, the electronic device detects that the motion parameter indicates that the electronic device is moving in X-axis or Z-axis direction, such that the electronic device can directly determine that the recognition result is valid.
- S370 the recognition result is determined to correspond to the motion posture, in response to detecting that the moving parameter in the direction does not meet the target condition.
- the target condition includes that a moving distance is greater than a specified distance, or a maximum acceleration of a moving electronic device is greater than a specified acceleration.
- the Y-axis direction defined in the foregoing is taken as an example.
- the ultrasonic emitter is provided, so when the electronic device detects that the user's operating gesture is to slide from the bottom to the top of the electronic device (i.e., the Y-axis direction) without touching the electronic device, and obtains that the electronic device is moving in the Y-axis direction, the electronic device further detects that the moving distance in the Y-axis direction is less than the specified distance, or the maximum acceleration of the moving electronic device is less than the specified acceleration, as such, the recognition result can still be determined to be valid.
- the electronic device detects that the user's operating gesture is to slide from the bottom to the top of the electronic device (i.e., the Y-axis direction) without touching the electronic device, and obtains that the electronic device is moving in the Y-axis direction, the electronic device further detects that the moving distance in the Y-axis direction is greater than the specified distance, or the maximum acceleration of the moving electronic device is greater than the specified acceleration, as such, the recognition result can be determined to be invalid.
- the gyroscope adopted in the aforementioned implementations is just an exemplary illustration, and the electronic device can also collect the motion parameter of the electronic device through other sensors, such as an acceleration sensor, a gyroscope sensor, an infrared sensor, a gravity sensor, a magnetic sensor, a direction sensor, a linear sensor, etc.
- a light sensor, a pressure sensor, a temperature sensor, a proximity sensor, etc. can also be built in to assist in collecting other information of the electronic device.
- the motion parameter can include at least one of: data collected by the gravity sensor, data collected by the acceleration sensor, and data collected by the gyroscope.
- a device control method is provided in the disclosure.
- the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can be obtained. Then whether the recognition result corresponds to the motion posture indicated by the motion parameter is detected.
- the recognition result can be determined to be valid in response to detecting that the recognition result corresponds to the motion posture, and the electronic device can be controlled based on the recognition result.
- the motion parameter of the electronic device in the specified direction can be further recognized in response to detecting that the recognition result does not correspond to the motion posture, such that recognition accuracy can be further improved.
- the motion parameter of the electronic device can be used to verify the recognition result of operating gesture obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- a device control apparatus 400 is provided in implementations of the present disclosure.
- the apparatus 400 runs on the electronic device which includes an ultrasonic emitter and an ultrasonic receiver.
- the apparatus 400 includes a gesture recognition unit 410, a posture obtaining unit 420, and a control unit 430.
- the gesture recognition unit 410 is configured to perform operating gesture recognition through ultrasonic waves, and generate a recognition result.
- the gesture recognition unit 410 includes a signal emitting sub-unit 411, a signal receiving unit 412, and a gesture recognition unit 413.
- the signal emitting sub-unit 411 is configured to trigger the ultrasonic emitter to emit an ultrasonic signal, once the electronic device is in a target scenario.
- the signal receiving unit 412 is configured to receive through the ultrasonic receiver a returned ultrasonic signal reflected by an object.
- the gesture recognition unit 413 is configured to perform the operating gesture recognition based on the returned ultrasonic signal, and generate the recognition result.
- the posture obtaining unit 420 is configured to obtain a motion parameter of the electronic device.
- the control unit 430 is configured to verify the recognition result based on the motion parameter, and control the electronic device based on a verification result.
- the control unit 430 is configured to detect whether the recognition result corresponds to the motion posture indicated by the motion parameter; determine that the recognition result is valid in response to detecting that the recognition result corresponds to the motion posture, and control the electronic device based on the recognition result; determine that the recognition result is invalid, in response to detecting that the recognition result does not correspond to the motion posture.
- control unit 430 is configured to determine that the recognition result corresponds to the motion posture indicated by the motion parameter, in response to detecting that a recognized operating gesture indicated by the recognition result is a target gesture, and the motion parameter indicates that the electronic device is stationary.
- the control unit 430 is further configured to determine that the recognition result does not correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in a specified direction.
- a moving direction of the target gesture corresponds to the specified direction.
- control unit 430 includes a first verifying sub-unit 431 and a second verifying sub-unit 432.
- the first verifying sub-unit 431 is configured to detect whether the recognition result corresponds to the motion posture indicated by the motion parameter.
- the second verifying sub-unit 432 is configured to: detect whether the motion parameter in the specified direction meets a target condition, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in the specified direction; determine that the recognition result does not correspond to the motion posture in response to detecting that the moving parameter in the direction meets the target condition; determine that the recognition result corresponds to the motion posture, in response to detecting that the moving parameter in the direction does not meet the target condition.
- the target condition includes that a moving distance is greater than a specified distance, or a maximum acceleration of a moving electronic device is greater than a specified acceleration.
- the apparatus implementations correspond to the aforementioned method implementations, and specific principles of the apparatus implementations can be referred to the aforementioned method implementations, which will not be repeated here.
- another electronic device 200 is configured to perform the aforementioned device control method, based on the aforementioned device control method and apparatus.
- the electronic device 200 includes one or more (just one illustrated in FIG. 14 ) coupled processors 102, a memory 104, a network module 106, a wireless communication module 108, an ultrasonic emitter 110, and an ultrasonic receiver 112.
- the memory 104 is configured to store one or more programs which can execute the aforementioned implementations
- the processor 102 is configured to execute the one or more programs stored in the memory 104.
- the processor 102 may include one or more cores for data process.
- the processor 102 is configured to connect with various parts of the whole electronic device 200 through various interfaces and circuits, and the processor 102 is configured to run or execute instructions, programs, code sets, or instruction sets stored in the memory 104, and invoke data stored in the memory 104 to perform various functions of the electronic device 200 and process data.
- the processor 102 may be implemented by adopting at least one of hardware forms of digital signal processing (DSP), field-programmable gate array (FPGA), programmable logic array (PLA).
- DSP digital signal processing
- FPGA field-programmable gate array
- PDA programmable logic array
- the processor 102 may integrate one of or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a modem, etc.
- the CPU is primarily configured to process an operating system, a user interface, applications, etc.
- the GPU is configured to render and draw display contents.
- the modem is configured to process the wireless communication. It can be understood that, the modem can also be implemented through a single communication chip and without being integrated into the processor 102.
- the memory 104 may include a random access memory (RAM), and may also include a read-only memory (ROM).
- the memory 104 is configured to store instructions, programs, codes, code sets, or instruction sets.
- a device control apparatus may be stored in the memory 104.
- the device control apparatus may be the aforementioned apparatus 400.
- the memory 104 may include a program storage region and a data storage region.
- the program storage region is configured to store instructions for implementing the operation system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image displaying function, etc.), and instructions for implementing undermentioned implementations, etc.
- the data storage region is further configured to store data (such as phonebooks, audio and video data, chat record data, etc.) which are created during usage of a storage terminal 100.
- a network module 106 is configured to receive and emit ultrasonic waves to realize conversion between the ultrasonic waves and electrical signals, thereby communicating with communication networks or other devices, for example, communicating with an audio playing device.
- the network module 106 may include circuit units (such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption/decryption chip, a subscriber identity module (SIM) card, a memory, etc.) for performing these functions.
- SIM subscriber identity module
- the network module 106 can be configured to communicate with various networks such as the Internet, a corporate internet, a wireless network, or other devices through the wireless network.
- the above wireless network may include a cellular telephone network, a wireless local area network (WLAN), or a metropolitan area network (MAN).
- the network module 106 may be configured to exchange information with base stations.
- a wireless communication module 108 may be configured to receive and emit Bluetooth signals or infrared signals.
- An ultrasonic emitter 110 may include a pick-up and a speaker.
- An ultrasonic receiver 112 may include a first microphone disposed close to the pick-up and a second microphone disposed close to the speaker.
- FIG. 15 is a structural block view illustrating a computer-readable storage medium provided in implementations of the present disclosure. As illustrated in FIG. 15 , A computer-readable storage medium 1100 is configured to store program codes which are invoked by a processor to execute methods described in the aforementioned method implementations.
- the computer-readable storage medium 1100 may be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a hard disk, a ROM, or the like.
- the computer-readable storage medium 1100 may include a non-transitory computer-readable storage medium.
- the computer-readable storage medium 1100 has a storage space for storing program codes 810 which are configured to execute any of operations in the aforementioned methods. These program codes may be read from one or more computer program products or written into the one or more computer program products. For example, program codes 1110 may be compressed in an appropriate form.
- a device control method, a device control apparatus, an electronic device, and a storage medium are provided in implementations of the present disclosure.
- the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can also be obtained. Then the recognition result is verified based on the motion parameter, and the electronic device is controlled based on the recognition result.
- the motion parameter of the electronic device can be used to verify the result of operating gesture recognition obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- This application claims priority to
Chinese Patent Application No. 201910701490.0, filed on July 31, 2019 - This disclosure relates to the technical field of terminals, and in particular to a device control method, a device control apparatus, an electronic device, and a storage medium.
- With development of terminal technology, more electronic devices are equipped with ultrasonic devices and are controlled based on the ultrasonic devices. However, in related ways of controlling devices based on ultrasonic waves, control accuracy still needs to be improved.
- In view of the above problems, a device control method, a device control apparatus, an electronic device, and a storage medium are provided in the present disclosure to improve the above problems.
- In a first aspect, a device control method is provided in the present disclosure. The method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver, and the method includes the following. Operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated. A motion parameter of the electronic device is obtained. The recognition result is verified based on the motion parameter, and the electronic device is controlled based on a verification result.
- In a second aspect, a device control apparatus is provided in the present disclosure. The apparatus is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver. The apparatus includes a gesture recognition unit, a posture obtaining unit, and a control unit. The gesture recognition unit is configured to perform operating gesture recognition through ultrasonic waves, and generate a recognition result. The posture obtaining unit is configured to obtain a motion parameter of the electronic device. The control unit is configured to verify the recognition result based on the motion parameter, and control the electronic device based on a verification result.
- In a third aspect, an electronic device is provided in the present disclosure. The electronic device includes an ultrasonic emitter, an ultrasonic receiver, a processor, and a memory. One or more programs are stored in the memory and are configured to be executed by the processor to realize the method in the first aspect.
- In a fourth aspect, a computer-readable storage medium is provided in the present disclosure. The computer-readable storage medium is configured to store program codes which, when operated by a processor, are operable with a computer to perform the method in the first aspect.
- In order to describe technical solutions of implementations more clearly, the following will give a brief introduction to the accompanying drawings used for describing the implementations. Apparently, the accompanying drawings hereinafter described are some implementations of the disclosure. Based on these drawings, those of ordinary skill in the art can also obtain other drawings without creative effort.
-
FIG. 1 is a schematic view illustrating a propagation path of ultrasonic waves provided in implementations of the present disclosure. -
FIG. 2 is a schematic view illustrating emission and reception of ultrasonic waves and transmission of sensing data. -
FIG. 3 is a schematic flowchart illustrating a device control method provided in implementations of the present disclosure. -
FIG. 4 is a schematic view illustrating controls in a device control method provided in implementations of the present disclosure. -
FIG. 5 is a schematic flowchart illustrating a device control method provided in other implementations of the present disclosure. -
FIG. 6 is a schematic view illustrating an operating gesture in a device control method provided in other implementations of the present disclosure. -
FIG. 7 is schematic view illustrating an electronic device moving close to a user provided in other implementations of the present disclosure. -
FIG. 8 is schematic view illustrating an electronic device moving away from a user provided in other implementations of the present disclosure. -
FIG. 9 is schematic view illustrating an extending direction in a device control method provided in other implementations of the present disclosure. -
FIG. 10 is a schematic flowchart illustrating a device control method provided in other implementations of the present disclosure. -
FIG. 11 is a structural block view illustrating a device control apparatus provided in implementations of the present disclosure. -
FIG. 12 is a structural block view illustrating a device control apparatus provided in other implementations of the present disclosure. -
FIG. 13 is a structural block view illustrating a device control apparatus provided in other implementations of the present disclosure. -
FIG. 14 is a structural block view illustrating an electronic device for executing a device control method according to implementations of the present disclosure. -
FIG. 15 is a structural block view illustrating a storage unit for saving or carrying program codes to realize a device control method provided in implementations of the present disclosure. - In order for those skilled in the art to better understand technical solutions of implementations, the technical solutions of implementations will be described clearly and completely with reference to accompanying drawings in the implementations. Apparently, implementations hereinafter described are merely some implementations, rather than all implementations, of present disclosure. All other implementations obtained by those of ordinary skill in the art based on the implementations herein without creative efforts shall fall within the protection scope of the present disclosure.
- With popularity of a full-screen design for electronic devices, more manufacturers have been using an ultrasonic proximity detecting solution to replace a traditional infrared proximity detecting solution on an electronic device, in order to save space on the top of the electronic device. The electronic device emits ultrasonic waves through an ultrasonic emitter (such as an earpiece, a speaker, a special ultrasonic emitter, etc.). A part of the ultrasonic waves directly reach an ultrasonic receiver (a pick-up) through air (such as path 1 in
FIG. 1 ), and the other part of the ultrasonic waves are reflected by an object to reach the ultrasonic receiver through air (such aspath 2 inFIG. 1 ). The ultrasonic receiver picks up superimposed signals of direct and reflected ultrasonic waves, and the superimposed signals are converted into audio signals by an analog to digital (A/D) converter. Audio data are processed through an algorithm to obtain a motion state of the object relative to the electronic device, so as to instruct a display screen of the electronic device to be in a screen-on state or a screen-off state. - Specifically, when the electronic device is monitored to be in a call state, an ultrasonic signal with a fixed frequency can be emitted through a built-in ultrasonic emitter of the electronic device. It can be understood that, a part of the ultrasonic waves emitted by the ultrasonic device directly reach the ultrasonic receiver through the air, and the other part of the ultrasonic waves are reflected by the object to reach the ultrasonic receiver through the air. The ultrasonic receiver picks up the superimposed signals of direct and reflected ultrasonic waves, and the superimposed signals are converted into the audio signals by the A/D converter. The object may include a human face, a human body, etc. For example, as illustrated in
FIG. 2 , the ultrasonic signal with the fixed frequency is emitted through a built-in earpiece, speaker or special ultrasonic emitter of the electronic device. The part of the ultrasonic waves directly reach the pick-up through the air, and the other part of the ultrasonic waves are reflected by the object to reach the pick-up through the air. The pick-up picks up the superimposed signals of direct and reflected ultrasonic waves, and the superimposed signals are converted into the audio signals by the A/D converter. - In addition, after obtaining the ultrasonic waves, the electronic device can convert the obtained ultrasonic waves into the audio data through the A/D converter, transmit the audio data to an audio digital signal processor (ADSP), and process the audio data through the ADSP. At the same time, after a sensor obtains sensing data, the sensing data can be transmitted to a communication digital signal processor (CDSP) and be processed by the CDSP, so as to obtain posture information of the electronic device. Then the CDSP can transmit the posture information of the electronic device to the ADSP at a fixed period. The ADSP can process obtained audio data to obtain a motion state of the object relative to the electronic device. At the same time, the ADSP can process the posture information of the electronic device transmitted from the CDSP, in other words, the ADSP can control display contents on a screen of the electronic device according to the obtained motion state and obtained the posture information of the electronic device. For example, while the electronic device is displaying a short video, the ADSP can control the short video to switch. While the electronic device is displaying a text content, the ADSP can control the screen to turn a page and other operations.
- However, after research, inventors find that misrecognition occurs during operating gesture recognition through ultrasonic waves.
- It should be noted that, in this implementation, the electronic device includes the ultrasonic emitter and the ultrasonic receiver. The ultrasonic emitter moves relative to an obstacle, in other words, the electronic device moves relative to the obstacle, such that the ultrasonic receiver also moves relative to the obstacle. According to Doppler effect, wavelength of a wave radiated by the object changes due to relative motion of a wave source (the electronic device) and an observer (the obstacle), and Doppler effect formula is
- It can be understood that, in the process of ultrasonic control based on the operating gesture, a user's hand (or other hand-held objects) usually moves from an end of the electronic device with the ultrasonic emitter to the other end of the electronic device with the ultrasonic emitter. During this process, the electronic device itself will not move or only shake slightly. As an implementation, if the user holds the electronic device to move the electronic device toward the object and then move the electronic device away from the object, the electronic device will still recognize changes of ultrasonic frequencies based on ultrasonic signals received by the ultrasonic receiver, such that the electronic device will misrecognize that the user is controlling electronic device.
- Therefore, a device control method, a device control apparatus, an electronic device, and a storage medium are provided in the present disclosure to improve the above problems. The method provided in implementations of the present disclosure can use a motion parameter of the electronic device, to verify a result of operating gesture recognition obtained by ultrasonic waves, such that accuracy of the gesture recognition through ultrasonic waves can be further improved.
- The following will describe specifically the implementations of the present disclosure, with reference to the accompanying drawings.
- As illustrated in
FIG. 3 , a device control method is provided in implementations of the present disclosure. The method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver, and the method includes the following. - S110: operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated.
- It should be noted that, the electronic device provided in the implementations of the present disclosure can have multiple ultrasonic emission ways. As an implementation, the electronic device can keep periodical ultrasonic signal emission after the electronic device is powered on. As an implementation, the electronic device can emit ultrasonic signals after entering a certain target scenario.
- It can be understood that, after the electronic device is powered on, some routine operations may be conducted on the electronic device. For example, several common applications are launched, or the electronic device is triggered into some operating modes, for example, a meeting mode or an outdoor mode, etc. When the electronic device is provided with a function of the operating gesture recognition based on the ultrasonic waves, the user can control the electronic device which is powered on through a non-touch operating gesture.
- In addition, when the electronic device is running a novel reading application or playing a video, display of the electronic device may be controlled. For example, the electronic device is controlled to turn a page to display a novel or a short video of a next page. On condition that the electronic device is provided with the function of the operating gesture recognition based on the ultrasonic waves, the novel and the short video displayed on the electronic device can be switched through the non-touch operating gesture. In this case, when the electronic device detects that specific applications start to run, the electronic device can determine that the specific applications have entered a specific scene, and then the ultrasonic signals are triggered to be emitted.
- It can be understood that, the operating gesture recognition based on the ultrasonic waves may increase a certain amount of power consumption, therefore, the user can configure some applications with a function of ultrasonic recognition operating gesture when some applications are running. As an implementation, the aforementioned specific applications (i.e., applications which are allowed to enable the operating gesture recognition based on the ultrasonic waves) can be recognized through a list.
- In the implementation that the aforementioned specific applications can be recognized through the list, the electronic device can be provided with a storage region for storing a list-based configuration file in a storage space corresponding to a system program. Package names of the specific applications which are allowed to enable the operating gesture recognition based on the ultrasonic waves or other contents for application identification are stored in the list-based configuration file. When the electronic device launches an application, the list-based configuration file is searched for a currently launched or foreground running application. If the currently launched or foreground running application is in the list-based configuration file, the ultrasonic signals can be emitted by the ultrasonic emitter. As an implementation, if the electronic device uses an Android operating system, a name of a currently foreground running application can be obtained by executing a getRunningTasks method of an ActivityManager. In addition, the electronic device can also obtain a list of applications used by the user through a UsageStatsManager, and recognize a recently used application recorded in the list as the currently foreground running application. Furthermore, the electronic device can also monitor a change of a window focus through an Android's own accessibility function, and take a package name corresponding to a focus window as the currently foreground running application.
- Optionally, applications in the configuration file can be controlled by the user. Correspondingly, the electronic device can set a configuration interface, such that the user can control which applications are allowed to enable the operating gesture recognition based on the ultrasonic waves.
- For example, as illustrated in
FIG. 4 , multiple applications and a state of a controlling control corresponding to each of the multiple applications are displayed in an interface. It can be understood that, if a state of a controlling control corresponding to an application is "off as illustrated inFIG. 4 , it indicates that the application is removed form or not added to the above list, in other words, the application will not be recorded in the aforementioned configuration file. If the state of the controlling control corresponding to the application is "on" as illustrated inFIG. 4 , it indicates that the application is added to the above list. For example, the state of the controlling control corresponding to an application named "application A" is "on", which indicates that "application A" will exist in the aforementioned list. The state of the controlling control corresponding to an application named "application B" is "off', which indicates that "application B" does not exist in the aforementioned list. It can be understood that, an application in the list allows to enable the operating gesture recognition based on the ultrasonic waves. - It can be understood that, the electronic device can set a variable, and mark the state of the controlling control through the value of the variable. For example, if the variable value is 0, it indicates that the controlling control is in a state which represents an application is not allowed to enable the operating gesture recognition based on the ultrasonic waves, and if the variable value is 1, it indicates that the controlling control is in a state which represents an application allows to enable the operating gesture recognition based on the ultrasonic waves.
- Furthermore, the operating gesture recognition can be performed through various methods of ultrasonic waves in implementations of the present disclosure.
- As an implementation, a time difference method can be used.
- As an implementation, the electronic device emits scanning signals in an ultrasonic frequency band at intervals through the ultrasonic receiver, and receives reflected and direct (i.e., non-stop) ultrasonic signals through the ultrasonic receiver. A relative distance between the obstacle and the electronic device is determined by comparing a time difference between different received ultrasonic signals through an algorithm, and a relative velocity is also calculated according to the relative distance. A relative motion state between the electronic device and the obstacle can be further determined according to the relative distance and the relative velocity, such that a corresponding gesture can be recognized according to the relative motion state.
- As an implementation, a phase difference method can be used.
- As an implementation, the electronic device emits ultrasonic signals continuously through the ultrasonic emitter. By calculating a correlation index between an emitted signal and a receiver signal, a receiving end determines a phase difference when reflected ultrasonic waves reach the ultrasonic receiver. The relative distance between the obstacle and the electronic device is determined according to the phase difference, the relative velocity can also be calculated by the relative distance, and the relative motion state between the electronic device and the obstacle can be further determined according to the relative distance and the relative velocity, such that the corresponding gesture can be recognized according to the relative motion state.
- As an implementation, a method that a difference of the Doppler effect areas is an audio feature can be used.
- In this implementation, the difference of the Doppler effect areas is obtained by subtracting a spectral intensity within a frequency range and less than an ultrasonic emitting frequency from a spectral intensity within the frequency range and greater than the ultrasonic emitting frequency: doppler_dif = sum_up - sum_low.
- Different motion states of the obstacle relative to the electronic device can cause regular changes of doppler_dif. When the obstacle is approaching the electronic device at a certain velocity, doppler_dif is a greater positive value. When the obstacle is moving away from the electronic device at a certain speed, doppler_dif is a smaller negative value. When the obstacle and the electronic device are relatively stationary, doppler_dif is a value close to 0. Furthermore, the motion state of the obstacle relative to the electronic device can be determined by setting a positive threshold and a negative threshold. When doppler_dif is greater than threshold1, the motion state of the obstacle relative to the electronic device is determined as a "close" state. When doppler_dif is less than threshold2, the motion state of the obstacle relative to the electronic device is determined as an "away" state. When doppler_dif is between threshold1 and threshold2, the motion state of the obstacle relative to the electronic device is determined as a "noraml" state. As such, the operating gesture is determined according to the "close" state and the "normal" state.
- S120: a motion parameter of the electronic device is obtained.
- The motion parameter of the electronic device includes at least one of: a motion direction, an acceleration, and a motion distance.
- S130: the recognition result is verified based on the motion parameter, and the electronic device is controlled based on a verification result.
- Referring to the foregoing, as an implementation, in the process of ultrasonic control based on the operating gesture, a user's hand (or other hand-held objects) usually moves from an end of the electronic device with the ultrasonic emitter to the other end of the electronic device with the ultrasonic emitter. During this process, the electronic device itself will not move or only shake slightly. As an implementation, if the user holds the electronic device to move the electronic device toward the object and then move the electronic device away from the object, the electronic device will still recognize changes of ultrasonic frequencies based on ultrasonic signals received by the ultrasonic receiver, such that the electronic device will misrecognize that the user is controlling electronic device.
- Therefore, on condition that the electronic device has obtained the recognition result of an operating gesture based on the ultrasonic waves, combined with the motion parameters of the electronic device itself, the electronic device can obtain whether or how the electronic device itself moves when the electronic device recognizes the operating gesture, so as to determine whether a recognized operating gesture is accurate.
- A device control method is provided in the present disclosure. In the method, the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can also be obtained. Then the recognition result is verified based on the motion parameter, and the electronic device is controlled based on the recognition result. According to the method, the motion parameter of the electronic device can be used to verify the recognition result of operating gesture obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- As illustrated in
FIG. 5 , a device control method is provided in implementations of the present disclosure. The method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver. The method includes the following. - S210: the operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated.
- S220: the motion parameter of the electronic device is obtained.
- As an implementation, the electronic device can collect the motion parameter by a gyroscope.
- S230: whether the recognition result corresponds to a motion posture indicated by the motion parameter is detected.
- It can be understood that, when the user performs a non-touch operation by a hand according to a specified rule, the electronic device itself will not move or only shake slightly. For example, as illustrated in
FIG. 6 , when the user performs a non-touch sliding operation by a hand in an arrow direction, the electronic device will recognize an operating gesture through ultrasonic signals emitted by the ultrasonic emitter and ultrasonic signals reflected by the user's hand. During this process, the electronic device itself will not move. As illustrated inFIG. 7 and FIG. 8 , the electronic device itself will emit the ultrasonic signals, and the electronic device will move toward the user and then move away from the user, such that the electronic device will also recognize an operating gesture during this process. However, the user does not perform gesture operations during this process. In addition, during this process illustrated inFIG. 7 and FIG. 8 , the electronic device itself is in the motion state. - As an implementation, by establishing a corresponding relationship between the operating gesture recognized by the electronic device through ultrasonic waves and the motion parameter of the electronic device, the electronic device can recognize whether the user is controlling the electronic device through a gesture currently
- As an implementation, whether the recognition result corresponds to the motion posture indicted by the motion parameter is detected as follows.
- The recognition result is determined to correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is a target gesture, and the motion parameter indicates that the electronic device is stationary. The recognition result is determined to not correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in a specified direction. A moving direction of the target gesture corresponds to the specified direction.
- It can be understood that, when the user actually wants to control the electronic device through the gesture, the electronic device is usually stationary. If the electronic device recognizes that the operating gesture through ultrasonic waves, and determines that a current motion parameter of the electronic device indicates that the electronic device is stationary, the electronic device can determine that the recognition result corresponds to the motion posture indicated by the motion parameter. On the contrary, if the electronic device recognizes that the operating gesture through ultrasonic waves, and determines that the current motion parameter of the electronic device indicates that the electronic device is moving in the specified direction, the electronic device can determine that the recognition result does not correspond to the motion posture indicated by the motion parameter, such that the electronic device can determine that the current gesture is a misrecognition gesture, and the electronic device will not respond.
- It can be understood that, a moving direction here is the same as or opposite to a gesture moving direction recognized by the electronic device. It should be noted that, in general, the gyroscope disposed in the electronic device is a six-axis gyroscope, which can sense positive accelerations in three directions. In this case, the electronic device can configure a direction of a certain axis of the gyroscope to be consistent with an extending direction from an ultrasonic emitter at one end of the electronic device and another ultrasonic emitter at the other end of the electronic device.
- As an implementation, as illustrated in
FIG. 9 , anearpiece 99 at the top of the electronic device is configured as the ultrasonic emitter, and aspeaker 98 at the bottom of the electronic device is also configured as the ultrasonic emitter. In this case, an extending direction from theearpiece 99 to thespeaker 98, in other words, a direction indicated by an arrow inFIG. 9 , can be configured to coincide with the direction of the certain axis of the gyroscope. For example, if Y axis of the gyroscope is configured to coincide with the direction indicted by the arrow inFIG. 9 , the electronic device can collect an acceleration along the Y axis to be the motion parameter of the electronic device, such that whether the electronic device moves along the Y axis can be recognized. It can be understood that, as illustrated inFIG. 9 , the aforementioned specified direction is the direction indicated by the arrow inFIG. 9 . - As an implementation, recognition results and motion parameters corresponding to the recognition results can be stored in the electronic device through a data table. After a recognition result and a motion parameter are obtained, whether the recognition result corresponds to the motion parameter can be detected by looking up the data table. It can be understood that, in this storing way of the data table, each of corresponding data uniquely identifies a recognition result and a motion parameter, which are same to the recognition result and the motion parameter in the data table, as a recognition result and a motion parameter corresponding to the recognition result.
- S240: in response to detecting that the recognition result corresponds to the motion posture indicated by the motion parameter, the recognition result is determined to be valid, and the electronic device is controlled based on the recognition result.
- On condition that the recognition result is determined to be valid based on the foregoing, the electronic device can be controlled based on the recognized operating gesture. For example, when the electronic device determines that the electronic device is in a scene of short video browsing, the electronic device performs the operating gesture recognition through ultrasonic waves, therefore, on condition that the electronic device recognizes that a control instruction corresponding to the recognized operating gesture is indicative of switching to a next video and the electronic device determines the recognition result to be valid, the electronic device will switch a currently displayed short video to the next video. For another example, when the electronic device determines that the electronic device is in a scene of novel reading, the electronic device performs the operating gesture recognition through ultrasonic waves, therefore, on condition that the electronic device recognizes that a control instruction corresponding to the recognized operating gesture is indicative of switching to a next page and the electronic device determines the recognition result to be valid, the electronic device will switch a currently displayed page to the next page.
- S250: the recognition result is determined to be invalid in response to detecting that the recognition result does not correspond to the motion posture indicated by the motion parameter.
- It can be understood that, if the recognition result is determined to be invalid, the electronic device will not be operated.
- The following is an illustration of how the electronic device detects whether the recognition result corresponds to the motion posture indicated by the motion parameter, and how to control the electronic device.
- As an implementation, the electronic device can be configured with a separate storage region for storing the recognition result. In this implementation, when the electronic device recognizes the operating gesture, the electronic device can generate the recognition result based on the operating gesture, and then store the recognition result in the storage region. For example, if the electronic device recognizes that the user's gesture is to slide from the bottom of the electronic device to the top of the electronic device, a corresponding recognition result which is to switch to a previous page or a previous video can be generated, and the recognition result can be stored in the storage region. Then, when the motion parameter of the electronic device is obtained, the recognition result stored in the storage region is verified based on the motion parameter. If the recognition result is determined to be valid, the electronic device can be controlled based on the recognition result, and the recognition result stored in the storage region can be deleted. If the recognition result is determined to be invalid, the recognition result can be directly deleted from the storage region.
- It should be noted that, in a possible case, the user's hand may shake accidentally back and forth between two ultrasonic emitters of electronic device. For example, if a mosquito flies on the screen of the electronic device, the user may drive the mosquito away, and in the process of driving the mosquito away, the user's hand will shake back and forth on the electronic device. In this case, the electronic device will continuously recognize multiple operating gestures, but the multiple operating gestures are not an operation effect actually wanted by the user.
- As an implementation, the electronic device can configure a generation time for each recognition result. When the electronic device detects that the configuration file storing the recognition result is updated, the electronic device can obtain the generation time corresponding to a historically updated recognition result by taking the generation time corresponding to a latest updated recognition result as a starting point, and then obtain the number of recognition results within a specified time period starting from the generation time corresponding to the latest updated recognition result. If the number is greater than 1, it means that the recognition result within the specified time period may be generated by user's misoperation, and the recognition result is determined to be invalid. If the number is 1, the recognition result is taken as a subsequent recognition result to be verified with the motion parameter of the electronic device.
- It should be noted that, a no-touch sliding speed of each user from the bottom of electronic device equipped with one ultrasonic emitter to the top of electronic device equipped with the other ultrasonic emitter is different. Therefore, when a process that the user slides from the bottom of electronic device to the top of electronic device is regarded as an operating gesture, a length of time taken by each user to complete the operating gesture may be different. As an implementation, in order to better adapt to a habit of each user's no-touch operating gesture, the length of time taken by each user to complete the operating gesture can be taken as the aforementioned specified time period. Optionally, the length of time taken by the user to complete the operating gesture can be collected periodically, and can also be collected through a configuration interface manually triggered by the user.
- A device control method is provided in the present disclosure. In the method, the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can be obtained. Then whether the recognition result corresponds to the motion posture indicated by the motion parameter is detected. The recognition result can be determined to be valid in response to detecting that the recognition result corresponds to the motion posture, and the electronic device can be controlled based on the recognition result. According to the method, the motion parameter of the electronic device can be used to verify the recognition result of operating gesture obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- As illustrated in
FIG. 10 , a device control method is provided in implementations of the present disclosure. The method is applicable to an electronic device which includes an ultrasonic emitter and an ultrasonic receiver, and the method includes the following. - S310: operating gesture recognition is performed by the electronic device through ultrasonic waves, and a recognition result is generated.
- S320: a motion parameter of the electronic device is obtained.
- S330: whether the recognition result corresponds to a motion posture indicated by the motion parameter is detected.
- S340: in response to detecting that a recognized operating gesture indicated by the recognition result is a target gesture, and the motion parameter indicates that the electronic device is stationary, the recognition result is determined to correspond to the motion posture.
- S350: in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in the specified direction, whether a moving parameter in the specified direction meets a target condition is detected.
- It should be noted that, when the user is watching contents, the user sometimes holds the electronic device in hands. For example, when the electronic device is a smart phone, the user may hold the electronic device with one hand to watch contents. In this case, when the user is performing a non-touch operation, the electronic device may move slightly. For example, the user holds the smart phone with a left hand, and uses a right hand to slide from the bottom of the smart phone to the top of the smart phone without touching the smart phone. In the operation process of the right hand, the user's left hand may instinctively move toward the user, which causes the smart phone to move toward the user. In this case, if the recognition result is directly determined based on the aforementioned contents, misjudgment may be caused.
- In order to improve the above problems, when the electronic device detects that the recognized operating gesture indicated by the recognition result is the target gesture and the motion parameter indicates that the electronic device is moving in the specified direction, the motion parameter in the specified direction can be further detected.
- It should be noted that, the specified direction in this implementation is the same as or opposite to the moving direction of the operating gesture. If the motion parameter indicates that the electronic device moves in a direction other than the specified direction, the electronic device can directly determine that the recognition result is valid. Y-axis direction defined in the foregoing is taken as an example. At the top and bottom of the electronic device each, the ultrasonic emitter is provided, when the electronic device detects that the user's operating gesture is to slide from the bottom to the top of the electronic device (i.e., the Y-axis direction) without touching the electronic device, the electronic device detects that the motion parameter indicates that the electronic device is moving in X-axis or Z-axis direction, such that the electronic device can directly determine that the recognition result is valid.
- S360: the recognition result is determined not to correspond to the motion posture, in response to detecting that the moving parameter in the direction meets the target condition.
- S370: the recognition result is determined to correspond to the motion posture, in response to detecting that the moving parameter in the direction does not meet the target condition.
- The target condition includes that a moving distance is greater than a specified distance, or a maximum acceleration of a moving electronic device is greater than a specified acceleration.
- It can be understood that, when the user actually performs the operating gesture, even if the electronic device will be moved, the electronic device will not move with a large moving amplitude, so the moving distance of the electronic device or an acceleration of the moving electronic device will not be too large. Therefore, by comparing the moving distance of the electronic device with the specified distance and comparing the maximum acceleration with the specified acceleration, whether the electronic device is currently moved because of an instinctive operation of the user can be determined.
- For example, the Y-axis direction defined in the foregoing is taken as an example. At The top and bottom of the electronic device each, the ultrasonic emitter is provided, so when the electronic device detects that the user's operating gesture is to slide from the bottom to the top of the electronic device (i.e., the Y-axis direction) without touching the electronic device, and obtains that the electronic device is moving in the Y-axis direction, the electronic device further detects that the moving distance in the Y-axis direction is less than the specified distance, or the maximum acceleration of the moving electronic device is less than the specified acceleration, as such, the recognition result can still be determined to be valid. On the contrary, when the electronic device detects that the user's operating gesture is to slide from the bottom to the top of the electronic device (i.e., the Y-axis direction) without touching the electronic device, and obtains that the electronic device is moving in the Y-axis direction, the electronic device further detects that the moving distance in the Y-axis direction is greater than the specified distance, or the maximum acceleration of the moving electronic device is greater than the specified acceleration, as such, the recognition result can be determined to be invalid.
- It should be noted that, the gyroscope adopted in the aforementioned implementations is just an exemplary illustration, and the electronic device can also collect the motion parameter of the electronic device through other sensors, such as an acceleration sensor, a gyroscope sensor, an infrared sensor, a gravity sensor, a magnetic sensor, a direction sensor, a linear sensor, etc. In addition, a light sensor, a pressure sensor, a temperature sensor, a proximity sensor, etc. can also be built in to assist in collecting other information of the electronic device. In this implementation, the motion parameter can include at least one of: data collected by the gravity sensor, data collected by the acceleration sensor, and data collected by the gyroscope.
- A device control method is provided in the disclosure. In the method, the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can be obtained. Then whether the recognition result corresponds to the motion posture indicated by the motion parameter is detected. The recognition result can be determined to be valid in response to detecting that the recognition result corresponds to the motion posture, and the electronic device can be controlled based on the recognition result. The motion parameter of the electronic device in the specified direction can be further recognized in response to detecting that the recognition result does not correspond to the motion posture, such that recognition accuracy can be further improved. According to the method, the motion parameter of the electronic device can be used to verify the recognition result of operating gesture obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- As illustrated in
FIG. 11 , adevice control apparatus 400 is provided in implementations of the present disclosure. Theapparatus 400 runs on the electronic device which includes an ultrasonic emitter and an ultrasonic receiver. Theapparatus 400 includes agesture recognition unit 410, aposture obtaining unit 420, and acontrol unit 430. - The
gesture recognition unit 410 is configured to perform operating gesture recognition through ultrasonic waves, and generate a recognition result. - As an implementation, as illustrated in
FIG. 12 , thegesture recognition unit 410 includes a signal emitting sub-unit 411, asignal receiving unit 412, and agesture recognition unit 413. - The signal emitting sub-unit 411 is configured to trigger the ultrasonic emitter to emit an ultrasonic signal, once the electronic device is in a target scenario. The
signal receiving unit 412 is configured to receive through the ultrasonic receiver a returned ultrasonic signal reflected by an object. Thegesture recognition unit 413 is configured to perform the operating gesture recognition based on the returned ultrasonic signal, and generate the recognition result. - The
posture obtaining unit 420 is configured to obtain a motion parameter of the electronic device. - The
control unit 430 is configured to verify the recognition result based on the motion parameter, and control the electronic device based on a verification result. - The
control unit 430 is configured to detect whether the recognition result corresponds to the motion posture indicated by the motion parameter; determine that the recognition result is valid in response to detecting that the recognition result corresponds to the motion posture, and control the electronic device based on the recognition result; determine that the recognition result is invalid, in response to detecting that the recognition result does not correspond to the motion posture. - As an implementation, the
control unit 430 is configured to determine that the recognition result corresponds to the motion posture indicated by the motion parameter, in response to detecting that a recognized operating gesture indicated by the recognition result is a target gesture, and the motion parameter indicates that the electronic device is stationary. Thecontrol unit 430 is further configured to determine that the recognition result does not correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in a specified direction. A moving direction of the target gesture corresponds to the specified direction. - As illustrated in
FIG. 13 , thecontrol unit 430 includes afirst verifying sub-unit 431 and asecond verifying sub-unit 432. - The
first verifying sub-unit 431 is configured to detect whether the recognition result corresponds to the motion posture indicated by the motion parameter. - The
second verifying sub-unit 432 is configured to: detect whether the motion parameter in the specified direction meets a target condition, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in the specified direction; determine that the recognition result does not correspond to the motion posture in response to detecting that the moving parameter in the direction meets the target condition; determine that the recognition result corresponds to the motion posture, in response to detecting that the moving parameter in the direction does not meet the target condition. - As an implementation, the target condition includes that a moving distance is greater than a specified distance, or a maximum acceleration of a moving electronic device is greater than a specified acceleration.
- It should be noted that, in the present disclosure, the apparatus implementations correspond to the aforementioned method implementations, and specific principles of the apparatus implementations can be referred to the aforementioned method implementations, which will not be repeated here.
- The following will describe an electronic device provided in the present disclosure, with reference to
FIG. 14 . - In implementations of the present disclosure, another
electronic device 200 is configured to perform the aforementioned device control method, based on the aforementioned device control method and apparatus. As illustrated inFIG. 14 , theelectronic device 200 includes one or more (just one illustrated inFIG. 14 ) coupledprocessors 102, amemory 104, anetwork module 106, awireless communication module 108, an ultrasonic emitter 110, and anultrasonic receiver 112. Thememory 104 is configured to store one or more programs which can execute the aforementioned implementations, and theprocessor 102 is configured to execute the one or more programs stored in thememory 104. - The
processor 102 may include one or more cores for data process. Theprocessor 102 is configured to connect with various parts of the wholeelectronic device 200 through various interfaces and circuits, and theprocessor 102 is configured to run or execute instructions, programs, code sets, or instruction sets stored in thememory 104, and invoke data stored in thememory 104 to perform various functions of theelectronic device 200 and process data. Optionally, theprocessor 102 may be implemented by adopting at least one of hardware forms of digital signal processing (DSP), field-programmable gate array (FPGA), programmable logic array (PLA). Theprocessor 102 may integrate one of or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a modem, etc. The CPU is primarily configured to process an operating system, a user interface, applications, etc. The GPU is configured to render and draw display contents. The modem is configured to process the wireless communication. It can be understood that, the modem can also be implemented through a single communication chip and without being integrated into theprocessor 102. - The
memory 104 may include a random access memory (RAM), and may also include a read-only memory (ROM). Thememory 104 is configured to store instructions, programs, codes, code sets, or instruction sets. For example, a device control apparatus may be stored in thememory 104. The device control apparatus may be theaforementioned apparatus 400. Thememory 104 may include a program storage region and a data storage region. The program storage region is configured to store instructions for implementing the operation system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image displaying function, etc.), and instructions for implementing undermentioned implementations, etc. The data storage region is further configured to store data (such as phonebooks, audio and video data, chat record data, etc.) which are created during usage of a storage terminal 100. - A
network module 106 is configured to receive and emit ultrasonic waves to realize conversion between the ultrasonic waves and electrical signals, thereby communicating with communication networks or other devices, for example, communicating with an audio playing device. Thenetwork module 106 may include circuit units (such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption/decryption chip, a subscriber identity module (SIM) card, a memory, etc.) for performing these functions. Thenetwork module 106 can be configured to communicate with various networks such as the Internet, a corporate internet, a wireless network, or other devices through the wireless network. The above wireless network may include a cellular telephone network, a wireless local area network (WLAN), or a metropolitan area network (MAN). For example, thenetwork module 106 may be configured to exchange information with base stations. - A
wireless communication module 108 may be configured to receive and emit Bluetooth signals or infrared signals. An ultrasonic emitter 110 may include a pick-up and a speaker. Anultrasonic receiver 112 may include a first microphone disposed close to the pick-up and a second microphone disposed close to the speaker. -
FIG. 15 is a structural block view illustrating a computer-readable storage medium provided in implementations of the present disclosure. As illustrated inFIG. 15 , A computer-readable storage medium 1100 is configured to store program codes which are invoked by a processor to execute methods described in the aforementioned method implementations. - The computer-
readable storage medium 1100 may be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a hard disk, a ROM, or the like. Optionally, the computer-readable storage medium 1100 may include a non-transitory computer-readable storage medium. The computer-readable storage medium 1100 has a storage space for storing program codes 810 which are configured to execute any of operations in the aforementioned methods. These program codes may be read from one or more computer program products or written into the one or more computer program products. For example,program codes 1110 may be compressed in an appropriate form. - A device control method, a device control apparatus, an electronic device, and a storage medium are provided in implementations of the present disclosure. In the method, the operating gesture recognition can be performed through ultrasonic waves and based on the ultrasonic emitter and the ultrasonic receiver, the recognition result can be generated, and the motion parameter of the electronic device can also be obtained. Then the recognition result is verified based on the motion parameter, and the electronic device is controlled based on the recognition result. According to the method, the motion parameter of the electronic device can be used to verify the result of operating gesture recognition obtained by the ultrasonic waves, such that the accuracy of the gesture recognition through ultrasonic waves can be further improved.
- It should be noted that, the above implementations are only used to illustrate the technical solution of the present disclosure, not to limit it. Although the present disclosure is illustrated in detail with reference to the aforementioned implementations, those skilled in the art should understand that they can still modify the technical solution recorded in the aforementioned implementations, or replace some of technical features equally. These modifications or replacements do not drive the essence of the corresponding technical solution to deviate from the spirit and scope of the technical solution of each implementations of the present disclosure.
Claims (20)
- A device control method, applicable to an electronic device comprising an ultrasonic emitter and an ultrasonic receiver, and comprising:performing operating gesture recognition by the electronic device through ultrasonic waves, and generating a recognition result;obtaining a motion parameter of the electronic device; andverifying the recognition result based on the motion parameter, and controlling the electronic device based on a verification result.
- The method of claim 1, wherein verifying the recognition result based on the motion parameter, and controlling the electronic device based on the verification result comprises:detecting whether the recognition result corresponds to a motion posture indicated by the motion parameter; anddetermining that the recognition result is valid in response to detecting that the recognition result corresponds to the motion posture, and controlling the electronic device based on the recognition result.
- The method of claim 2, further comprising:
determining that the recognition result is invalid in response to detecting that the recognition result does not correspond to the motion posture. - The method of claim 2 or 3, wherein detecting whether the recognition result corresponds to the motion posture indicated by the motion parameter comprises:determining that the recognition result corresponds to the motion posture, in response to detecting that a recognized operating gesture indicated by the recognition result is a target gesture, and the motion parameter indicates that the electronic device is stationary; anddetermining that the recognition result does not correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in a specified direction, wherein a moving direction of the target gesture corresponds to the specified direction.
- The method of claim 4, wherein determining that the recognition result does not correspond to the motion posture, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in the specified direction comprises:detecting whether a moving parameter in the specified direction meets a target condition, in response to detecting that the recognized operating gesture indicated by the recognition result is the target gesture, and the motion parameter indicates that the electronic device is moving in the specified direction; anddetermining that the recognition result does not correspond to the motion posture, in response to detecting that the moving parameter in the specified direction meets the target condition.
- The method of claim 5, further comprising:
determining that the recognition result corresponds to the motion posture, in response to detecting that the moving parameter in the direction does not meet the target condition. - The method of claim 5 or 6, wherein the target condition comprises:a moving distance is greater than a specified distance; ora maximum acceleration when moving is greater than a specified acceleration.
- The method of claim 1, wherein the recognition result is configured with generation time, and the method further comprises:
before verifying the recognition result based on the motion parameter, and controlling the electronic device based on the verification result,obtaining generation time corresponding to a historically updated recognition result by taking generation time corresponding to a latest updated recognition result as a starting point, in response to detecting that a configuration file storing the recognition result is updated;obtaining the number of recognition results within a specified time period starting from the generation time corresponding to the latest updated recognition result, wherein the specified time period is a length of time taken by a user to complete a non-touch operating gesture;determining that the recognition result is invalid, in response to the number being greater than 1; andtaking the recognition result as a recognition result to be verified with the motion parameter of the electronic device, in response to the number being 1. - The method of claim 1, wherein obtaining the motion parameter of the electronic device comprises:
collecting the motion parameter through a configured gyroscope. - The method of claim 1, wherein the motion parameter of the electronic device comprises at least one of: a motion direction, an acceleration, and a motion distance.
- The method of claim 1, wherein the motion parameter comprises at least one of: data collected by a gravity sensor, data collected by an acceleration sensor, and data collected by a gyroscope.
- The method of claim 1, further comprising:
emitting an ultrasonic signal, in response to detecting that a specified application is launched, wherein the specified application allows to enable the operating gesture recognition based on the ultrasonic waves. - The method of claim 1, further comprising:searching a stored list-based configuration file for a currently launched or foreground running application, wherein an application in the list allows to enable the operating gesture recognition based on the ultrasonic waves; andemitting an ultrasonic signal, on condition that the currently launched or foreground running application exists in the stored list-based configuration file.
- The method of claim 13, wherein searching the stored list-basted configuration file for the currently launched or foreground running application comprises:obtaining a state of a controlling control corresponding to the currently launched or foreground running application; andsearching the stored list-based configuration file for the currently launched or foreground running application, based on the state.
- The method of any of claims 1 to 14, wherein performing the operating gesture recognition by the electronic device through ultrasonic waves, and generating the recognition result comprises:triggering the ultrasonic emitter to emit an ultrasonic signal, once the electronic device is in a target scenario;receiving through the ultrasonic receiver a returned ultrasonic signal reflected by an object; andperforming the operating gesture recognition based on the returned ultrasonic signal and generating the recognition result.
- A device control apparatus, applicable to an electronic device comprising an ultrasonic emitter and an ultrasonic receiver, and comprising:a gesture recognition unit configured to perform operating gesture recognition through ultrasonic waves, and generate a recognition result;a posture obtaining unit configured to obtain a motion parameter of the electronic device; anda control unit configured to verify the recognition result based on the motion parameter, and control the electronic device based on a verification result.
- The apparatus of claim 16, wherein the gesture recognition unit comprises:a signal emitting sub-unit configured to trigger the ultrasonic emitter to emit an ultrasonic signal, once the electronic device is in a target scenario;a signal receiving unit configured to receive through the ultrasonic receiver a returned ultrasonic signal reflected by an object; anda gesture recognition unit configured to perform the operating gesture recognition based on the returned ultrasonic signal, and generate the recognition result.
- The apparatus of claim 16, wherein
the control unit is configured to detect whether the recognition result corresponds to a motion posture indicated by the motion parameter; determine that the recognition result is valid in response to detecting that the recognition result corresponds to the motion posture, and control the electronic device based on the recognition result; and determine that the recognition result is invalid, in response to detecting that the recognition result does not correspond to the motion posture. - An electronic device, comprising an ultrasonic emitter, an ultrasonic receiver, a processor, and a memory, wherein one or more programs are stored in the memory and are configured to be executed by the processor to perform the method of any of claims 1 to 15.
- A computer-readable storage medium configured to store program codes which, when operated by a processor, are configured to perform the method of any of claims 1 to 15.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910701490.0A CN110502108B (en) | 2019-07-31 | 2019-07-31 | Equipment control method and device and electronic equipment |
PCT/CN2020/103088 WO2021017927A1 (en) | 2019-07-31 | 2020-07-20 | Method and device for controlling apparatus, electronic apparatus, and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3974951A1 true EP3974951A1 (en) | 2022-03-30 |
EP3974951A4 EP3974951A4 (en) | 2022-08-24 |
Family
ID=68586902
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20845961.0A Withdrawn EP3974951A4 (en) | 2019-07-31 | 2020-07-20 | Method and device for controlling apparatus, electronic apparatus, and storage medium |
Country Status (4)
Country | Link |
---|---|
US (1) | US11693484B2 (en) |
EP (1) | EP3974951A4 (en) |
CN (1) | CN110502108B (en) |
WO (1) | WO2021017927A1 (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110502108B (en) * | 2019-07-31 | 2021-08-17 | Oppo广东移动通信有限公司 | Equipment control method and device and electronic equipment |
CN113359139A (en) * | 2020-03-03 | 2021-09-07 | 阿里巴巴集团控股有限公司 | Intelligent device, switch device, electronic device and interaction device |
CN111580660B (en) * | 2020-05-09 | 2022-03-18 | 清华大学 | Operation triggering method, device, equipment and readable storage medium |
CN112924857B (en) * | 2021-01-22 | 2024-06-11 | 维沃移动通信有限公司 | Electronic equipment and key state detection method thereof |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140282280A1 (en) * | 2013-03-14 | 2014-09-18 | Samsung Electronics Co., Ltd. | Gesture detection based on time difference of movements |
US20140282272A1 (en) * | 2013-03-15 | 2014-09-18 | Qualcomm Incorporated | Interactive Inputs for a Background Task |
US9811311B2 (en) * | 2014-03-17 | 2017-11-07 | Google Inc. | Using ultrasound to improve IMU-based gesture detection |
CN104978014B (en) * | 2014-04-11 | 2018-05-11 | 维沃移动通信有限公司 | A kind of method and its mobile terminal of quick calling application program or systemic-function |
US9354709B1 (en) * | 2014-06-17 | 2016-05-31 | Amazon Technologies, Inc. | Tilt gesture detection |
CN104267819B (en) * | 2014-10-09 | 2017-07-14 | 苏州触达信息技术有限公司 | Can gesture wake up electronic equipment and electronic equipment gesture awakening method |
CN104463119B (en) * | 2014-12-05 | 2017-10-31 | 苏州触达信息技术有限公司 | Combined type gesture identification equipment and its control method based on ultrasound with vision |
CN105278667A (en) | 2014-12-16 | 2016-01-27 | 维沃移动通信有限公司 | Data interaction method, data interaction system and mobile terminal |
KR102462856B1 (en) * | 2015-06-24 | 2022-11-03 | 삼성전자주식회사 | Usability notification providing method for extendable area of display and electronic device supporting the same |
US10488937B2 (en) * | 2015-08-27 | 2019-11-26 | Verily Life Sciences, LLC | Doppler ultrasound probe for noninvasive tracking of tendon motion |
CN105391854A (en) * | 2015-10-15 | 2016-03-09 | 广东欧珀移动通信有限公司 | Audio incoming call processing method and audio incoming call processing device |
CN105242861B (en) * | 2015-10-15 | 2019-08-02 | Oppo广东移动通信有限公司 | A kind of parameter adjusting method based on ultrasound and device |
CN109344586B (en) * | 2018-10-24 | 2021-04-06 | Oppo广东移动通信有限公司 | Unlocking method and device, storage medium and mobile terminal |
CN110502108B (en) * | 2019-07-31 | 2021-08-17 | Oppo广东移动通信有限公司 | Equipment control method and device and electronic equipment |
-
2019
- 2019-07-31 CN CN201910701490.0A patent/CN110502108B/en active Active
-
2020
- 2020-07-20 EP EP20845961.0A patent/EP3974951A4/en not_active Withdrawn
- 2020-07-20 WO PCT/CN2020/103088 patent/WO2021017927A1/en unknown
-
2021
- 2021-12-17 US US17/555,039 patent/US11693484B2/en active Active
Also Published As
Publication number | Publication date |
---|---|
US20220107689A1 (en) | 2022-04-07 |
US11693484B2 (en) | 2023-07-04 |
WO2021017927A1 (en) | 2021-02-04 |
CN110502108A (en) | 2019-11-26 |
EP3974951A4 (en) | 2022-08-24 |
CN110502108B (en) | 2021-08-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3974951A1 (en) | Method and device for controlling apparatus, electronic apparatus, and storage medium | |
US10699093B2 (en) | Mobile terminal, method and device for displaying fingerprint recognition region | |
TWI679585B (en) | Method for displaying fingerprint identification area and mobile terminal | |
CN106778585B (en) | A kind of face key point-tracking method and device | |
US10963661B2 (en) | Method for optical fingerprint collection and related products | |
EP4109218B1 (en) | Mobile phone comprising a touch screen with an in-display fingerprint sensor | |
US20180314875A1 (en) | Electronic device, fingerprint recognition control method and device | |
WO2018201849A1 (en) | Optical fingerprint acquisition method, and related product | |
CN109035914B (en) | Learning method based on intelligent desk lamp and intelligent desk lamp | |
EP3588341A1 (en) | Unlocking control method and related product | |
CN109857245B (en) | Gesture recognition method and terminal | |
CN110113528B (en) | Parameter obtaining method and terminal equipment | |
CN107463897B (en) | Fingerprint identification method and mobile terminal | |
CN109240551B (en) | Method for controlling electronic device by using gestures and related product | |
CN108984066B (en) | Application icon display method and mobile terminal | |
CN113919390A (en) | Method for identifying touch operation and electronic equipment | |
CN109472825B (en) | Object searching method and terminal equipment | |
CN109164908B (en) | Interface control method and mobile terminal | |
CN111638522B (en) | Proximity detection method and electronic device | |
CN109947345B (en) | Fingerprint identification method and terminal equipment | |
CN112256135A (en) | Equipment control method and device, equipment and storage medium | |
CN110031860B (en) | Laser ranging method and device and mobile terminal | |
CN109618234A (en) | Video playing control method, device, mobile terminal and storage medium | |
CN108958505B (en) | Method and terminal for displaying candidate information | |
CN107995416B (en) | A kind of focus adjustment method and mobile terminal |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20211224 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20220725 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/043 20060101ALI20220719BHEP Ipc: G06F 3/01 20060101AFI20220719BHEP |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20240125 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN |
|
18W | Application withdrawn |
Effective date: 20240321 |