CN117354662A

CN117354662A - Control method and related device of terminal equipment

Info

Publication number: CN117354662A
Application number: CN202210753754.9A
Authority: CN
Inventors: 徐弘笛; 曹明轩
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2022-06-29
Filing date: 2022-06-29
Publication date: 2024-01-05

Abstract

According to the control method and the related device of the terminal equipment, the first instruction input by the user is received, and the first instruction is used for setting the name of the first gesture customized by the user; recording first audio data corresponding to the first gesture through a first wearable device, wherein the first wearable device comprises a microphone, and communication connection is established between the first wearable device and a terminal device; establishing a first mapping relation among a first audio feature of the first audio data, a name of a first gesture and a first control instruction in the terminal equipment; receiving second audio data from the first wearable device; and controlling the terminal equipment based on the second audio data and the first mapping relation. By enabling the user to define the control command corresponding to the sound sent by the gesture, the structural complexity of the wearable device is reduced, and meanwhile, the convenience of man-machine interaction between the earphone and the user is improved.

Description

Control method and related device of terminal equipment

Technical Field

The present disclosure relates to the field of terminal devices, and in particular, to a control method and a related device for a terminal device.

Background

With the development and popularity of wireless headphones, currently, the mainstream wireless headphones are separated from the connected terminal device, and individually respond to the operation of the user. For example, when it is detected that the user presses a key provided on the headphone, pause/play, up/down, and the like operations may be realized.

However, in order to realize the above functions, additional function keys, pressure sensors, and the like are required.

Disclosure of Invention

The control method and the related device for the terminal equipment reduce the structural complexity of the wearable equipment and improve the control convenience of the terminal equipment.

In a first aspect, the present application provides a control method of a terminal device, including: receiving a first instruction input by a user, wherein the first instruction is used for setting the name of a first gesture customized by the user; recording first audio data corresponding to the first gesture through a first wearable device, wherein the first wearable device comprises a microphone, and communication connection is established between the first wearable device and a terminal device; establishing a first mapping relation among a first audio feature of the first audio data, a name of a first gesture and a first control instruction in the terminal equipment; receiving second audio data from the first wearable device; and controlling the terminal equipment based on the second audio data and the first mapping relation.

Optionally, the user-defined gesture may be a gesture form in which the user determines to make a specific friction sound by himself; as an example, the gesture may be in the form of a ringing finger, a finger striking another finger, a finger scraping another knuckle. Or, the user-defined gesture may also refer to a gesture form corresponding to friction sounds emitted by the user in different directions of the wearable device; as an example, a gesture corresponding to a first friction sound recorded by an earphone worn on a left ear may be the same as a gesture corresponding to a second friction sound recorded by an earphone worn on a right ear, and the corresponding first friction sound and second friction sound may be the same, and names corresponding to the left ear and the right ear may be defined respectively. Or, the user-defined gesture may also refer to a gesture form corresponding to a friction sound generated by the user at a different position from the wearable device.

According to the method, the user can define the control commands corresponding to different gestures by himself, the man-machine interaction convenience between the earphone and the user is improved, and meanwhile the structural complexity of the wearable device is reduced.

With reference to the first aspect, in some implementation manners of the first aspect, recording, by a first wearable device, first audio data corresponding to the first gesture includes: receiving a second instruction input by a user, wherein the second instruction is used for indicating the terminal equipment to start recording audio data corresponding to the gesture; sending first indication information to the first wearable device, wherein the first indication information is used for indicating the first wearable device to start a microphone; first audio data from a first wearable device is received.

According to the method, through the visual option of starting to record the sound, man-machine interaction between the user and the equipment is enhanced.

With reference to the first aspect and the foregoing implementation manner, in some implementation manners of the first aspect, after sending the first indication information to the first wearable device, before receiving the first audio data from the first wearable device, the method further includes: and sending second indication information to the first wearable device, wherein the second indication information is used for indicating the first wearable device to close the microphone and sending the collected audio data to the terminal device.

According to the method, the user can actively select the time for stopping sound recording, so that richer audio data content can be provided, and a basis is provided for improving the accuracy of the identification of the follow-up gesture.

With reference to the first aspect and the foregoing implementation manner, in some implementation manners of the first aspect, after sending the first indication information to the first wearable device, before sending the second indication information to the first wearable device, the method further includes: outputting second prompt information, wherein the second prompt information is used for prompting a user to select audio data corresponding to the gesture of stopping recording; and receiving a third instruction input by the user, wherein the third instruction is used for selecting the audio data corresponding to the recording stopping gesture.

According to the method, through the visual option of stopping recording the sound, man-machine interaction between the user and the equipment is enhanced.

With reference to the first aspect and the foregoing implementation manners, in some implementation manners of the first aspect, before receiving the first audio data from the first wearable device, the method further includes: and sending third indication information to the first wearable device, wherein the third indication information is used for indicating the first wearable device to turn off the microphone after the first microphone is started for a first time period and sending the collected audio data to the terminal device.

Alternatively, the third indication information may be carried in the first indication information, or may be carried in the second indication information. Or, the third indication information may be sent separately, and when the first indication information or the second indication information is sent to the wearable device, the third indication information is sent simultaneously, or when the starting time of the microphone reaches the first time, the third indication information is sent.

According to the method, through presetting the recording duration of the sound, the user does not need to manually trigger the recording stopping instruction, excessive audio data caused by overlong recording duration is avoided, the consumption of computing resources is increased, and therefore the efficiency of sound recording is improved.

With reference to the first aspect and the foregoing implementation manner, in some implementation manners of the first aspect, the method further includes: and outputting third prompt information, wherein the third prompt information is used for prompting a user to make a first gesture around the first wearable device.

According to the method, the terminal equipment is used for displaying the guide interface for gesture recording, so that a user is guided to record gestures, human interaction is improved, learning cost is reduced, and the user is convenient to operate by quickly.

With reference to the first aspect and the foregoing implementation manners, in some implementation manners of the first aspect, before receiving a first instruction input by a user, the method further includes: receiving a fourth instruction input by a user, wherein the fourth instruction is used for requesting to customize gestures associated with control instructions in terminal equipment; and outputting fourth prompt information, wherein the fourth prompt information is used for prompting a user to input the name of the custom gesture.

According to the method, the user can edit the name of the gesture to be input by self-definition, so that the memory of the user on the gesture is enhanced, and misoperation in the subsequent use process is reduced.

With reference to the first aspect and the foregoing implementation manner, in some implementation manners of the first aspect, before receiving a fourth instruction input by a user, the method further includes: receiving a fifth instruction input by a user, wherein the fifth instruction is used for requesting to set a mapping relation between the gesture and a control instruction in the terminal equipment; and outputting fifth prompt information, wherein the fifth prompt information is used for a user to select gestures which are customized and are used for being associated with control instructions in the terminal equipment.

Optionally, the fifth prompt information may be output through a display screen of the terminal device, or may be output through other manners such as voice.

According to the method, the user can select the control command corresponding to the gesture of the custom edit name by himself, so that the learning and memory cost of the user is reduced, the user can conveniently memorize the gesture and the control command corresponding to the gesture, and the subsequent misoperation during gesture command control is further reduced.

With reference to the first aspect and the foregoing implementation manner, in some implementation manners of the first aspect, before establishing a first mapping relationship between the first audio feature of the first audio data, the name of the first gesture, and the first control instruction in the terminal device, the method further includes: extracting first audio features in the first audio data; and determining that the gesture audio features for associating the control instruction in the terminal device do not contain the first audio features.

According to the method, whether the audio features of the currently entered gesture exist or not is judged, and the situation that the gesture corresponds to multiple control commands is avoided.

With reference to the first aspect and the foregoing implementation manners, in some implementation manners of the first aspect, the first wearable device includes a plurality of microphones, and the first audio data is recorded by a first microphone of the plurality of microphones; the method for establishing the first mapping relation among the first audio feature of the first audio data, the name of the first gesture and the first control instruction in the terminal device comprises the following steps: establishing a first mapping relation among the first microphone, a first audio feature of the first audio data, a name of a first gesture and a first control instruction in the terminal equipment; controlling the terminal device based on the second audio data and the first mapping relation, including: and controlling the terminal equipment based on the second audio data, the microphone recorded to the second audio data and the first mapping relation.

Alternatively, the wearable device may comprise two or more microphones.

According to the method, input or recognition of sound is performed asynchronously or simultaneously through two or more microphones, so that gesture input and gesture recognition efficiency can be improved.

In a second aspect, the present application provides a control apparatus of a terminal device. The apparatus comprises respective functional modules for implementing the method in any one of the possible implementations of the first aspect, each functional module being implemented in software and/or hardware.

For example, the apparatus may include: and a receiving module: the method comprises the steps of receiving a first instruction input by a user, wherein the first instruction is used for setting the name of a first gesture customized by the user; receiving second audio data from the first wearable device; the processing module is used for: the first audio data corresponding to the first gesture are recorded through the first wearable device, the first wearable device comprises a microphone, and communication connection is established between the first wearable device and the terminal device; establishing a first mapping relation among a first audio feature of the first audio data, a name of a first gesture and a first control instruction in the terminal equipment; and controlling the terminal equipment based on the second audio data and the first mapping relation.

Alternatively, the apparatus may be a computing device, or may be a chip that can be applied to the computing device.

In a third aspect, the present application provides an electronic device, comprising: a processor, a memory communicatively coupled to the processor; the memory stores computer-executable instructions; the processor executes computer-executable instructions stored in the memory to implement the method of controlling a terminal device of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, are adapted to carry out the method of controlling a terminal device of the first aspect.

In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method of controlling a terminal device of the first aspect.

According to the control method and the related device of the terminal equipment, the first instruction input by the user is received, and the first instruction is used for setting the name of the first gesture customized by the user; recording first audio data corresponding to the first gesture through a first wearable device, wherein the first wearable device comprises a microphone, and communication connection is established between the first wearable device and a terminal device; establishing a first mapping relation among a first audio feature of the first audio data, a name of a first gesture and a first control instruction in the terminal equipment; receiving second audio data from the first wearable device; and controlling the terminal equipment based on the second audio data and the first mapping relation. The user can define the control command corresponding to the sound sent by the gesture by himself, so that the learning cost of the user is reduced, and the convenience of man-machine interaction between the earphone and the user is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

Fig. 1 is an interaction schematic diagram between terminal devices according to an embodiment of the present application;

fig. 2 is an interaction schematic diagram between terminal devices in an application scenario provided in an embodiment of the present application;

fig. 3 is an interaction schematic diagram of a wireless earphone according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 5 is a block diagram of a software architecture of an electronic device according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a wireless earphone according to an embodiment of the present disclosure;

fig. 7 is a flowchart of a control method of a terminal device according to an embodiment of the present application;

fig. 8 is a schematic diagram of a user interface of a terminal device according to an embodiment of the present application;

fig. 9 is an interaction schematic diagram of another wireless earphone according to an embodiment of the present application;

fig. 10 is a schematic diagram of a user interface of another terminal device according to an embodiment of the present application;

fig. 11 is a flowchart of another control method of a terminal device according to an embodiment of the present application;

Fig. 12 is a schematic diagram of a user interface of another terminal device according to an embodiment of the present application;

fig. 13 is a flowchart of another control method of a terminal device according to an embodiment of the present application;

fig. 14 is a flowchart of another control method of a terminal device according to an embodiment of the present application;

fig. 15 is a flowchart of another control method of a terminal device according to an embodiment of the present application;

fig. 16 is a schematic structural diagram of a control device of a terminal device according to an embodiment of the present application;

fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

The terms "first" and "second" are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature.

Fig. 1 is a diagram of an exemplary system architecture to which embodiments of the present application are applicable. As shown in fig. 1, a terminal device 100 and a wireless headset 200 may be included in the system.

The terminal device 100 may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (personal digital assistant, PDA), or the like. The wireless headset 200 may be a true wireless headset (truer wireless headset), a bluetooth headset, or the like.

As an example, when the wireless headset 200 detects that the terminal device 100 is nearby, or the terminal device 100 detects that the wireless headset 200 is nearby, both may establish a communication connection. For example, both may establish a communication connection through Bluetooth (BT), near field communication technology (near field communication, NFC), or the like.

Fig. 2 is an exemplary diagram of an application scenario in an embodiment of the present application. As shown in fig. 2, after a communication connection is established between the terminal device 100 and the wireless headset 200, the user can operate the wireless headset 200, the wireless headset 200 receives an operation instruction input by the user and transmits the operation instruction to the terminal device 100, and then the terminal device 100 controls the wireless headset 200 or the terminal device 100 based on the operation instruction.

For example, the terminal device 100 turns up the play volume of the wireless earphone, turns down the play volume of the wireless earphone 200, makes a call, hangs up a call, displays a short message content or a voice play short message content, and the like based on the operation instruction.

In some implementations, keys and/or buttons may be added to the wireless headset in order to allow a user to control the terminal device by operating the wireless headset. The user may control the terminal device via keys and/or buttons on the wireless headset.

But limited by the volume of the wireless headset, adding keys and/or buttons to the wireless headset is not ideal. Aiming at the problems, the embodiment of the application provides a new technical scheme.

In the technical solution provided in the present application, after the terminal device 100 and the wireless headset 200 establish a communication connection, a user may make a gesture capable of rubbing out a sound beside the wireless headset 200, the wireless headset 200 collects the sound and sends the sound to the terminal device 100; the terminal device 100 determines a control instruction corresponding to a gesture of a user based on a mapping relationship between a sound and the control instruction stored in advance, and executes the control instruction.

In some implementations, the terminal device 100 or the wireless headset 200 may pre-detect and record the audio signal manufactured by the user, recognize the friction sound therein, and establish the mapping relationship between the friction sound and the control command according to the user's instruction.

Optionally, a mapping relation between the gesture and the control instruction may be preset in the terminal device, and then the terminal device outputs a prompt message to guide the user to make the gesture, so that the wireless earphone may record a friction sound corresponding to the gesture and send the friction sound to the terminal device; after receiving the friction sound, the terminal equipment establishes a mapping relation between the friction sound and a control instruction corresponding to the gesture, or establishes a mapping relation between the friction sound and the gesture.

Optionally, the terminal device may output a prompt message to guide the user to customize a gesture and make the gesture beside the wireless earphone, so that the wireless earphone may record a friction sound corresponding to the gesture and send the friction sound to the terminal device; after the terminal device receives the friction sound, a mapping relation between the friction sound and the control instruction can be established based on the instruction input by the user, or a mapping relation between the friction sound and the gesture and a mapping relation between the gesture and the control instruction can be established.

In some implementations, when the terminal device establishes a mapping relationship between the friction sound and the control command, or establishes a mapping relationship between the friction sound and the gesture and between the gesture and the control command, the mapping relationship may be performed in combination with an earphone position, where the earphone position includes whether the earphone is a left earphone or a right earphone.

For example, friction sounds collected by headphones in different orientations correspond to different control commands even for the same gesture, either the friction sounds or the gesture.

In which the input of the friction sound may refer to that the friction sound generated by the body part is used as an input signal of the electronic device or electronic equipment. An electronic device or electronic apparatus according to one or more embodiments may receive an audio signal using a microphone, detect a friction sound from the received audio signal, and use the friction sound as an input signal to the electronic device or electronic apparatus. Such input may be referred to as friction sound input. The electronic apparatus or the electronic device may receive the frictional sound input by using the microphone.

For example, taking the wireless headset 200 as a real wireless headset, as shown in fig. 3, the real wireless headset 200 may receive audio information sent by a user through a finger-ringing gesture mode, and at the same time, detect the type of friction sound (i.e., finger-ringing gesture) in the audio information.

It will be appreciated that the type of friction sound may be determined by the body part that generated the friction sound, or the type of body movement that generated the friction sound. Wherein the type of body movement may be determined based on at least one of a direction of movement of the location where the friction occurs and a method of friction, or a combination thereof. Wherein, the body parts generating friction sound can be classified into fingers, back of hand, palm, etc.; the type of movement of the body may be classified into a gesture of rubbing another finger with a finger, a gesture of flicking another finger with a finger, and the like.

The terminal device applicable to the embodiment of the application can be a mobile phone, a tablet personal computer, a personal computer (personal computer, PC), an intelligent screen, an artificial intelligence (artificial intelligence, AI) sound box, a car machine device, a smart watch and other wearable terminal devices, various teaching auxiliary tools (such as a learning machine, an early education machine), an intelligent toy, a portable robot, a personal digital assistant (personal digital assistant, a PDA), an augmented reality technology (augmented reality, AR) device, a Virtual Reality (VR) device and the like, and also can be a device with a mobile office function, a device with an intelligent home function, a device with an audio-video entertainment function, a device supporting intelligent travel and the like. It should be understood that the embodiments of the present application do not limit the specific technology and specific device configuration adopted by the terminal device.

In order to better understand the embodiments of the present application, the following describes a hardware structure of the terminal device of the embodiments of the present application. Fig. 4 is a schematic structural diagram of a terminal device according to an embodiment of the present application.

The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a sensor module 180, keys 190, an indicator 192, a camera 193, a display screen 194, and the like.

Alternatively, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device may include more or less components than illustrated, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units. Wherein the different processing units may be separate devices or may be integrated in one or more processors. A memory may also be provided in the processor 110 for storing instructions and data.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge a terminal device, or may be used to transfer data between the terminal device and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other terminal devices, such as AR devices, etc.

The charge management module 140 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used for connecting the charge management module 140 and the processor 110.

The wireless communication function of the terminal device may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Antennas in the terminal device may be used to cover single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas.

The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G or the like applied on a terminal device. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wirelesslocal area networks, WLAN) (e.g., wireless fidelity (wireless fidelity, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), etc. as applied on a terminal device.

The terminal device implements display functions through a graphics processor (graphics processing unit, GPU), a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. GPUs may also be referred to as display cores, visual processors, display chips, and the like.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, N being a positive integer greater than 1.

The terminal device can implement a photographing function through an image processor (image signal processor, ISP), a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The camera 193 is used to capture still images or video. In some embodiments, the terminal device may include 1 or N cameras 193, N being a positive integer greater than 1.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to realize expansion of the memory capability of the terminal device. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 121 may be used to store computer-executable program code that includes instructions. The internal memory 121 may include a storage program area and a storage data area.

The terminal device may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an application processor, and the like. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The speaker 170A, also referred to as a "horn," is used to convert audio electrical signals into sound signals. The terminal device can listen to music through the speaker 170A or listen to hands-free calls. A receiver 170B, also referred to as a "earpiece", is used to convert the audio electrical signal into a sound signal. When the terminal device picks up a call or voice message, the voice can be picked up by placing the receiver 170B close to the human ear. Microphone 170C, also referred to as a "microphone" or "microphone", is used to convert sound signals into electrical signals.

The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The gyro sensor 180B may be used to determine a motion gesture of the terminal device. The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a hall sensor. The acceleration sensor 180E may detect the magnitude of acceleration of the terminal device in various directions (typically three axes). A distance sensor 180F for measuring a distance. The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense ambient light level. The fingerprint sensor 180H is used to collect a fingerprint. The temperature sensor 180J is for detecting temperature. The touch sensor 180K, also referred to as a "touch device". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The bone conduction sensor 180M may acquire a vibration signal.

The keys 190 include a power-on key, a volume key, etc. The keys 190 may be mechanical keys. Or may be a touch key. The terminal device may receive key inputs, generating key signal inputs related to user settings of the terminal device and function control. The indicator 192 may be an indicator light, may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc.

The software system of the terminal device can adopt a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture or a cloud architecture. The layered architecture may adopt an Android (Android) system, an apple (IOS) system, or other operating systems, which is not limited in this embodiment of the present application. Taking an Android system with a layered architecture as an example, a software structure of the terminal device is illustrated.

Fig. 5 is a software architecture block diagram of a terminal device applicable to the embodiment of the present application. The layered architecture divides the software system of the terminal device into a plurality of layers, each layer having a distinct role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system may be divided into four layers, an application layer (applications), an application framework layer (application framework), an Zhuoyun rows (Android run) and system libraries, and a kernel layer (kernel) in order from top to bottom.

The application layer may include a display module and a series of application packages, the application layer running the application by calling an application program interface (application programming interface, API) provided by the application framework layer.

The application framework layer provides APIs and programming frameworks for application programs of the application layer. The application framework layer includes a number of predefined functions.

The android system runtime comprises a core library and a virtual machine. And the android system is responsible for scheduling and managing the android system when running. The core library consists of two parts: one part is a function which needs to be called by Java language, and the other part is a core library of android. The application layer and the application framework layer run in a virtual machine. The virtual machine executes Java files of the application layer and the application framework layer as binary files. The virtual machine is used for executing the functions of object life cycle management, stack management, thread management, security and exception management, garbage collection and the like. The system library may contain modules for a number of functions, such as: a surface manager, a media library, a three-dimensional graphics processing library, an identification algorithm module, and the like.

The surface manager is used to manage the display subsystem and provides a fusion of the two-dimensional and three-dimensional layers for the plurality of applications. Media libraries support a variety of commonly used audio, video format playback and recording, still image files, and the like. The media library may support a variety of audio video encoding formats, such as: JPG, PNG, etc. The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like. The recognition algorithm module can be used for sign language recognition, voice recognition and text semantic recognition. The sign language recognition means that the voice or the characters are recognized as the sign language, the voice recognition means that the sign language or the characters are recognized as the voice, and the character semantic recognition means that the sign language or the voice is recognized as the characters.

The kernel layer is a layer between hardware and software. The kernel layer is used for driving the hardware so that the hardware works. The kernel layer at least comprises a display driver, a fingerprint screen driver, a camera driver, a Bluetooth driver and the like, which is not limited in the embodiment of the application.

It will be appreciated that at least one virtual machine may be deployed in a terminal device, where each virtual machine in the at least one virtual machine has an independent operating system and application program, each virtual machine may independently run software, save data, and the running environment of each virtual machine may be different.

Each virtual machine may be created by a virtual machine monitor, which is a middleware layer running between underlying hardware and an operating system that allows the operating system and applications of at least one virtual machine to share hardware so that each of the at least one virtual machine can access all hardware devices on the terminal device.

For example, each virtual machine may access a display device of the terminal device through a virtual machine monitor to display relevant content of an application program in each virtual machine.

In this embodiment, the terminal device obtains the audio information generated by the user in the surrounding environment through the microphone 170C in the audio module 170, the processor 110 detects and analyzes the type of the friction sound in the audio information, and displays the type of the friction sound through the display screen 194, and suggests the mapping relationship between the type of friction sound and the corresponding control command according to the instruction of the user.

As an example, a microphone of a terminal device may detect audio information within a preset range (e.g., without limitation, 15 cm) centered on the device in real time or periodically, audio information of a user gesture (e.g., without limitation, fingering, clicking, double clicking, scraping, etc.) is converted into an electrical signal by the microphone and transmitted to a processor, which may analyze the audio information through a machine learning algorithm, then set the audio information to a new start signal, and may prompt the user to indicate a new control command (e.g., without limitation, cutting songs, adjusting volume, opening a specific application, etc.) to the start signal.

As yet another example, the terminal device may display a guiding interface for inputting the sound of a gesture (such as, but not limited to, finger-ringing, clicking, double-clicking, scraping, etc.) through a display screen, according to which a user emits the friction sound of the gesture displayed by the guiding interface within a preset range (such as, but not limited to, 15 cm) centered on the device, and the microphone converts the friction sound into an electrical signal and transmits the electrical signal to the processor, and the processor may compare the input friction sound with a preset sound template through a machine learning algorithm, thereby adjusting the classification algorithm to achieve better classification accuracy. Optionally, after the user completes the input of the gesture sound according to the guiding interface, the control command corresponding to the gesture may be adjusted, that is, the terminal device establishes a mapping relationship between the friction sound and a new control command (for example, but not limited to, cutting song, adjusting volume, opening a specific application program, etc.) according to the instruction of the user.

It can be understood that when the user performs audio information recording, that is, the microphone converts the friction sound into a corresponding electric signal, the terminal device can prompt the user to repeatedly record in different directions of the terminal device, so that correction of the machine learning classification algorithm is facilitated, and recognition accuracy of the subsequent terminal device on the friction sound is improved.

In order to better understand the embodiments of the present application, the following describes a hardware structure of the wireless earphone of the embodiments of the present application. Fig. 6 is a schematic structural diagram of a wireless headset 200 according to an embodiment of the present application.

As shown in fig. 6, the wireless headset may include a processor 201, a memory 202, a wireless communication module 203, a power switch 204, a microphone 205, a diaphragm 206, and the like. The modules may be connected by a bus or other manner, and in this embodiment, the modules are exemplified by a bus connection.

Processor 201 may be used to read and execute computer readable instructions. In a specific implementation, the processor 201 may mainly include a controller, an operator, and a register. The controller is mainly responsible for instruction decoding and sending out control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for storing register operands, intermediate operation results and the like temporarily stored in the instruction execution process. In particular implementations, the hardware architecture of the processor 201 may be an application specific integrated circuit (application specific integrated circuit, ASIC) architecture, a microprocessor without internal interlocking pipeline stages (microprocessor without interlocked piped stages architecture, MIPS) architecture, an advanced reduced instruction set machine (advanced reduced instruction set computer machine, ARM) architecture, or a network processor (network processor, NP) architecture, among others.

In some embodiments, the processor 201 may be configured to parse data information received by the wireless communication module 203. The processor 201 may respond to the data information by performing the corresponding operation. The processor 201 may also be configured to generate signals, such as bluetooth signals, that are transmitted outwardly by the wireless communication module 203.

Memory 202 is coupled to processor 201 for storing various software programs and/or sets of instructions. In particular implementations, memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 202 may also store communication programs that may be used to communicate with the terminal device 100, or other electronic devices.

The wireless communication module 203 may include a bluetooth communication module 203A or others. The electronic device may receive or transmit wireless signals through one or more of the bluetooth communication modules 203A to establish wireless communication connections with other electronic devices. The electronic equipment can acquire data transmitted by other electronic equipment through the wireless communication connection, and can also send data instructions to other electronic equipment through the wireless communication connection. Among other things, bluetooth communication module 203A may provide a solution that includes one or more of classical bluetooth (BR/EDR) or bluetooth low energy (bluetooth low energy, BLE) bluetooth communication. Optionally, the wireless communication module 203 may further include a WLAN communication module (not shown in fig. 2) that may provide a solution including one or more WLAN communications of Wi-Fi direct, wi-Fi LAN, or Wi-Fi softAP. In some embodiments, the WLAN module may be integrated with the bluetooth communication module 203A. The electronic equipment can establish wireless communication connection with other electronic equipment through the wireless communication technology provided by the WLAN module, and perform data interaction with other electronic equipment based on the wireless communication connection.

The power switch 204 may be used to control the power supply to power the electronic device.

The microphone 205 may be used to convert sound signals into electrical signals. The microphone may include a pickup, a signal amplification circuit, and the like. The microphone can be transmitted to the vibrating diaphragm of the microphone by the vibration of sound, the magnet in the microphone is pushed to form a changed current, and the changed current is sent to the sound processing circuit for amplification processing. The microphones may include aluminum ribbon microphones, moving coil microphones, condenser microphones, electret microphones, and the like. In some embodiments, the wireless headset may include 1 or N microphones 205, N being a positive integer greater than 1.

The diaphragm 206 may be used to convert an electrical signal into an acoustic signal. When the current of the audio information passes through the earphone coil, the electric energy is converted into a magnetic field, and the change of the intensity of the magnetic field drives the change of the vibrating diaphragm, so that different sounds are emitted.

Optionally, the electronic device may further include a universal serial bus (universal serial bus, USB) interface, and the electronic device may establish a wired communication connection with other electronic devices through the USB interface, and transmit data information with each other through the wired communication connection.

It will be appreciated that the electronic device shown in fig. 6 is merely one implementation of the embodiments of the present application and is not meant to be limiting in any way. In practice, the electronic device may include more or less components than those illustrated, without limitation.

In the embodiment of the present application, the microphone 205 of the wireless earphone receives an audio signal sent by a gesture of a user; the microphone 205 converts sound transmitted through a medium such as air or water into an electrical signal; the input of the friction sound is transmitted from the outside thereof to the wireless earphone in the form of an audio signal, and the microphone 205 converts the friction sound input as the form of the audio signal into an electrical signal and detects the electrical signal; and the processor 201 may detect and analyze the type of friction sound in the audio information.

As an example, a microphone of the wireless headset may detect audio information within a preset range (e.g., without limitation, 15 cm) centered on the device in real time or periodically, audio information of a user gesture (e.g., without limitation, finger-sounding, clicking, double clicking, scraping, etc.) is converted into an electrical signal by the microphone and transmitted to the processor, and the processor may analyze the audio information by a machine learning algorithm, set the audio information as a new start signal, and establish a mapping relationship between the start signal and a control command according to an instruction of the user after analyzing the audio information from other audio information that has been input.

As yet another example, the wireless headset may play a gesture (e.g., without limitation, a finger, a click, a double click, a scratch, etc.) to the user through the diaphragm, the user may input a guiding voice according to which the user emits a friction sound of the gesture prompted by the guiding voice within a preset range (e.g., without limitation, 15 cm) centered on the device, the microphone may convert the friction sound into an electrical signal and transmit the electrical signal to the processor, and the processor may compare the input friction sound with a preset sound template through a machine learning algorithm, thereby adjusting the classification algorithm to achieve a better classification accuracy. Optionally, after the user completes the input of the gesture sound according to the guiding interface, the control command corresponding to the gesture may be adjusted, that is, the terminal device establishes a mapping relationship between the friction sound and a new control command (for example, but not limited to, cutting song, adjusting volume, opening a specific application program, etc.) according to the instruction of the user.

It can be understood that when the user performs audio information recording, namely, the microphone converts friction sound into a corresponding electric signal, the wireless earphone can prompt the user to repeatedly record in different directions of the terminal equipment, so that correction of a machine learning classification algorithm is facilitated, and recognition accuracy of the subsequent terminal equipment on the friction sound is improved.

Alternatively, the processor 201 may activate the microphone 205 alone to receive input of the friction sound to achieve the mode of operation of the corresponding command execution. For example, when music is being played on the wireless headset 200, if it is desired to control a function regarding music play through the input of friction sound, the microphone 205 may be activated to enable the input of friction sound to be received while music is being played. As yet another example, when in the sleep mode at the wireless headset 200, to enable input by friction sound, changing the mode of the wireless headset 200 to the active mode may cause the processor 201 to activate the microphone 205 in the sleep mode.

Wherein the processor 201 may determine the type of the friction sound by determining the body part generating the friction sound or the type of the body motion generating the friction sound according to the input of the friction sound, thereby executing a preset control command corresponding to the type of the friction sound.

The following describes an interaction method between the terminal device and the earphone in order to realize the control of the terminal device.

Fig. 7 is a schematic flowchart of a control method of a terminal device provided in an embodiment of the present application. Here, the following description will be given by taking the example in which the terminal device 100 is a mobile phone and the wireless headset 200 is a real wireless headset in fig. 1. As shown in fig. 7, the method may include S710, S720, S721, S730, S740, and S750.

It is to be understood that although the various steps of the methods are presented in a particular order in the embodiments of the present application, the order of the steps may be altered in different embodiments. Alternatively, in some embodiments, one or more steps shown in sequence in this specification may be performed simultaneously. Alternatively, in some embodiments, only a portion of the steps of fig. 7 may be performed, e.g., only S720 and S721, or only S710, S720, S730, S740, and S750 may be performed.

S710, sending a sound recording command to the earphone in response to a first operation of the user on the mobile phone.

As an example, after the mobile phone establishes a communication connection with the headset, the user may implement sending of the sound recording command through an application program of the mobile phone.

For example, as shown in fig. 8, the mobile phone 800 includes an application 810, the user clicks the application 810, that is, the mobile phone detects that the user operates to start the application, the mobile phone displays an interface 820, the interface 820 includes a sound recording option, after the mobile phone detects that the user clicks the sound recording option, the mobile phone displays an interface 831, and when the mobile phone detects that the user clicks a start icon therein, the mobile phone sends a sound recording command to the earphone.

It will be appreciated that the operation of sending the sound recording command is performed on the mobile phone is merely an example, and the present application is not limited thereto.

S720, the earphone receives the sound recording command and executes the operation of collecting the audio information.

Alternatively, when the earphone starts to collect the audio information, the vibration frequency information corresponding to the audio information may be returned in real time, so as to be displayed in the interface 832 as shown in fig. 8.

Optionally, when the earphone performs the operation of collecting the audio information, or after performing the operation, step S721 may also be performed to detect the feature information in the audio information, and if the feature information is not recorded, record the feature information as a start signal, and establish a mapping relationship with the corresponding command.

The characteristic information in the audio information may be information for determining a type of a friction sound, which may be a sound generated by a finger scraping another finger, or a sound generated by a finger tip tapping another finger tip, by detecting a friction sound generated by the different body part or the different type of body movement, as an example.

As another example, the type of friction sound may also be determined based on the different orientations of the friction sound at the headset or cell phone. For example, as shown in fig. 9, the earphone 200a detects a first friction sound (e.g., without limitation, a finger ring) in the side direction, and the friction sound may be set to a first friction type; the earphone 200b detects a second friction sound (e.g., without limitation, a finger ring) in the side direction, which may be set to a second friction type; the first friction sound and the second friction sound may be the same friction sound, and since the friction sounds are recorded or detected on different sides of the earphone, the friction sounds may be set to two types of friction sound and respectively correspond to different execution commands. It will be appreciated that this orientation may also be used to indicate the range of friction sounds compared to the recorded range of the earphone, e.g. the earphone may detect friction sounds within 15 cm of it, or it may be set to a different type of friction sound depending on the friction sounds emitted by the user at different distance locations within 15 cm of the earphone.

In the embodiment of the application, in the process of detecting the collected audio information, a machine learning algorithm may be used to analyze the friction sound in the audio information and determine the type of the friction sound or set a new start signal. As an example, the processor 110 shown in fig. 4 or the processor 201 shown in fig. 6 may recognize the vibration frequency, tone characteristic, azimuth, etc. of the finger using a machine learning algorithm for voice recognition, and compare the recognized vibration frequency, tone characteristic, and azimuth with a pre-stored template, thereby determining the type of friction sound; when the identified vibration frequency, tone characteristic, and orientation do not exist in the pre-stored template, it may be set as a new activation signal (i.e., a new template of the type of friction sound); meanwhile, in order to enhance the classifying capability of the machine learning model, the same type of friction sound can be recorded in multiple times and multiple directions.

It will be appreciated that when the type of friction sound of the input audio information exists in the template, the user may be prompted that the type of friction sound already exists or be prompted to establish a new mapping command for the type of friction sound.

And S730, sending a sound recording termination command to the earphone in response to the second operation of the user on the mobile phone.

As an example, after the mobile phone sends the sound recording command to the earphone, the user may also implement terminating the sending of the sound recording command through an application program of the mobile phone.

For example, as shown in fig. 8, when the earphone performs recording of audio information, an interface 832 may be displayed, including a stop icon, and when the mobile phone detects that the user clicks the stop icon, a command to terminate sound recording is sent to the earphone.

S740, the earphone receives the command for stopping sound recording and sends the collected audio information to the mobile phone.

Optionally, when the earphone collects audio information, the collected audio information can be sent to the mobile phone in real time.

Optionally, after the earphone receives the command for terminating sound recording, the mobile phone may display a corresponding interface indicating that recording is completed to prompt the user that recording of the audio information is currently completed. For example, as shown in fig. 10, the mobile phone 800 includes an interface 1010, and the interface 1010 displays that the recording of sound is completed, so as to prompt the user that the recording of the audio information is completed.

S750, the mobile phone receives the audio information sent by the earphone, detects the characteristic information in the audio information, if the characteristic information is not recorded, the characteristic information is recorded as starting information, and a mapping relation is established with the corresponding command.

The step of detecting the audio information by the mobile phone can refer to step S721, which is not described herein.

As an example, after the mobile phone receives the recorded audio information, a mapping relationship can be established between the feature information in the audio information and a preset command, or a mapping relationship can be established between a command selected by a user and the feature information in the audio information by detecting the operation of the user.

For example, after the mobile phone sends a command to the earphone to terminate sound recording, the user interface of the mobile phone 800 shown in fig. 10 jumps from the interface 1010 to the interface 1030, and the user clicks a selection corresponding command option in the interface 1030, that is, after the mobile phone detects that the user clicks the operation of the selection corresponding command option, the interface 1040 is displayed, which includes turning up the volume, turning down the volume, pausing, playing, and the next, where the user can select one or more of them, and the mobile phone corresponds the audio information to the option selected by the user by detecting the operation of the user.

Optionally, after the audio information recording is completed on the display interface 1010, a user interface 1020 may be displayed to prompt the user to edit the name of the gesture corresponding to the audio information that has just been recorded. The above manner facilitates the user to recognize the gesture and the control command corresponding thereto.

It can be understood that the name editing of the gesture for inputting the audio information may be performed before the audio information is recorded, after the audio information is recorded, or after the mapping relationship between the audio information and the corresponding control command is established, which is not limited herein.

In summary, by recording audio information and analyzing and detecting the type of friction sound therein, a mapping relation is established between the audio information and a corresponding command, and a foundation is provided for executing corresponding command control by using the friction sound.

Fig. 11 is a schematic flowchart of another control method of a terminal device provided in an embodiment of the present application. Here, the following description will be given by taking the example in which the terminal device 100 is a mobile phone and the wireless headset 200 is a real wireless headset in fig. 1. As shown in fig. 11, the method may include S1110, S1120, S1121, S1130, and S1140.

It is to be understood that although the various steps of the methods are presented in a particular order in the embodiments of the present application, the order of the steps may be altered in different embodiments. Alternatively, in some embodiments, one or more steps shown in sequence in this specification may be performed simultaneously. Alternatively, in some embodiments, only a portion of the steps of fig. 11 may be performed, e.g., only S1120 and S1121, or only S1110, S1120, S1130, and S1140 may be performed.

S1110, sending a sound recording command to the earphone in response to the operation of the user on the mobile phone.

The step of sending the sound recording command may refer to step S710, which is not described herein.

S1120, the earphone receives the sound recording command and executes the operation of collecting the audio information.

The step of performing the operation of collecting the audio information is referred to the above step S720, and will not be described herein.

Optionally, when the earphone performs the operation of collecting the audio information, or after performing the operation, step S1121 may also be performed to detect the feature information in the audio information, and if the feature information is not recorded, record the feature information as a start signal, and establish a mapping relationship with the corresponding command.

The step of detecting the mapping relationship between the feature information and the suggestion in the audio information may refer to step S721, which is not described herein.

S1130, the earphone terminates the sound recording command and sends the collected audio information to the mobile phone.

As an example, after receiving the sound recording command, the earphone starts to perform sound recording operation for a preset duration, and when the duration of the recorded audio information reaches the preset duration, the earphone stops the sound recording command and sends the collected audio information to the mobile phone.

For example, as shown in fig. 12, when the earphone 1230 starts to perform the recording operation of the sound, the frequency information of the audio information generated by the current user may be transmitted to the mobile phone 1200 in real time, the mobile phone displays the frequency chart of the audio information in the interface 1210, and displays an icon in the sound recording to prompt the user to record, and when the recording duration of the audio information reaches a preset duration (for example, but not limited to, 10 seconds), the interface 1210 may switch to the interface 1220, and displays an icon that the sound recording is completed to prompt the user to complete the recording of the audio information.

S1140, the mobile phone receives the audio information sent by the earphone, detects the characteristic information in the audio information, if the characteristic information is not recorded, records the characteristic information as starting information, and establishes a mapping relation with the corresponding command.

The step of detecting the mapping relationship between the feature information and the suggestion in the audio information may refer to step S750, which is not described herein.

Fig. 13 is a schematic flowchart of a control method of a terminal device provided in an embodiment of the present application. Here, the following description will be given by taking the example in which the terminal device 100 is a mobile phone and the wireless headset 200 is a real wireless headset in fig. 1. As shown in fig. 13, the method may include S1310, S1320, S1321, S1330, S1340, and S1350.

It is to be understood that although the various steps of the methods are presented in a particular order in the embodiments of the present application, the order of the steps may be altered in different embodiments. Alternatively, in some embodiments, one or more steps shown in sequence in this specification may be performed simultaneously. Alternatively, in some embodiments, only a portion of the steps of fig. 13 may be performed, e.g., only S1310, S1321, and S1350, or only S1310, S1320, S1330, S1340, and S1350 may be performed.

S1310, periodically collecting audio information by the earphone.

Alternatively, the earphone may collect audio information of the surrounding environment in real time through the microphone.

Optionally, the mobile phone can also collect the audio information of the surrounding environment in real time or periodically through the microphone, or collect the audio information of the environment collected by the earphone in an auxiliary manner, so as to improve the accuracy or the integrity of the collected audio information, and further improve the accuracy of the subsequent gesture and voice recognition.

As an example, the headset may collect audio information of the surrounding environment in real time or periodically in an operating state. For example, when the headset is playing music or playing voice calls, the microphone of the headset may collect sounds of the surrounding environment in real time.

As another example, headphones may collect audio information of the surrounding environment in real-time or periodically in a dormant state. For example, when the headset is not performing any operation, the microphone of the headset may collect sounds of the surrounding environment in real time.

Optionally, when the earphone is in the sleep state, in-ear detection can be performed, that is, whether the current earphone is in the wearing state is detected, if so, the audio information of the surrounding environment is collected in real time or periodically, otherwise, the collection of the audio information of the surrounding environment is stopped. By the method, unnecessary awakening of the earphone can be reduced, and the battery endurance of the earphone is effectively improved.

S1320, the earphone sends the acquired audio information to the mobile phone.

Alternatively, the earphone can transmit the audio information to the mobile phone in real time while collecting the audio information.

Optionally, when the earphone performs the operation of periodically collecting the audio information, or after the operation of collecting the audio information is completed, step S1321 may also be performed, and the earphone may match the corresponding command with the mapping relationship according to the audio information.

The matching process of the mapping relation of the earphone in executing the command can utilize the trained machine learning model, classify the audio input information collected by the earphone through the model, identify the type of friction sound therein, and determine the control command corresponding to the type of friction sound according to the preset mapping relation.

S1330, the mobile phone receives the audio information and matches the corresponding command with the mapping relation according to the audio information.

The corresponding command matching the mapping relationship may refer to the aforementioned step S1321, and will not be described herein.

S1340, the mobile phone sends the command to the headset.

S1350, the headset receives the command and executes it.

As an example, the earphone receives a command corresponding to a friction sound generated by a user and performs an operation of the command. For example, as shown in fig. 9, the user uses a finger gesture at the earphone 200a, and the earphone receives a volume-down command corresponding to the finger gesture to perform an operation of reducing the horn power. For another example, the user uses a finger-sounding gesture at the earphone 200b, and the earphone receives a volume-increasing command corresponding to the finger-sounding gesture to perform an operation of increasing the power of the speaker.

Optionally, before the corresponding command with the mapping relation is matched, the judgment of the specific scene can be executed, and when the mobile phone or the earphone is in the preset scene, the corresponding operation in the scene is executed. For example, when the earphone or the mobile phone detects the sound of the finger gesture of the user, the earphone or the mobile phone performs switching to play the next song; when the mobile phone is in the do-not-disturb mode, under the current scene, when the earphone or the mobile phone detects the sound of the finger-ringing gesture of the user, the playing of the current music is stopped.

In one possible implementation manner, when the microphone of the earphone detects the audio information reaching the preset acoustic intensity detection threshold, the coarse classifier is activated, and the earphone judges whether the audio information contains the sound waveform or frequency sent by the identifiable gesture through a machine learning coarse classification algorithm; if so, further activating the fine classifier, otherwise, stopping execution.

Further, if the audio information includes a sound waveform or frequency generated by the recognizable gesture, the earphone further determines the type of the friction sound through a machine learning fine classification algorithm, and executes a command matched according to the determined type of the friction sound.

Optionally, when the command corresponding to the collected friction sound is executed on the mobile phone, as shown in fig. 14, fig. 14 is a schematic flowchart of a control method of the terminal device provided in the embodiment of the present application. Here, the following description will be given by taking the example in which the terminal device 100 is a mobile phone and the wireless headset 200 is a real wireless headset in fig. 1. As shown in fig. 14, the method may include S1410, S1420, S1421, S1430, S1431, and S1440.

It is to be understood that although the various steps of the methods are presented in a particular order in the embodiments of the present application, the order of the steps may be altered in different embodiments. Alternatively, in some embodiments, one or more steps shown in sequence in this specification may be performed simultaneously. Alternatively, in some embodiments, only a portion of the steps of fig. 14 may be performed, e.g., only S1410, S1421, S1431, and S1440, or only S1410, S1420, S1430, and S1440 may be performed.

S1410, the earphone periodically collects audio information.

The step of the earphone collecting the audio information may refer to the aforementioned step S1310, and will not be described herein.

S1420, the earphone sends the collected audio information to the mobile phone.

The step of transmitting the audio information by the earphone may refer to the aforementioned step S1320, and will not be described herein.

Optionally, when the earphone performs the operation of periodically collecting the audio information, or after the operation of collecting the audio information is completed, step S1421 may be further performed, and the earphone may match the corresponding command with the mapping relationship according to the audio information; reference is made to the aforementioned step S1321, and a detailed description thereof is omitted here.

After step S1421 is performed, after determining a command corresponding to the friction sound in the collected audio information, step S1431 is performed, and the command is sent to the mobile phone.

S1430, the mobile phone receives the audio information and matches the corresponding command with the mapping relation according to the audio information.

S1440, the mobile phone receives the command and executes the command.

As an example, the mobile phone receives a command corresponding to a friction sound generated by a user and performs an operation of the command. For example, when the mobile phone receives a command corresponding to a user using a finger-sounding gesture, if the command corresponding to the finger-sounding gesture is to switch songs, the mobile phone performs an operation of switching a next song in an application program that is playing music.

In summary, according to the control method of the terminal device provided by the application, the user can input the friction sound information of the gesture by himself, set the friction sound information as the starting signal of the corresponding control command, and when the user generates the audio information containing the friction sound again, the earphone detects the audio information through the microphone and recognizes the type of the friction sound therein through the machine learning algorithm, and executes the command corresponding to the type. The user can define the control command corresponding to the sound sent by the gesture by himself, the structural complexity of the wearable device is reduced, the learning cost of the user is reduced, and the convenience of man-machine interaction between the earphone and the user is improved.

Besides, besides the definition and input of the names of the gesture sounds by the user, the gesture and the corresponding friction sounds can be preset in the terminal device, the user can send out the friction sounds according to the preset gesture, and the terminal device can correct the preset classification algorithm according to the audio information of the friction sounds.

Fig. 15 is a schematic flowchart of a gesture sound input method according to an embodiment of the present application, as shown in fig. 15. Here, the following description will be given by taking the example in which the terminal device 100 is a mobile phone and the wireless headset 200 is a real wireless headset in fig. 1. As shown in fig. 15, the method may include S1511, S1510, S1520, S1530, S1540, and S1550.

It is to be understood that although the various steps of the methods are presented in a particular order in the embodiments of the present application, the order of the steps may be altered in different embodiments. Alternatively, in some embodiments, one or more steps shown in sequence in this specification may be performed simultaneously. Alternatively, in some embodiments, only a portion of the steps of fig. 15 may be performed, e.g., only S1510, S1520, S1530, S1540, and S1550 may be performed.

S1510, sending a sound recording command to the earphone in response to the first operation of the user on the mobile phone.

The step of sending the sound recording command to the earphone may refer to the aforementioned step S710, and will not be described herein.

Optionally, before the user performs the operation and the mobile phone sends the command according to the operation, the mobile phone may perform step S1511 to display a gesture guidance interface, where one or more gestures and templates corresponding to the gestures may be stored in the mobile phone in advance. The user can make a friction sound by selecting one of the gestures according to the guide interface of the gesture sound displayed by the mobile phone.

S1520, the earphone receives the sound recording command, and performs an operation of collecting audio information.

The operation step of collecting the audio information may refer to the aforementioned step S720, and will not be described herein.

S1530, in response to the second operation of the user on the mobile phone, sending a sound recording termination command to the earphone.

The step of sending the command to terminate sound recording to the earphone may refer to step S730, and will not be described herein.

S1540, the earphone receives the sound recording termination command and sends the collected audio information to the mobile phone.

The step of sending the collected audio information to the mobile phone may refer to the aforementioned step S740, which is not described herein.

S1550, the mobile phone receives the audio information sent by the earphone, detects the characteristic information in the audio information, and corrects a pre-stored classification algorithm according to the characteristic information.

In one possible implementation manner, a trained machine learning model may be preset in the mobile phone, the user sends out a friction sound corresponding to the gesture according to the gesture guiding interface, the mobile phone detects characteristic information of the friction sound, and the characteristic information is compared with characteristic information preset in the machine learning model, so that a classification algorithm in the model is corrected and optimized.

Fig. 16 is a schematic structural diagram of a control device of a terminal device according to an embodiment of the present application. As shown in fig. 16, the control apparatus 1600 of the terminal device includes: a receiving module 1610 and a processing module 1620. As an example, the identification means may be the terminal device 100 in fig. 1, may be a chip applied to the terminal device 100 in fig. 1, or may be a computer program product applied to the terminal device 100 in fig. 1. Wherein:

The control means 1600 of the terminal device may be used to implement the method in the embodiment shown in fig. 7. As an example, the receiving module 1610 may be used to implement S710 and S740 in the embodiment shown in fig. 7; the processing module 1620 may be configured to implement S750 in the embodiment illustrated in fig. 7;

further, the control device 1600 of the terminal device may also be used to implement the methods in the embodiments shown in fig. 11, 13 and 14. It can be understood that the control device of the terminal device provided in the embodiment of the present application may be used to implement the technical scheme of any of the method embodiments described above, and the implementation principle and technical effects are similar, and specific reference may be made to the above method embodiments, which are not described herein.

Fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 17, the electronic device provided in the present application may include: a memory 1710, and a processor 1720.

The memory 1710 is used to store a computer application for implementing the method in any of the embodiments shown in fig. 7, 11, 13, and 14.

The processor 1720 is configured to execute a computer application in the memory 1710 that when executed by the processor 1720 implements a method as in any one of the embodiments shown in fig. 7, 11, 13, and 14.

Alternatively, the memory 1710 may be separate or integrated with the processor 1720.

The implementation principle and technical effects of the electronic device provided in this embodiment may be referred to the foregoing embodiments, and will not be described herein again.

The present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the control method of a terminal device as provided in any of the foregoing embodiments.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules may be combined or integrated into another system, or some features may be omitted or not performed.

The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or processor to perform some of the steps of the methods of the various embodiments of the invention.

It should be appreciated that the processor may be a central processing unit (Central Processing Unit, CPU for short), other general purpose processors, digital signal processor (Digital Signal Processor, DSP for short), application specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.

The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile memory NVM, such as at least one magnetic disk memory, and may also be a U-disk, a removable hard disk, a read-only memory, a magnetic disk or optical disk, etc.

The storage medium may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application specific integrated circuit (Application Specific Integrated Circuits, ASIC for short). It is also possible that the processor and the storage medium reside as discrete components in an electronic device or a master device.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method of the embodiments of the present invention.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A control method of a terminal device, the method comprising:

receiving a first instruction input by a user, wherein the first instruction is used for setting the name of a first gesture customized by the user;

recording first audio data corresponding to the first gesture through a first wearable device, wherein the first wearable device comprises a microphone, and communication connection is established between the first wearable device and the terminal device;

establishing a first mapping relation among a first audio feature of the first audio data, a name of the first gesture and a first control instruction in the terminal equipment;

receiving second audio data from the first wearable device;

and controlling the terminal equipment based on the second audio data and the first mapping relation.

2. The method of claim 1, wherein recording, by the first wearable device, the first audio data corresponding to the first gesture, comprises:

Receiving a second instruction input by a user, wherein the second instruction is used for indicating the terminal equipment to start recording audio data corresponding to a gesture;

sending first indication information to the first wearable device, wherein the first indication information is used for indicating the first wearable device to start a microphone;

first audio data is received from the first wearable device.

3. The method of claim 2, wherein after the sending the first indication information to the first wearable device, before the receiving the first audio data from the first wearable device, the method further comprises:

and sending second indication information to the first wearable device, wherein the second indication information is used for indicating the first wearable device to close a microphone and sending the collected audio data to the terminal device.

4. The method of claim 3, wherein after the sending the first indication information to the first wearable device, before the sending the second indication information to the first wearable device, the method further comprises:

outputting second prompt information, wherein the second prompt information is used for prompting a user to select audio data corresponding to the gesture of stopping recording;

And receiving a third instruction input by a user, wherein the third instruction is used for selecting the audio data corresponding to the recording stopping gesture.

5. The method of claim 2, wherein prior to the receiving the first audio data from the first wearable device, the method further comprises:

and sending third indication information to the first wearable device, wherein the third indication information is used for indicating the first wearable device to turn off a microphone after the first microphone is started for a first time period and sending collected audio data to the terminal device.

6. The method according to any one of claims 2 to 5, further comprising:

and outputting third prompt information, wherein the third prompt information is used for prompting a user to make the first gesture around the first wearable device.

7. The method of any one of claims 1 to 6, wherein prior to the receiving the first instruction input by the user, the method further comprises:

receiving a fourth instruction input by a user, wherein the fourth instruction is used for requesting to customize a gesture associated with a control instruction in the terminal equipment;

and outputting fourth prompt information, wherein the fourth prompt information is used for prompting a user to input the name of the custom gesture.

8. The method of claim 7, wherein prior to receiving the fourth instruction input by the user, the method further comprises:

receiving a fifth instruction input by a user, wherein the fifth instruction is used for requesting a mapping relation between a setting gesture and a control instruction in the terminal equipment;

and outputting fifth prompt information, wherein the fifth prompt information is used for a user to select a gesture which is customized and is used for being associated with a control instruction in the terminal equipment.

9. The method according to any one of claims 1 to 8, wherein before the establishing a first mapping relation between the first audio feature of the first audio data, the name of the first gesture and the first control instruction in the terminal device, the method further comprises:

extracting first audio features in the first audio data;

and determining that the gesture audio features for associating the control instruction in the terminal equipment do not contain the first audio features.

10. The method of any of claims 1-9, wherein the first wearable device comprises a plurality of microphones and the first audio data is recorded for a first microphone of the plurality of microphones;

The establishing a first mapping relationship among the first audio feature of the first audio data, the name of the first gesture and the first control instruction in the terminal device includes:

establishing a first mapping relation among the first microphone, a first audio feature of the first audio data, a name of the first gesture and a first control instruction in the terminal equipment;

the controlling the terminal device based on the second audio data and the first mapping relation includes:

and controlling the terminal equipment based on the second audio data, the microphone recorded to the second audio data and the first mapping relation.

11. A control apparatus for a terminal device, comprising:

and a receiving module: the method comprises the steps of receiving a first instruction input by a user, wherein the first instruction is used for setting the name of a first gesture customized by the user; receiving second audio data from the first wearable device;

the processing module is used for: the first wearable device is used for recording first audio data corresponding to the first gesture, the first wearable device comprises a microphone, and communication connection is established between the first wearable device and the terminal device; establishing a first mapping relation among a first audio feature of the first audio data, a name of the first gesture and a first control instruction in the terminal equipment; and controlling the terminal equipment based on the second audio data and the first mapping relation.

12. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1-10.

13. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1-10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-10.