WO2022111579A1

WO2022111579A1 - Voice wakeup method and electronic device

Info

Publication number: WO2022111579A1
Application number: PCT/CN2021/133119
Authority: WO
Inventors: 江昱成; 赵安; 林龙
Original assignee: 华为技术有限公司
Priority date: 2020-11-27
Filing date: 2021-11-25
Publication date: 2022-06-02
Also published as: CN114566171A

Abstract

A voice wakeup method and an electronic device, relating to the field of terminal artificial intelligence. The method comprises: by utilizing ambient sound acquired by each device, on one hand, relative positions of a user and the multiple devices in a space can be positioned to construct a position map; and on the other hand, a face orientation of the user can be acquired by a master device having an image acquisition module in the multiple devices. Thus, a device that the user wants to wake up can be determined by combining the position map and the face orientation of the user acquired by the master device. The method facilitates improving the accuracy of device wakeup in a multi-device environment, and the application effect is relatively good.

Description

A kind of voice wake-up method and electronic device

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Chinese patent application with the application number 202011362525.1 and the application title "A voice wake-up method and electronic device" filed with the China Patent Office on November 27, 2020, the entire contents of which are incorporated into this application by reference middle.

technical field

The present application relates to the technical field of terminals, and in particular, to a voice wake-up method and an electronic device.

Background technique

Currently, the user can wake up the electronic device by speaking a wake-up word, thereby realizing the interaction between the user and the electronic device. Usually, the wake-up word is preset in the electronic device by the user, or the wake-up word is set before the electronic device leaves the factory. In a multi-device scenario (such as a smart home scenario), the user may set the same wake-up word for multiple devices in order to facilitate memory. For example, the user sets the wake-up word for smart screens, smart speakers, and smart switches to be "Xiaoyi". Little Art". As shown in Figure 1, it is assumed that the user only wants to wake up the smart screen, but when the user says "Xiaoyi Xiaoyi", the smart screen and the speaker are both awakened, and both feedback the voice response of "I am" to the user. Trouble the user and affect the user experience.

SUMMARY OF THE INVENTION

The present application provides a voice wake-up method and an electronic device, which help to improve the accuracy of voice wake-up of an electronic device in a multi-device scenario, thereby improving user experience.

In a first aspect, an embodiment of the present application provides a voice wake-up method, which can be applied to a first electronic device, and relates to the field of terminal artificial intelligence (artificial intelligence, AI). The method includes:

The first electronic device receives the user's voice wake-up instruction; the first electronic device also acquires the user image, and detects the user's face orientation; The user's position and the orientation of the user's face, and the target device to which the user's face is facing is determined from the first electronic device and at least one second electronic device; finally, the first electronic device instructs the target device to wake up in response to the voice instruction.

The first electronic device may have an image acquisition function, and the first electronic device acquires the user image from the image acquisition module, or the first electronic device may not have the image acquisition function, and the first electronic device acquires the user image from the second electronic device.

In the embodiment of the present application, the first electronic device can determine the device that the user wants to wake up by using the relative positions of the first electronic device and at least one second electronic device and the face orientation of the user collected by the device. This method helps In order to improve the accuracy of device wake-up in multi-device scenarios, the application effect is relatively good.

In a possible design, when the second electronic device determines that the number of candidate devices to which the user's face faces is greater than or equal to two, the first electronic device needs to determine the relationship between the user and the at least two candidate devices. relative distance; then determine the priority of the candidate device according to the relative distance, wherein the smaller the relative distance, the smaller the priority of the candidate device; finally determine the candidate device corresponding to the highest priority as the target device.

In the embodiment of the present application, on the basis of directional wake-up, it helps to improve the accuracy of nearby wake-up in a multi-device scenario.

In another possible design, when the second electronic device determines that the number of candidate devices to which the user's face faces is greater than or equal to two, the first electronic device needs to determine the user and the at least two candidate devices The relative distance; the candidate device corresponding to the minimum relative distance is finally determined as the target device.

In a possible design, the first electronic device may acquire information of the first audio of the first electronic device, and acquire information of the second audio from at least one second electronic device; then according to the information of the first audio and the second audio audio information to determine the user location.

In this embodiment of the present application, the sound collected by the multi-microphone array of the electronic device can effectively determine the user's position, so as to ensure the accuracy of the user's position positioning result.

In a possible design, the first electronic device includes a first microphone and a second microphone; the information of the first audio includes: a first arrival time when the voice wake-up command reaches the first microphone, and a first time when the voice wake-up command reaches the second microphone. The second arrival time, and the first phase when the voice wake-up command reaches the first microphone, and the second phase when the voice wake-up command reaches the second microphone; at least one second electronic device includes a third microphone and a fourth microphone; the information of the second audio includes : The voice wake-up command reaches the third arrival time of the third microphone, the voice wake-up command reaches the fourth arrival time of the fourth microphone, and the voice wake-up command reaches the third phase of the third microphone, and the voice wake-up command reaches the fourth microphone of the fourth microphone. phase.

In a possible design, the first electronic device may determine the user's location according to the information of the first audio and the information of the second audio, which specifically includes the following steps:

Determine the relative distance between the user and the first electronic device according to the time difference between the first arrival moment and the second arrival moment, and determine the relative distance between the user and the first electronic device according to the phase difference between the first phase and the second phase Azimuth; determining the relative distance between the user and the at least one second electronic device according to the time difference between the second arrival moment and the third arrival moment; and determining the relative distance between the user and the at least one second electronic device according to the phase difference between the third phase and the fourth phase Azimuth of at least one second electronic device; based on the relative distance between the user and the first electronic device, the user's azimuth relative to the first electronic device and the relative distance between the user and the at least one second electronic device, and the user relative to the at least one The azimuth of the second electronic device determines the user's location.

In a possible design, the method further includes: the first electronic device acquiring historical audio information from the first electronic device and the at least one second electronic device;

From the historical audio information, the first electronic device obtains the arrival times and phases of the voice wake-up commands issued by the user N times to different electronic devices, where N is a positive integer; Determine the relative azimuth and distance difference corresponding to the voice wake-up command issued by the user N times;

The first electronic device uses the relative azimuth angle and the distance difference corresponding to the voice wake-up commands issued by the user N times as the observed values, and establishes an objective function; the first electronic device solves the objective function by an exhaustive search method to obtain the first electronic device. The relative position of an electronic device and at least one second electronic device.

In this embodiment of the present application, the first electronic device may locate the relative positions of multiple devices in space according to the above method, and construct a location map including the relative positions of the devices. The device can also reversely deduce the relative positions of multiple pickup devices through multiple voice recognition of the user's voice even in an interference environment. With the increase of voice wake-up messages, the positioning between devices will become more accurate.

In a possible design, the first electronic device and the at least one second electronic device are connected to the same local area network, or the first electronic device and the at least one second electronic device are pre-bound with the same user account , or the first electronic device and the at least one second electronic device are bound to different user accounts, and the different user accounts establish a binding relationship.

In a second aspect, an embodiment of the present application provides a voice wake-up method, the method can be applied to a second electronic device, and the method includes:

The second electronic device collects the sound of the surrounding environment and converts it into the second audio, then the second electronic device sends the second audio to the first electronic device, and when the second electronic device detects the wake-up word in the second audio, it sends the second audio to the second electronic device. The first electronic device sends a wake-up message, and the first electronic device can determine the user's location according to the information of the first audio and the information of the second audio of the first electronic device, and can determine the user's position according to the information of the first electronic device and the at least one second electronic device. Relative position, user position and the face orientation of the user, when it is determined that the target device is the second electronic device, send a wake-up response to the second electronic device, and after the second electronic device receives the wake-up response from the first electronic device , in response to the user's voice wake-up command.

In another possible situation, if the first electronic device determines that the target device is not the second electronic device according to the relative positions of the first electronic device and at least one second electronic device, the user's position, and the user's face orientation, The first electronic device does not send a wake-up response to the second electronic device, or sends a wake-up prohibition response to the second electronic device, and the second electronic device does not respond to the user's voice wake-up instruction.

In a third aspect, the present application provides a voice wake-up system, which includes a first electronic device and at least one second electronic device. The first electronic device can implement the method of any possible implementation manner of the first aspect, and at least two second electronic devices can implement the method of any possible implementation manner of the second aspect.

In a fourth aspect, an electronic device provided by an embodiment of the present application includes: one or more processors and a memory, wherein program instructions are stored in the memory, and when the program instructions are executed by the device, the above aspects of the embodiments of the present application are implemented and any possible design method involved in the various aspects.

In a fifth aspect, an embodiment of the present application provides a chip system, wherein the chip system is coupled with a memory in an electronic device, so that the chip system invokes program instructions stored in the memory when running to implement the embodiments of the present application The various aspects described above, as well as the methods of any possible designs involved in the various aspects.

In a sixth aspect, a computer-readable storage medium according to an embodiment of the present application stores program instructions, and when the program instructions are executed on an electronic device, the device enables the device to perform the above aspects of the embodiments of the present application. and any possible design method involved in the various aspects.

In a seventh aspect, a computer program product of an embodiment of the present application, when the computer program product runs on an electronic device, enables the electronic device to execute and implement the above-mentioned aspects and any possibility involved in the various aspects of the embodiments of the present application. method of design.

In addition, for the technical effect brought by any of the possible design methods in the fourth aspect to the seventh aspect, reference may be made to the technical effect brought by different design methods in the related method section, which will not be repeated here.

Description of drawings

1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

2 is a schematic structural diagram of a mobile phone according to an embodiment of the present application;

FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 4 is an interactive schematic diagram of a voice wake-up method provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a wake-up method provided by an embodiment of the present application;

FIG. 6A is a schematic diagram of a user location positioning method according to an embodiment of the present application;

6B to 6D are schematic diagrams of another application scenario provided by the embodiment of the present application;

FIG. 7 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 8A is a schematic diagram of a device location positioning method according to an embodiment of the present application;

8B is a schematic diagram of a wake-up speech analysis method provided by an embodiment of the present application;

FIG. 8C is a schematic diagram of a device map provided by an embodiment of the present application;

FIG. 9 is an interactive schematic diagram of a device location positioning method provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a group of perception capability layers according to an embodiment of the present application;

11 is a schematic structural diagram of a device according to an embodiment of the application;

FIG. 12 is a schematic structural diagram of another device according to an embodiment of the present application.

Detailed ways

It should be understood that, unless otherwise specified in this application, "/" means or means, for example, A/B can mean A or B; "and/or" in this application is only an association relationship that describes an associated object , which means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more, and "plurality" means two or more.

In this application, "exemplary," "in some embodiments," "in other embodiments," and the like are used to mean by way of example, illustration, or illustration. Any embodiment or design described in this application as "exemplary" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word example is intended to present a concept in a concrete way.

In addition, terms such as "first" and "second" involved in this application are only used for the purpose of distinguishing and describing, and should not be interpreted as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Not to be construed as indicating or implying order.

The electronic device in the embodiment of the present application is an electronic device with a voice wake-up function, that is, a user can wake up the electronic device by voice. Specifically, the user wakes up the electronic device by speaking the wake-up word. The wake-up word may be preset in the electronic device by the user according to his own needs, or may be set by the electronic device before leaving the factory, and the setting method of the wake-up word is not limited in this embodiment of the present application. It should be noted that, in this embodiment of the present application, the user who wakes up the electronic device may be arbitrary or specific. For example, the specific user may be a user who pre-stores the sound of emitting the wake-up word in the electronic device, such as the owner of the device.

Currently, electronic devices trigger device wake-up by detecting whether a wake word is included in the audio. Specifically, when the wake-up word is included in the audio, the electronic device is awakened, otherwise the electronic device is not awakened. After the electronic device is awakened, the user can interact with the electronic device through voice. For example, the wake-up word is "Xiaoyi Xiaoyi", and when the electronic device detects that "Xiaoyi Xiaoyi" is included in the audio, the electronic device is woken up. The electronic device acquires audio by collecting or receiving ambient sound through a multi-microphone array on the device. However, in a multi-device scenario (such as a smart home scenario), the voice including the "wake-up word" spoken by the user may be received or captured by multiple electronic devices, causing two or more electronic devices to wake up , which brings confusion to the user's voice interaction process and affects the user experience.

At present, in order to solve the problem of devices being woken up by mistake in multi-device scenarios, the priority of each device is usually specified manually. Assuming that the priority of the smart screen in Figure 1 is higher than the priority of the smart speaker, when both the smart screen and the smart speaker collect the "Xiaoyi Xiaoyi" spoken by the user, only the smart screen is awakened. Although this method can restrict the wake-up of multiple devices by setting rules, it is not intelligent enough because the rules need to be manually set in advance, and the user can only actively modify the setting rules manually to adjust the wake-up according to actual needs. The priority of the device, so there is a problem of poor flexibility.

In addition, there is also a related technology that triggers the device facing the face to wake up by monitoring the face orientation, although this method can improve the flexibility and accuracy of the human-computer interaction method. However, because most devices do not have the image acquisition function, the application effect is not good. For example, devices such as smart speakers and smart voice-activated switches in smart home scenarios are subject to cost constraints, and generally do not integrate image acquisition modules, so the method of directional wake-up based on face orientation cannot be applied to such devices.

Therefore, when the existing voice wake-up method is applied to a multi-device scenario, it still cannot effectively solve the problem that multiple devices are woken up at the same time. In view of this, the embodiment of the present application provides a voice wake-up method. On the one hand, the relative positions of the user and multiple devices in space can be located, and a location map can be constructed; In this way, by combining the location map and the user's face orientation collected by the main device, the device that the user wants to wake up can be determined. This method helps to improve the accuracy of device wake-up in multi-device scenarios. , the application effect is relatively good.

Hereinafter, an electronic device is taken as an example, and FIG. 2 shows a schematic structural diagram of the electronic device 200 .

The voice wake-up method provided by the embodiment of the present application can be applied to an electronic device. In some embodiments, the electronic device may be a portable terminal including functions such as a personal digital assistant and/or a music player, such as a mobile phone, a tablet computer, a wearable device (such as a smart watch) with wireless communication capabilities, a vehicle-mounted device, etc. . Exemplary embodiments of the portable terminal include, but are not limited to, the Harmony operating system

Or portable terminals of other operating systems. The aforementioned portable terminal may also be, for example, a laptop computer (Laptop) having a touch-sensitive surface (eg, a touch panel). It should also be understood that, in some other embodiments, the above-mentioned terminal may also be a desktop computer having a touch-sensitive surface (eg, a touch panel).

FIG. 2 shows a schematic structural diagram of an electronic device 200 .

The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2 , mobile communication module 250, wireless communication module 260, audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, sensor module 280, buttons 290, motor 291, indicator 292, camera 293, display screen 294, and Subscriber identification module (subscriber identification module, SIM) card interface 295 and so on. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, and ambient light. Sensor 280L, bone conduction sensor 280M, etc.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 200 . In other embodiments of the present application, the electronic device 200 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The electronic device 200 implements a display function through a GPU, a display screen 294, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.

The electronic device 200 may implement a shooting function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, an application processor, and the like.

The SIM card interface 295 is used to connect a SIM card. The SIM card can be contacted and separated from the electronic device 200 by inserting into the SIM card interface 295 or pulling out from the SIM card interface 295 . The electronic device 200 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. The same SIM card interface 295 can insert multiple cards at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 is also compatible with external memory cards. The electronic device 200 interacts with the network through the SIM card to realize functions such as call and data communication. In some embodiments, the electronic device 200 employs an eSIM, ie: an embedded SIM card.

The wireless communication function of the electronic device 200 may be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modulation and demodulation processor, the baseband processor, and the like. Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 250 may provide a wireless communication solution including 2G/3G/4G/5G, etc. applied on the electronic device 200 . The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), and the like. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and then convert it into electromagnetic waves for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 250 may be provided in the processor 210 . In some embodiments, at least part of the functional modules of the mobile communication module 250 may be provided in the same device as at least part of the modules of the processor 210 .

The wireless communication module 260 can provide applications on the electronic device 200 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellites Wireless communication solutions such as global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (infrared radiation, IR) technology. The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2 , modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210 . The wireless communication module 260 can also receive the signal to be sent from the processor 210 , perform frequency modulation on the signal, amplify the signal, and then convert it into an electromagnetic wave for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the electronic device 200 is coupled with the mobile communication module 250, and the antenna 2 is coupled with the wireless communication module 260, so that the electronic device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.

It can be understood that the components shown in FIG. 2 do not constitute a specific limitation on the electronic device 200, and the electronic device 200 may also include more or less components than those shown in the figure, or combine some components, or separate some components components, or a different arrangement of components. In addition, the combination/connection relationship between the components in FIG. 2 can also be adjusted and modified.

The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take a layered architecture as an example, where the layered architecture may include a Harmony operating system

or other operating systems. The voice wake-up method provided by the embodiment of the present application may be applicable to a terminal integrated with the foregoing operating system.

The above FIG. 2 is the hardware structure of the electronic device to which the embodiment of the present application is applied. In order to solve the problems raised in the background technology, the embodiment of the present application provides a voice wake-up method, which can utilize a location map including the user's location and the device's location. And the face orientation of the user collected by the device can accurately determine the device the user wants to wake up, and improve the accuracy of the directional device wake-up result in the multi-device scenario.

Illustratively, as shown in FIG. 3 , it is a schematic diagram of a multi-device scenario to which this embodiment of the present application is applied. Specifically, in the multi-device scenario shown in FIG. 3 , it is assumed that the electronic device 10 , the electronic device 20 and the electronic device 30 are all sound pickup devices with multi-microphone arrays, and the same wake-up word is preset, for example, the wake-up word is "Little Art". When the user speaks "Xiaoyi Xiaoyi", the electronic device 10, the electronic device 20 and the electronic device 30 can all collect or receive the voice. On the one hand, the electronic device 30 can use the wake-up voice to determine the position of the user in the device map obtained by pre-training; and a location map including the user's location, to determine which of the electronic device 10 , the electronic device 20 and the electronic device 30 the target device the face is facing is.

It should be noted that FIG. 3 is only an example of a multi-device scenario, and the embodiment of the present application does not limit the number of electronic devices in the multi-device scenario, nor does it limit the pre-set wake-up words in the electronic devices. In addition, it should be noted that in other possible situations, the electronic device 30 may not collect the face image, but obtain the face image collection result from other devices (such as the electronic device 10 or the electronic device 20 ). 30 may be a central device with strong data processing capabilities, such as a smart speaker or a smart screen in a smart home scenario. In the following description, for the convenience of description, the electronic device 30 has a face image acquisition function and is the central device for description.

With reference to the multi-device scenario shown in FIG. 3 , a voice wake-up method according to an embodiment of the present application will be specifically described for the voice wake-up method according to the embodiment of the present application. As shown in FIG. 4 , the method flow specifically includes the following steps.

In steps 401a to 401c, the electronic device 10, the electronic device 20, and the electronic device 30 all collect ambient sounds in real time, and convert the collected ambient sounds into audio.

Specifically, the multi-microphone array of the electronic device 10 collects ambient sounds and converts the collected ambient sounds into audio, and the multi-microphone array of the electronic device 20 collects ambient sounds and converts the collected ambient sounds into audio. The multi-microphone array of the device 30 collects ambient sound and converts the collected ambient sound into audio. At this time, if the user sends out a wake-up voice, for example, when the user sends out "Xiaoyi Xiaoyi", the sounds collected by the electronic device 10, the electronic device 20 and the electronic device 30 will include the user's wake-up voice.

Exemplarily, referring to FIG. 1 , the user issues a voice wake-up command of "Xiaoyi Xiaoyi" facing the smart screen shown in Converted to audio, so the sound collected by smart screens, smart speakers and smart switches will include the user's wake-up voice.

In steps 402a to 402c, the electronic device 10, the electronic device 20 and the electronic device 30 all perform wake word detection on the generated audio.

Specifically, the electronic device 10, the electronic device 20, and the electronic device 30 can perform one-dimensional convolution on the data in each sliding window in the audio collected by their own devices, and extract the characteristics of different frequency bands in the data, so that when the When the audio recognizes an audio segment that is consistent with the user's preset voice characteristics, it means that the audio segment includes a wake-up word; otherwise, it does not include a wake-up word.

In steps 403a to 403b, when the electronic device 10 and the electronic device 20 detect the wake-up word, both send a wake-up message to the electronic device 30, and send the wake-up message with audio information generated by their own devices.

Specifically, when the electronic device 10 detects the wake-up word, the electronic device 10 sends a first wake-up message and audio information to the electronic device 30, where the first wake-up message is used to request confirmation of whether to wake up the electronic device 10; the electronic device 20 detects the wake-up The electronic device 20 sends a second wake-up message and audio information to the electronic device 30 when the word is activated, and the second wake-up message is used to request confirmation whether to wake up the electronic device 20 . The audio information may include all data of the audio, or the audio information may include information such as arrival time and phase related to the voice wake-up command.

For example, the information of the first audio generated by the electronic device 30 includes: the first time of arrival when the voice wake-up command reaches the first microphone, the second time of arrival when the voice wake-up command reaches the second microphone, and the voice The wake-up command reaches the first phase of the first microphone, and the voice wake-up command reaches the second phase of the second microphone. It should be noted that the first arrival time refers to the time when the first microphone picks up sound to the voice wake-up command, and the second arrival time refers to the first time when the second microphone picks up the voice wake-up command.

The electronic device 20 includes a third microphone and a fourth microphone; the information of the second audio generated by the electronic device 20 includes: the voice wake-up command reaches the third arrival time of the third microphone, the voice wake-up command reaches the fourth arrival time of the fourth microphone, And the voice wake-up command reaches the third phase of the third microphone, and the voice wake-up command reaches the fourth phase of the fourth microphone. It should be noted that the third arrival time refers to the time when the third microphone picks up the voice at the earliest to the voice wake-up command, and the fourth arrival time refers to the first time when the fourth microphone picks up the voice wake-up command.

It should be noted that the electronic device 10 , the electronic device 20 , and the electronic device 30 may further include other microphones, and the number of microphones is not limited in the embodiment of the present application, and other microphones may also collect sound according to the above method.

In addition, it should be noted that if the electronic device 10 or the electronic device 20 does not detect the wake-up word, there is no need to send a wake-up message to the electronic device 30 , and the electronic device 10 or the electronic device 20 only needs to send audio information to the electronic device 30 . Similarly, if the electronic device 20 does not detect a wake-up word, it does not need to send a wake-up message to the electronic device 30 , and the electronic device 20 only needs to send audio information to the electronic device 30 , which is not shown one by one in this embodiment.

In step 403c, when the electronic device 30 also detects the wake-up word, the third wake-up message is also generated, otherwise, the third wake-up message is not generated. Wherein, the third wake-up message is used to request confirmation whether to wake up the electronic device 30 .

Step 404, the electronic device 30 determines the relative position of the user in the device map according to the pre-trained device map and the audios collected by any two electronic devices in the electronic device 10, the electronic device 20 and the electronic device 30, thereby generating an image including the user Location map of the location.

Among them, the information between the electronic devices is synchronized based on a multi-device interconnection technology (such as HiLink (a multi-device interconnection technology)). Specifically, the electronic device 10, the electronic device 20, and the electronic device 30 can be connected to the same local area network. In order to realize mutual communication, or electronic device 10, electronic device 20 and electronic device 30 can be pre-bound with the same user account (such as a HUAWEI ID), or electronic device 10, electronic device 20 and electronic device 30 can be bound with different users account, and different user accounts have a binding relationship (such as pre-binding a family member's user account, that is, authorizing one's own device to connect with the family's device) to ensure secure communication between devices.

Exemplarily, as shown in FIG. 5 , the first microphone and the second microphone of the electronic device 30 both collect sound, and record the information of the first audio. On the one hand, the electronic device 30 may determine the electronic device 30 and the electronic device 30 according to the phase difference between the first phase of the voice wake-up command reaching the first microphone of the electronic device 30 and the second phase of the voice wake-up command reaching the second microphone of the electronic device 30 . The direction angle between the users; the electronic device 30 obtains the information of the second audio from the electronic device 20, so the electronic device 30 can determine the electronic device 20 and the user according to the phase difference between the third phase and the fourth phase of the electronic device 20 direction angle between. On the other hand, because the electronic device 30 can determine the first relative distance between the electronic device 30 and the user according to the time difference between the first arrival time and the second arrival time of the electronic device 30, and furthermore, the electronic device 30 can determine the first relative distance between the electronic device 30 and the user according to the time difference between the first arrival time and the second arrival time of the electronic device 30. The time difference between the third arrival time and the fourth arrival time of the electronic device 20 determines the second relative distance between the electronic device 20 and the user. In this way, the user position can be determined by combining the first azimuth angle, the second azimuth angle, the first relative distance and the second relative distance.

Exemplarily, as shown in FIG. 6A, point A refers to the position of the electronic device 20 in the pre-trained device map, point B refers to the position of the electronic device 30 in the pre-trained device map, and θA is The first azimuth angle of the electronic device 20 relative to the user, θB is the second azimuth angle of the electronic device 30 relative to the user, PA is the second relative distance of the electronic device 20 relative to the user, and PB is the first relative distance of the electronic device 30 relative to the user. a relative distance. It can be seen from the figure that the intersection point P of the two azimuth rays θA and θB is the position where the user sends out the wake-up speech.

Similarly, according to the above method, the electronic device 30 can also determine the direction angle between the electronic device 20 and the user according to the phase difference between different microphones of the electronic device 10 reached by the voice wake-up command.

It should be noted that, in this embodiment, TDoA (a method for localization using time difference) or MUSIC (a method for sound source localization) can be used to calculate the position of the sounding user and the distance from the device. The embodiments of the present application are not limited. The user location determined in the embodiment of the present application refers to a relative location. For example, the user to be located is due south of the electronic device 30, and the distance between the user and the electronic device 30 is 1 meter.

In step 405, the electronic device 20 collects an image of the user and detects the orientation of the face.

For example, the user is facing the smart screen in Figure 1 and sends out a wake-up voice of "Xiaoyi Xiaoyi", and the camera on the smart screen takes pictures or videos of the user. The smart screen analyzes the face image to determine that the user's face is facing the smart screen. For another example, the user is facing the smart speaker next to the smart screen in Figure 1 and sends out a wake-up voice of "Xiaoyi Xiaoyi", and the camera on the smart screen takes pictures or videos of the user. By analyzing the face image, the smart screen determines that the user's face is facing the first azimuth (for example, the first azimuth is the front left of the user).

It should be noted that this embodiment takes the smart screen as the main control device (or the central device) as an example for description. The smart screen has an image acquisition function, so the user image collected by the smart screen is preferentially used for face orientation detection. If the main control device (or the central device) does not have the image acquisition function, the user image may also be acquired from other electronic devices with the face acquisition function, and then analyzed based on the acquired user image. The examples are not shown one by one here.

Step 406 , the electronic device 30 determines that the target device to which the user's face faces is the electronic device 10 according to the face orientation and the location map including the user's location.

Exemplarily, the location map determined by the electronic device 30 and corresponding to the multi-device scenario shown in FIG. 3 is shown in FIG. 6B . In this figure, at the location of the user, the face of the user faces the corresponding The field of view is the range shown by the angle θ shown in the figure, and the electronic device 10 exists in this range.

Step 407a, when the electronic device 30 determines that the target device is the electronic device 10, it sends a wake-up permission instruction to the electronic device to instruct the electronic device 10 to respond to the user's wake-up voice.

In a possible embodiment, this embodiment may further include the following steps: step 407b, the electronic device 30 may further indicate that the electronic device 30 itself is not to be awakened when it is determined that the target device is the electronic device 10, and step 407c, the electronic device 30 The device 30 sends a wake-up prohibition instruction to the electronic device 20, where the wake-up prohibition instruction is used to instruct the electronic device 20 not to respond to the user's wake-up voice.

Or, in another possible embodiment, the electronic device 30 may not respond to the electronic device 30 itself and the electronic device 20 when it is determined that the target device is the electronic device 10, so that the electronic device 30 and the electronic device 20 also Does not respond to the user's wake-up voice.

To sum up, the first electronic device (such as the electronic device 30) receives the user's voice wake-up command; the first electronic device 30 obtains the user's image from other devices locally, and detects the user's face orientation; then according to the first electronic device and at least one The relative position of the second electronic device (such as the electronic device 10 and the electronic device 20), the user's position (such as the P point position) and the orientation of the user's face, determine the position of the user's face from the first electronic device and at least one second electronic device. The facing target device (eg, the target device is the electronic device 10 ), and instruct the target device to respond to the voice wake-up command. It can be seen that the above method can effectively improve the problem of "multiple responses with one call" or "multiple calls with one call", so that when the user issues a wake-up command to the electronic device 10, the electronic device 10 will make a voice interactive response to it. , and other devices will not respond.

In this embodiment, it is not required that the electronic devices in the multi-device scenario need to have an image capture function, that is, as long as one device in the multi-device scenario has an image capture function and can capture a face image, it can be combined with the location Maps and face images determine the device that the user is currently pointing to, and the pointed device may not have an image acquisition function at all, so to a certain extent, this method has wider application scenarios.

In addition, in this embodiment, when the user moves to another position in the scene to send out the wake-up voice, the current position of the user can still be relocated according to the methods shown in the above steps 401a to 404, so that the collected people can still be combined Face image, accurately locate the user's position after moving. Exemplarily, when the user walks around the smart speaker indoors, the electronic device 30 can locate the user's position from the position P1 to the position P2 according to the above method. It can be seen that this method can locate the current position of the vocal user in real time. Further, the target device can also be determined to be the electronic device 20 shown in FIG. 6C in combination with the currently collected face image and the current position P2 of the user. It can be seen that, according to the above method, the awakened device can always be located in a directional manner, and is not limited by whether the user's position changes.

Furthermore, in a possible embodiment, if it is determined that there are more than two candidate devices for the device to which the user's face is facing by combining the user's position and the face orientation, at this time, the candidate device and the user's position can be further combined. The relative distance between them determines the target device, that is, the candidate device with the shortest relative distance is selected from the two or more candidate devices as the target device. That is to say, the electronic device 30 calculates the relative distance between the candidate device within the face orientation range and the user's position, and then determines the wake-up priority of each candidate device according to the size of the relative distance. The device with the smaller relative distance has the wake-up priority. The higher the value, and vice versa, the device with the larger relative distance has a lower wake-up priority. In this way, the electronic device 30 can directionally wake up the candidate device with the highest priority. Exemplarily, as shown in FIG. 6D , when the user walks around the smart speaker indoors, the electronic device 30 can locate the user’s position from P1 to P2 according to the above method, and then combine the currently collected face image and the user’s face image. At the current position P2, candidate devices within the visual field corresponding to the face orientation are determined to include electronic device 20 (smart speaker as shown in the figure) and electronic device 40 (smart alarm clock as shown in the figure). The electronic device 30 determines that the relative distance between the electronic device 20 and P2 is D2, and determines that the relative distance between the electronic device 40 and P2 is D1. Since D1 is greater than D2, the wake-up priority of the electronic device 20 is high, Therefore, the electronic device 30 determines that the electronic device 10 is the target device to be awakened.

It can be seen that in the above method, a pre-trained device map is required, that is, a relative position map of each device in a multi-device scenario needs to be constructed first, and then the relative position of the user in the device map can be determined. To this end, an embodiment of the present application provides a method for training a device map, which reversely deduces the relative positions of a plurality of sound pickup devices by performing voice analysis on a user's historical wake-up speech. The construction of the device map is exemplarily given below with reference to FIGS. 7 to 9 .

Exemplarily, as shown in FIG. 7 , it is a schematic diagram of a multi-device scenario provided by an embodiment of the present application. In this scene, a plurality of sound pickup devices are deployed, wherein the main device with the image acquisition module is the smart screen device 71 in the figure, and a plurality of sound pickup devices 72a to the sound pickup device are also deployed in the space where the smart screen device 71 is located. 72f. It should be noted that, in the figure, multiple speakers are used to indicate the sound pickup device 72a to the sound pickup device 72f. In the actual home scene, the speakers can also be replaced with devices such as smart alarm clocks, smart cameras, and smart switches. Again, one by one is illustrated with an illustration. In the figure, only a sound box is used to indicate the sound pickup devices around the smart screen device 71 .

In the smart home scene shown in Figure 7, the user may wake up any sound pickup device at any position in the scene. For example, the user may sit on the sofa and call "Xiaoyi Xiaoyi" to wake up the smart screen device. For another example, The user calls "Xiaoyi Xiaoyi" facing the sound pickup device 72a (such as a speaker) to wake up the sound pickup device 72a. By analogy, during the user's use, each pickup device records the user's historical wake-up voice and synchronizes it to the central device. The central device can be the smart screen device shown in Figure 7, or it can be a smart speaker or router.

As shown in FIG. 8A , it is assumed that the sounding position when the user calls "Xiaoyi Xiaoyi" for the first time is the sounding point P1, and the sounding position when the user calls "Xiaoyi Xiaoyi" for the second time is the sounding point P2. Of course, during the use of the device, the user may also call other pickup devices at other locations. Here, the pickup device 72a and the pickup device 72b are awakened by two wake-up voices as an example. The sound pickup device 72a and the sound pickup device 72b convert the collected sounds into audio and then synchronize to the central device. The central device can perform voiceprint analysis on the sounds collected by the sound pickup device 72a and the sound pickup device 72b respectively. The audio device 72a and the pickup device 72b can perform one-dimensional convolution on the data in each sliding window in the audio collected by their own devices, and extract the voiceprint features of different frequency bands in the data, so that when the audio is identified from the audio with When an audio clip with the same voiceprint characteristics of the wake-up voice is preset, it means that the audio clip includes the wake-up word issued by the user; otherwise, the wake-up word issued by the user is not included. As shown in FIG. 8B , according to the above-mentioned voice detection method, the arrival time t1 at which the wake-up voice sent by the user at point P1 reaches the sound pickup device 72a, and the arrival time t2 at which the wake-up voice sent by the user at point P1 reaches the sound pickup device 72a, The audio voiceprints in Δt shown in the figure are the same, that is, the wake-up voice of the same user, so that the arrival time difference Δτ1=t2-t1 is obtained.

In a possible embodiment, in some special cases, there may be multiple users who send out the same wake-up voice at the same time. For example, the father and the child call "Xiaoyi Xiaoyi" in the living room at the same time. The method can detect the audio segment corresponding to the wake-up voice in this time period, but because multiple voiceprints appear in this time period, the audio overlaps, which is not conducive to accurately determine the phase of the wake-up voice. Therefore, the central device can further The audio is filtered, and the audio that overlaps with the audio in the same period is filtered out, so as to improve the accuracy of the calculation result of the device map.

Specifically, a polar coordinate system A is established on the sound pickup device 72a, a polar coordinate system B is established on the sound pickup device 72b, and a rectangular coordinate system C is established with two points AB as the X axis. As shown in Figure 8A,

is the angle between the polar coordinate system A and the rectangular coordinate system C; θA1 is the angle between the sounding point P1 to point A and the x-axis of the rectangular coordinate system C; θB1 is the sounding point P1 to point B and the rectangular coordinate system C. The included angle of the x-axis; dA1 is the distance from point P1 to point A; dB1 is the distance from point P1 to point B. in addition,

is the intersection angle between the polar coordinate system B and the rectangular coordinate system C; θA2 is the angle between the sound point P2 to point A and the x-axis of the rectangular coordinate system C; θB2 is the sound point P2 to point B and the x-axis of the rectangular coordinate system C The included angle of the axis; dA2 is the distance from the sounding point P2 to point A; dB2 is the distance from the sounding point P2 to point B.

In this embodiment, the distance dA1-dB1 can be calculated by using the time difference Δτ1 between the wake-up signal sent by the user at point P1 and the polar coordinate system A and the polar coordinate system B (that is, the sound pickup device 72a and the sound pickup device 72b ). poor, that is:

dA1-dB1=Δτ1·v=a1…Formula 1

Among them, dA1 is the distance from sound point P1 to point A; dB1 is the distance from sound point P1 to point B, Δτ1 is the time difference between the wake-up signal sent by the user at point P1 and polar coordinate system A and polar coordinate system B, namely Δτ1 is the time difference between the wake-up signal sent by the user and two different electronic devices, v is the speed of sound transmission, and a1 is the distance difference between dA1-dB1, that is, a1 is the distance difference between the user's voice position and two different electronic devices.

In addition, using the phase of the wake-up signal sent by the user at point P1 to polar coordinate system A, and the phase of the wake-up signal sent by the user at point P1 to polar coordinate system B, the following formula can be obtained:

180°-θA1-θB1=b1 Formula 2

Among them, θA1 is the angle between the sounding point P1 and point A and the x-axis of the rectangular coordinate system C; θB1 is the angle between the sounding point P1 and point B and the x-axis of the rectangular coordinate system C, and b1 is the distance from P1 to point A The angle between P1 and the distance from point B.

Specifically, θA1 and θB1 can be obtained by using the time difference and phase of the wake-up speech reaching different microphones as shown in FIG. 6A , and the calculation process is not repeated here.

in addition,

and

The difference between them is an unknown constant, and the unknown constant is used to characterize the included angle between the sound pickup direction corresponding to the sound pickup device 72a and the sound pickup direction corresponding to the sound pickup device 72b. which is

and

The following formulas are satisfied between:

Furthermore, using the cosine theorem, the following formula can be obtained:

Further derivation of the above formula 4, the following formula 5 can be obtained:

By combining the above formula 1, formula 2, formula 3 and formula 5, and simplifying the constants with the following letters, the following equations can be obtained:

in,

Using formula 2 and formula 3 to further simplify the following formula 6,

Among them, x1 is an unknown number, which refers to dA1.

By analogy, because the sounding position when the user calls "Xiaoyi Xiaoyi" for the second time is the sounding point P2, according to θA2 is the angle between the sounding point P2 and point A and the x-axis of the rectangular coordinate system C; θB2 is the sounding point P2 The angle between point B and the x-axis of the Cartesian coordinate system C; dA2 is the distance from point P2 to point A; dB2 is the distance from point P2 to point B. According to the above calculation method, the following equations can be obtained simultaneously:

Likewise, one can get

Among them, x ₂ is an unknown number, which refers to dA2.

And so on, because the sounding position when the user calls "Xiaoyi Xiaoyi" for the nth time is the sounding point Pn, according to θAn is the angle between the sounding point Pn to point A and the x-axis of the Cartesian coordinate system C; θBn is the sounding point Pn The angle between point B and the x-axis of the rectangular coordinate system C; dAn is the distance from the sound point Pn to point A; dBn is the distance from the sound point Pn to point B. According to the above calculation method, the following equations can be obtained simultaneously:

Likewise, one can get

Among them, x _n is an unknown number, which refers to dAn.

That is to say, according to the user's historical wake-up voice, the following equations can be combined according to the above method:

Simplify the constants using letters to combine the simplified equations:

In this way, using the analysis results of a large number of historical wake-up voices, a number of equations can be created simultaneously, and by solving algorithms (such as numerical solutions), the distance between devices and the angle between devices of the optimal device can be obtained, and the device can be calculated. The relative positions between , exemplarily, are shown in FIG. 8C . That is to say, the side length calculated by the distance divergence attenuation formula is brought into the above equation, and the distance between the devices and the angle between the devices are searched through the grid search method (Grid search), and the least squares method is obtained to minimize the loss function of all equations. The device spacing and the device angle that minimize the loss function are the precise device relative positions obtained by offline learning at night. In this way, through the exhaustive search method of the grid method, the optimal relative position between the devices can be obtained. With the increase of the accumulated user's historical wake-up voice, the device will always record the dosing data when the user calls Xiaoyi, and the device can be used at night time. Use the saved data for offline learning and training to achieve the effect of getting smarter the more you use it, and the smaller the error calculated by the exhaustive search method.

To sum up, as shown in FIG. 9 , a device map training method according to an embodiment of the present application is used to locate the relative positions of each device in a multi-device scenario. As shown in FIG. 9 , the method flow specifically includes the following steps.

In steps 901a to 901c, the electronic device 10, the electronic device 20, and the electronic device 30 all collect ambient sounds in real time, and convert the collected ambient sounds into audio.

Specifically, the multi-microphone array of the electronic device 10 collects ambient sounds and converts the collected ambient sounds into audio, and the multi-microphone array of the electronic device 20 collects ambient sounds and converts the collected ambient sounds into audio. The multi-microphone array of the device 30 collects ambient sound and converts the collected ambient sound into audio. At this time, if the user sends a wake-up voice, for example, when the user sends a voice command of "Xiaoyi Xiaoyi", the sound collected by the electronic device will include the user's wake-up voice.

Exemplarily, the user sends out the wake-up voice of "Xiaoyi Xiaoyi" facing the smart screen shown in Figure 1. Because the smart screen, smart speakers and smart switches all collect the sound of the surrounding environment in real time and convert it into audio , smart speakers and smart switches will include the user's wake-up voice.

In steps 902 a to 902 b , the electronic device 10 and the electronic device 20 synchronize the generated audio to the electronic device 30 .

That is, the electronic device 10 synchronizes the generated audio to the electronic device 30 ; the electronic device 20 synchronizes the generated audio to the electronic device 30 . It should be noted that the electronic device 10 and the electronic device 20 may synchronize the generated audio to the electronic device 30 at regular intervals, or both may synchronize the generated audio to the electronic device 30 at a fixed time point (eg, one o'clock in the morning). This application does not limit this.

In addition, this embodiment is described by taking a multi-device scenario including the electronic device 10 , the electronic device 20 , and the electronic device 30 as an example, and the electronic device 30 is the central device. In other possible embodiments, other electronic devices may also be included, or the central device may be other electronic devices, and so on. For other electronic devices, reference may also be made to the above-mentioned electronic devices 10 to 30, which will not be described one by one here. .

Step 903, the electronic device 30 analyzes the audio generated by the electronic device 10, the electronic device 20, and the electronic device 30 itself according to the method shown in FIG.

In this embodiment, the calculation process of the relative positions between the multiple devices relies on the information of the historical wake-up voices. By receiving the arrival time difference information of the wake-up voices for multiple devices multiple times, and the arrival time difference of the wake-up voices reaching different microphones of the same device, Get the relative position of each device. Because the above method can continuously use the historical user wake-up voice in the recent period of time to locate the device, even if the location of the device changes during the use of the device, for example, the smart speaker is moved from the living room to the dining room, according to the above method, By accumulating the information of the historical wake-up voice within a period of time after the smart speaker is moved, the latest relative position between the devices can still be updated. The feeling to the user is that the device gets smarter the longer it is used.

In addition, this embodiment does not require the use environment of the sound pickup device, that is to say, even if the device is in an environment with noise interference, it is possible to reversely deduce the relationship between multiple sound pickup devices through multiple voice recognition of the user's voice. relative position between.

In order to realize the training process of the above device map, in the embodiment of the present application, in each device in a multi-device scenario, a multi-layer perception capability layer of the device is constructed, as shown in FIG. 10 , which specifically includes: a basic perception capability layer, a second layer Perceptual ability layer, high-level perception ability layer.

Among them, the basic perception capability layer refers to the capability after the existing functions of some devices or software are simply encapsulated by the intelligent perception framework. For example, the chip layer provides the face orientation capability, the audio source positioning capability for multi-microphone devices, and the incoming and outgoing power status monitoring capability provided by the Android layer.

The second-layer perception capability layer refers to the computing result obtained after processing the basic perception capability through calculation, and the capability after being encapsulated by the framework. For example, the device-to-device localization capability and the user's voice localization capability are all calculation results obtained by processing the sound localization data (basic perception capability) reported by different underlying devices.

The high-level perception capability layer refers to the complex calculation results reported to the upper layer calculated through complex calculations, fusions, models, and rules. For example, the directional awakening ability, through the fusion of multiple basic perception capabilities and second-level perception capabilities, the calculated advanced capabilities are finally combined.

In addition, in the software implementation process of the present application, a fence mechanism is also provided, which means that a virtual fence encloses a virtual boundary. When the mobile phone enters or leaves a certain geographic area/specific action, etc., the upper-layer application of the mobile phone can receive automatic notifications and warnings. Fence thinking is the core of intelligent perception, and each ability will be triggered with a corresponding fence. Capability and fences form a platform that provides the upper layer with the ability to perceive user behavior, and upper-layer applications can receive reports when users enter and exit specific events.

As shown in FIG. 11 , which is a schematic diagram of software modules in a multi-device scenario provided by an embodiment of the present application, the modules in the slave device 1 to the slave device n and the master device can cooperate to implement the device map training method provided by the embodiment of the present application. . The master device refers to the above-mentioned central device for locating the device position, such as the electronic device 30, and the slave device refers to the above-mentioned device for collecting wake-up voice, such as the electronic device 10 and the electronic device 20.

In the scenario shown in FIG. 11 , each slave device includes an audio collection module 1101 , and the master device includes an audio collection module 1101 , an audio processing module 1102 , and an audio identification module 1103 .

Among them, the audio collection module 1101 is used to collect the sound in the environment by using a multi-microphone array, and convert the collected sound into audio.

The audio collection module 1101 of each device may send the audio corresponding to each sampling period to the audio processing module 1102 of the main device (eg, the electronic device 30 ) for processing. The main device may be the smart screen device shown in FIG. 3 , or may be a device with strong computing power in a smart home scenario, such as a router or a smart speaker.

The audio processing module 1102 of the main device is used to preprocess the audio corresponding to each sampling period, such as channel conversion, smoothing, noise reduction, etc., so that the audio recognition module 1103 can perform subsequent wake-up speech detection.

The audio identification module 1103 of the main device is used to identify the wake-up signal for the audio of each sampling period after preprocessing, and identify the information of the same wake-up signal from the audio of different sampling periods, such as the arrival time of the audio where the wake-up signal is located, etc. . The audio recognition module 1103 sends the information identifying the same wake-up signal to the device map calculation module 1104 .

The device map calculation module 1104 is configured to calculate the relative positions between different devices according to the arrival time of the same wake-up voice to different devices and the arrival time of the wake-up signal to different microphones of the same device.

In addition, the modules in the slave device 1 to the slave device n and the master device can cooperate to implement the voice wake-up method provided by the embodiment of the present application. In order to implement the voice wake-up method, each slave device in the scenario shown in FIG. 12 may further include a user position positioning module 1105 , a face orientation recognition module 1106 and a directional wake-up module 1107 .

Wherein, the user position positioning module 1105 is used to use the multi-microphone array in a single device to calculate the arrival time of the wake-up voice currently sent by the user to the multi-microphone device, and the arrival time of the multi-microphone device based on the wake-up voice collected by at least two devices, Calculate the position of the user's voice.

The face orientation recognition module 1106 is used to calculate the face orientation of the user in the location map by using the face orientation recognition capability of the chip layer within the range of the wide-angle camera of the camera of the device.

The directional wake-up module 1107 is used to obtain the target device to be woken up by using the device map calculated and processed by the device map calculation module 1104, the user position located by the user position location module 1105, and the face orientation.

It should be understood that the software structure shown in FIG. 11 is only an example. The electronic device of the embodiments of the present application may have more or less modules than the electronic device shown in the figure, two or more modules may be combined, and so on. The various modules shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

It should be noted that the audio processing module 1102, the audio recognition module 1103, the device map calculation module 1104, the user position positioning module 1105, the face orientation recognition module 1106 and the directional wake-up module 1107 shown in FIG. 11 can be integrated in FIG. 2 Among the one or more processing units in the processor 210 shown, for example, the audio processing module 1102, the audio recognition module 1103, the device map calculation module 1104, the user location positioning module 1105, the face orientation recognition module 1106, and the directional wake-up Some or all of the modules 1107 may be integrated into one or more processors such as an application processor, a special purpose processor, or the like. It should be noted that, the dedicated processor in the embodiment of the present application may be a DSP, an application specific integrated circuit (application specific integrated circuit, ASIC) chip, or the like.

The following embodiments can all be implemented in an electronic device having the above-mentioned hardware structure and/or software structure.

Based on the same concept, FIG. 12 shows a device 1200 provided by the present application. Device 1200 includes at least one processor 1210 , memory 1220 and transceiver 1230 . The processor 1210 is coupled with the memory 1220 and the transceiver 1230. The coupling in this embodiment of the present application is an indirect coupling or communication connection between devices, units, or modules, which may be electrical, mechanical, or other forms for the device , information exchange between units or modules. The connection medium between the transceiver 1230, the processor 1210, and the memory 1220 is not limited in the embodiments of the present application. For example, in the embodiment of the present application, in FIG. 12 , the memory 1220 , the processor 1210 , and the transceiver 1230 may be connected through a bus, and the bus may be divided into an address bus, a data bus, a control bus, and the like.

Specifically, the memory 1220 is used to store program instructions.

The transceiver 1230 is used to receive or transmit data.

The processor 1210 is configured to invoke the program instructions stored in the memory 1220, so that the device 1200 executes the steps performed by the electronic device 30 or the steps performed by the electronic device 10 or the electronic device 20 in the above method.

In this embodiment of the present application, the processor 1210 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which can implement Alternatively, each method, step, and logic block diagram disclosed in the embodiments of the present application are executed. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

In this embodiment of the present application, the memory 1220 may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), Such as random-access memory (random-access memory, RAM). Memory is, but is not limited to, any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory in this embodiment of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and/or data.

It should be understood that the device 1200 can be used to implement the methods shown in the embodiments of the present application, and the relevant features can be referred to above, which will not be repeated here.

Those skilled in the art can clearly understand that the embodiments of the present application may be implemented by hardware, firmware, or a combination thereof. When implemented in software, the functions described above may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available medium that a computer can access. Taking this as an example but not limited to: computer readable media may include RAM, ROM, electrically erasable programmable read only memory (EEPROM), compact disc read-Only memory (CD- ROM) or other optical disk storage, magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and that can be accessed by a computer. also. Any connection can be appropriately made into a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, Then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, wireless, and microwave are included in the fusing of the pertinent medium. As used in the embodiments of the present application, disks and discs include compact discs (CDs), laser discs, optical discs, digital video discs (DVDs), floppy disks, and Blu-ray discs, wherein Disks usually reproduce data magnetically, while discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

In a word, the above descriptions are merely examples of the present application, and are not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included within the protection scope of this application.

Claims

A voice wake-up method, applied to a first electronic device, characterized in that the method comprises:

Receive the user's voice wake-up command;

Obtain the user image and detect the user's face orientation;

According to the relative positions of the first electronic device and at least one second electronic device, the user's position and the orientation of the user's face, determine the orientation of the user's face from the first electronic device and the at least one second electronic device target device;

The target device is instructed to respond to the voice wake-up command.
The method according to claim 1, wherein the target device to which the user's face is facing is determined according to the relative positions of the first electronic device and at least one second electronic device, the user's position, and the user's face orientation ,include:

According to the relative positions of the first electronic device and at least one second electronic device, the user's position and the orientation of the user's face, determine the orientation of the user's face from the first electronic device and the at least one second electronic device candidate device;

When the number of the candidate devices is greater than or equal to two, determining the relative distance between the user and the at least two candidate devices;

According to the relative distance, the priority of the candidate device is determined, wherein the smaller the relative distance is, the smaller the priority of the candidate device is;

Determine the candidate device corresponding to the highest priority as the target device.
The method according to claim 1 or 2, characterized in that,

acquiring first audio information of the first electronic device, and acquiring second audio information from the at least one second electronic device;

The user location is determined according to the information of the first audio and the information of the second audio.
The method according to claim 3, wherein the first electronic device comprises a first microphone and a second microphone; the information of the first audio comprises: the voice wake-up instruction reaches the first microphone of the first microphone. At the arrival time, the voice wake-up command reaches the second arrival time of the second microphone, and the voice wake-up command reaches the first phase of the first microphone, and the voice wake-up command reaches the second microphone. second phase;

The at least one second electronic device includes a third microphone and a fourth microphone; the second audio information includes: the voice wake-up command arrives at the third arrival time of the third microphone, and the voice wake-up command reaches the third microphone. the fourth arrival time of the fourth microphone, the third phase at which the voice wake-up command reaches the third microphone, and the fourth phase at which the voice wake-up command reaches the fourth microphone.
The method according to claim 4, wherein determining the user location according to the information of the first audio and the information of the second audio comprises:

Determine the relative distance between the user and the first electronic device according to the time difference between the first arrival moment and the second arrival moment, and determine the relative distance between the first phase and the second phase according to the phase difference between the first phase and the second phase an azimuth of the user relative to the first electronic device;

determining a relative distance between the user and the at least one second electronic device according to the time difference between the second arrival moment and the third arrival moment; and according to the phase difference between the third phase and the fourth phase , determining the azimuth of the user relative to the at least one second electronic device;

According to the relative distance of the user and the first electronic device, the azimuth of the user relative to the first electronic device and the relative distance of the user and the at least one second electronic device, and the user The user location is determined relative to the azimuth of the at least one second electronic device.
The method according to any one of claims 1 to 5, wherein the method further comprises:

Obtain historical audio information from the first electronic device and the at least one second electronic device;

From the information of the historical audio, obtain the arrival times and phases of the voice wake-up commands issued by the user N times to different electronic devices, where N is a positive integer;

Determine the relative azimuth and distance difference corresponding to the voice wake-up commands issued by the user N times according to the arrival times and phases of the voice wake-up commands issued by the user N times to different electronic devices;

The objective function is established by taking the relative azimuth and distance difference corresponding to the voice wake-up commands issued by the user N times as the observed values;

The objective function is solved by an exhaustive search method to obtain the relative positions of the first electronic device and at least one second electronic device.
The method according to claim 6, wherein, according to the arrival times and phases of the voice wake-up commands issued by the user N times to different electronic devices, the relative azimuth angle and the relative azimuth angle corresponding to the voice wake-up commands issued by the user N times are determined distance differences, including:

For the voice wake-up command issued at the Kth time among the voice wake-up commands issued by the user N times, the Kth time is any one of the N times, and the distance difference corresponding to the voice wake-up command issued by the user at the Kth time The following formula 1 is satisfied:

Δτ1·v=a1 Formula 1

Among them, Δτ1 is the time difference between the wake-up signal sent by the user and two different electronic devices, v is the speed of sound transmission, and a1 is the distance difference between the user's voice position and two different electronic devices;

The relative azimuth angle corresponding to the voice wake-up command issued by the user for the Kth time satisfies the following formula 2:

180°-θA1-θB1=b1 Formula 2

Among them, θA1 is the azimuth angle of the user's position relative to the second electronic device, θB1 is the azimuth angle of the user's sounding position relative to the first electronic device; θB1 is the sounding point P1 to point B and the Cartesian coordinate system C The included angle of the x-axis, b1 is the relative azimuth angle of the user's voice position relative to two different electronic devices.
The method according to any one of claims 1 to 7, wherein the first electronic device and the at least one second electronic device are connected to the same local area network, or the first electronic device and the at least one second electronic device are connected to the same local area network. The electronic device is pre-bound with the same user account, or the first electronic device and the at least one second electronic device are bound with different user accounts, and the different user accounts establish a binding relationship.
The method according to any one of claims 1 to 7, wherein the first electronic device has an image capture function, or the at least one second electronic device has an image capture function.
A voice wake-up system, characterized in that the system includes a first electronic device and at least one second electronic device;

The second electronic device is configured to collect the sound of the surrounding environment through a multi-microphone array and convert it into a second audio frequency, where the sound of the surrounding environment includes a voice wake-up instruction issued by the user;

the second electronic device, further configured to send second audio information to the first electronic device;

The first electronic device is configured to acquire information of the first audio of the first electronic device, and receive information from the at least one second electronic device to acquire the second audio; according to the information of the first audio and the information of the second audio, to determine the user position; according to the relative position of the first electronic device and at least one second electronic device, the user position and the face orientation of the user, from the first electronic device and the user determining the target device to which the user's face faces in at least one second electronic device;

The first electronic device is further configured to instruct the target device to respond to the voice wake-up instruction.
The system according to claim 10, wherein the first electronic device determines the user's personality according to the relative positions of the first electronic device and the at least one second electronic device, the user's position and the face orientation of the user The target device the face is facing, specifically for:

According to the relative positions of the first electronic device and at least one second electronic device, the user's position and the orientation of the user's face, determine the orientation of the user's face from the first electronic device and the at least one second electronic device candidate device;

When the number of the candidate devices is greater than or equal to two, determining the relative distance between the user and the at least two candidate devices;

According to the relative distance, the priority of the candidate device is determined, wherein the smaller the relative distance is, the smaller the priority of the candidate device is;

Determine the candidate device corresponding to the highest priority as the target device.
A system according to claim 10 or 11, characterized in that:

The first electronic device includes a first microphone and a second microphone; the information of the first audio includes: a first arrival time of the voice wake-up command to the first microphone, and the voice wake-up command reaches the first arrival time. The second arrival time of the second microphone, and the first phase at which the voice wake-up command reaches the first microphone, and the second phase at which the voice wake-up command reaches the second microphone;

The at least one second electronic device includes a third microphone and a fourth microphone; the second audio information includes: the voice wake-up command arrives at the third arrival time of the third microphone, and the voice wake-up command reaches the third microphone. the fourth arrival time of the fourth microphone, the third phase at which the voice wake-up command reaches the third microphone, and the fourth phase at which the voice wake-up command reaches the fourth microphone.
The system according to claim 12, wherein the first electronic device determines the user position according to the information of the first audio and the information of the second audio, and is specifically used for:

Determine the relative distance between the user and the first electronic device according to the time difference between the first arrival moment and the second arrival moment, and determine the relative distance between the first phase and the second phase according to the phase difference between the first phase and the second phase an azimuth of the user relative to the first electronic device;

determining a relative distance between the user and the at least one second electronic device according to the time difference between the second arrival moment and the third arrival moment; and according to the phase difference between the third phase and the fourth phase , determining the azimuth of the user relative to the at least one second electronic device;

According to the relative distance of the user and the first electronic device, the azimuth of the user relative to the first electronic device and the relative distance of the user and the at least one second electronic device, and the user The user location is determined relative to an azimuth of the at least one second electronic device.
The system according to any one of claims 10 to 13, wherein the first electronic device is further configured to:

Obtain historical audio information from the first electronic device and the at least one second electronic device;

From the information of the historical audio, obtain the arrival times and phases of the voice wake-up commands issued by the user N times to different electronic devices, where N is a positive integer;

Determine the relative azimuth angle and distance difference corresponding to the voice wake-up commands sent by the user N times according to the arrival times and phases of the voice wake-up commands sent by the user N times to different electronic devices;

The objective function is established by taking the relative azimuth and distance difference corresponding to the voice wake-up commands issued by the user N times as the observed values;

The objective function is solved by an exhaustive search method to obtain the relative positions of the first electronic device and at least one second electronic device.
An electronic device, characterized in that the electronic device includes a processor and a memory;

Program instructions are stored in the memory;

When the program instructions are executed, the electronic device is caused to perform the method of any one of claims 1 to 9.
A chip system, characterized in that the chip system is coupled with a memory in an electronic device, so that the chip invokes program instructions stored in the memory when running, so as to implement the method according to any one of claims 1 to 9. method.
A computer-readable storage medium, characterized in that the computer-readable storage medium comprises program instructions, which, when the program instructions are executed on an electronic device, cause the electronic device to perform any one of claims 1 to 9 the method described.