WO2020244355A1

WO2020244355A1 - Microphone signal-based voice interaction wake-up electronic device, method, and medium

Info

Publication number: WO2020244355A1
Application number: PCT/CN2020/089551
Authority: WO
Inventors: 史元春; 喻纯
Original assignee: 清华大学
Priority date: 2019-06-03
Filing date: 2020-05-11
Publication date: 2020-12-10
Also published as: CN110097875A; CN110097875B

Abstract

A smart electronic device having a built-in microphone; the smart electronic portable device performs voice input-based interaction with a user by means of the following operations: processing a sound signal captured by a microphone to determine whether a voice signal is present in the sound signal; in response to confirming that a voice signal is present in the sound signal, further determining on the basis of the sound signal collected by the microphone whether the distance between the smart electronic device and the mouth of the user is less than a predetermined threshold; and in response to determining that the distance between the electronic device and the mouth of the user is less than the predetermined threshold, using the sound signal collected by the microphone as a voice input and processing same. The described interaction method is suitable for a user performing voice input while carrying a smart electronic device, operation being natural and simple; in addition, the step of voice input is simplified, and the burden and difficulty of interaction are reduced so that interaction is more natural.

Description

Voice interaction based on microphone signal to wake up electronic equipment, method and medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on June 3, 2019, the application number is 201910475949.X and the invention title is "Microphone signal-based voice interaction wake-up electronic equipment, methods and media", and its entire content Incorporated in this application by reference.

Technical field

This application relates to the field of voice input, and more specifically, to smart electronic devices and voice input triggering methods.

Background technique

With the development of computer technology, speech recognition algorithms are becoming more and more mature, and speech input is becoming more and more important due to its high naturalness and effectiveness in interactive methods. Users can interact with mobile devices (mobile phones, watches, etc.) through voice to complete various tasks such as command input, information query, and voice chat.

As for when to trigger the voice input, the existing solutions have some shortcomings:

1. Physical button trigger

After pressing (or holding down) a certain (or certain) physical buttons of the mobile device, voice input is activated.

The disadvantages of this solution are: physical buttons are required; easy to trigger by mistake; user buttons are required.

2. Interface element trigger

Tap (or hold) an interface element (such as an icon) on the screen of the mobile device to activate voice input.

The disadvantages of this solution are: the device needs to have a screen; the trigger element occupies the screen content; the limitation of the software UI may lead to a cumbersome trigger method; it is easy to trigger by mistake.

3. Wake word (voice) detection

Using a specific word (such as product nickname) as the wake-up word, the device activates voice input after detecting the corresponding wake-up word.

The disadvantages of this scheme are: poor privacy and sociality; low interaction efficiency.

Summary of the invention

In view of the above situation, this application is submitted:

According to one aspect of the present application, a smart electronic device with a built-in microphone, the smart electronic portable device interacts with a user based on voice input as follows: process the sound signal captured by the microphone to determine whether there is a voice signal in the sound signal; After confirming that there is a voice signal in the sound signal, it is further determined based on the sound signal collected by the microphone whether the distance between the smart electronic device and the user’s mouth is less than a predetermined threshold; in response to determining that the distance between the electronic device and the user’s mouth is less than the predetermined threshold, the sound collected by the microphone is The signal is processed as voice input.

Preferably, the predetermined threshold is 3 cm.

Preferably, the predetermined threshold is 1 cm.

Preferably, there is a proximity light sensor at the microphone of the electronic device, and the proximity light sensor determines whether an object is approaching the electronic device.

Preferably, there is a distance sensor at the microphone of the electronic device, and the distance between the electronic device and the user's mouth is directly measured by the distance sensor.

Preferably, it is determined whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold value through the characteristics of the sound signal collected by the microphone.

Preferably, the voice signal includes one or a combination of the following items: the sound produced by the user speaking at a normal volume; the sound produced by the user speaking in a low voice; the sound produced by the user's vocal cord without speaking.

Preferably, the electronic device is further operable: in response to determining that the user is speaking to the electronic device at close range; determining that the user is speaking in one of the following ways, including: the user's voice at a normal volume; the user with a low volume The voice of speaking; the voice that the user speaks in a way that the vocal cords are silent; and according to the different results of the judgment, the voice signal is processed differently.

Preferably, the different processing is activating different applications to process voice input.

Preferably, the characteristics used in the judgment include volume, frequency spectrum characteristics, energy distribution, and the like.

Preferably, the features used when judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold include time-domain features and frequency-domain features in the sound signal, including volume and spectral energy.

Preferably, the judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold includes: extracting a voice signal from a sound signal collected by a microphone through a filter; judging whether the energy of the voice signal exceeds a certain threshold; responding to the voice If the signal strength exceeds a certain threshold, it is determined that the distance between the electronic device and the user's mouth is less than the predetermined threshold.

Preferably, the judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold includes: using a deep neural network model to process data collected by a microphone to determine whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold.

Preferably, the judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold includes: recording the user's voice signal when the user is not making a voice input; and comparing the voice signal currently collected by the microphone with the voice signal when the user is not making a voice input. Comparison; if it is determined that the volume of the voice signal currently collected by the microphone exceeds a certain threshold of the volume of the voice signal when no voice input is made, it is determined that the distance between the smart electronic device and the user's mouth is less than the predetermined threshold.

Preferably, the processing of using the sound signal as the user's voice input includes one or more of the following: storing the sound signal on a storage medium on the electronic device; sending the sound signal through the Internet; sending the voice in the sound signal The signal is recognized as text and stored on the storage medium of the electronic device; the voice signal in the sound signal is recognized as text and sent through the Internet; the voice signal in the sound signal is recognized as text, the user’s voice command is understood, and the corresponding operating.

Preferably, the electronic device also recognizes a specific user through voiceprint analysis, and only processes the voice signal containing the voice of the specific user.

Preferably, the electronic device is a smart phone, a smart watch, a smart ring, etc.

The mobile devices here include, but are not limited to, mobile phones, head-mounted displays, watches, and smaller smart wearable devices such as smart rings and watches.

According to another aspect of the present application, there is provided a voice input triggering method executed by a smart electronic device equipped with a microphone, including the following operations for the smart electronic device to interact with a user based on voice input: processing sound signals captured by the microphone Determine whether there is a voice signal in the sound signal; in response to confirming that there is a voice signal in the sound signal, further determine whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold based on the sound signal collected by the microphone; in response to determining whether the electronic device and the user's mouth The distance is less than a predetermined threshold, and the sound signal collected by the microphone is processed as a voice input.

According to another aspect of the present application, there is provided a computer-readable medium having computer-executable instructions stored thereon, and when the computer-executable instructions are executed by a computer, a voice interactive wake-up method can be executed. The voice interactive wake-up method includes: processing The sound signal captured by the microphone determines whether there is a voice signal in the sound signal; in response to confirming that there is a voice signal in the sound signal, it is further determined based on the sound signal collected by the microphone whether the distance between the smart electronic device and the user’s mouth is less than a predetermined threshold; The distance between the device and the user's mouth is less than a predetermined threshold, and the sound signal collected by the microphone is processed as a voice input.

According to one aspect of the present application, there is provided an electronic device equipped with a microphone. The electronic device has a memory and a central processing unit. The memory stores computer executable instructions. When the computer executable instructions are executed by the central processing unit, the following operations can be performed : Analyze the sound signal collected by the microphone, identify whether the sound signal contains human speech and whether it contains the wind noise generated by the airflow generated by the human speech hitting the microphone, and respond to determining that the sound signal contains the human speech and the user's speech production The wind noise generated by the airflow hitting the microphone is processed as the user’s voice input.

Preferably, the voice spoken by the user includes: the voice of the user speaking at a normal volume, the voice of the user speaking at a low volume, and the voice of the user speaking in a non-voicing manner.

Preferably, the electronic device is further operable: in response to determining that the user is speaking to the electronic device at a close distance, it is determined that the user is speaking in one of the following ways, including: the user speaking at a normal volume, and the user speaking at a low volume Speaking sound is the sound that the user speaks in a way that the vocal cords are silent; according to the different judgment results, the sound signal is processed differently.

Preferably, the processing of using the sound signal as the user's voice input includes one or more of the following: storing the sound signal on a storage medium on the electronic device; sending the sound signal through the Internet; sending the voice in the sound signal The signal is recognized as text and stored on the storage medium of the electronic device; the voice signal in the sound signal is recognized as text and sent out via the Internet; the voice signal in the sound signal is recognized as text, the user’s voice command is understood, and the corresponding operating.

Preferably, the electronic device is also operable to identify a specific user through voiceprint analysis, and only process the voice signal containing the voice of the specific user.

Preferably, the electronic device is one of a smart phone, a smart watch, and a smart ring.

Preferably, the electronic device is further operable to use a neural network model to determine whether the voice signal of the user is included in the voice signal and the wind noise generated by the airflow generated by the speech hitting the microphone.

Preferably, the electronic device is further operable to recognize whether the sound signal contains the voice spoken by a human and whether it contains the wind noise generated by the airflow generated by the human speech hitting the microphone includes recognizing whether the voice signal contains the voice spoken by the user; in response to determining the sound The signal contains the user's spoken voice, recognizes the phonemes in the voice, and expresses the voice signal as a phoneme sequence; for each phoneme in the phoneme sequence, determine whether the phoneme is an exhaled phoneme, that is: there is airflow from the mouth when the user utters the phoneme Out; cut the sound signal into a sequence of sound segments according to a fixed window length; use frequency characteristics to identify whether each sound segment contains wind noise; identify the exhaled phoneme in the speech phoneme sequence and the segment in the sound segment sequence as wind noise For comparison, the non-exhaled phoneme and wind noise segment in the phoneme sequence are compared at the same time. When the overlap degree of the exhaled phoneme and the wind noise segment is higher than a certain threshold, and the overlap degree of the non-exhaled phoneme and the non-wind noise segment is lower than a certain threshold, it is judged The sound signal contains wind noise generated by the airflow generated by the user's speech hitting the microphone.

Preferably, recognizing whether the sound signal contains human speech and whether it contains wind noise generated by the airflow generated by human speech hitting the microphone includes: recognizing the sound characteristics of wind noise contained in the sound signal; and in response to determining that the sound signal contains wind noise , Recognize that the sound signal contains a voice signal; In response to determining that the sound signal contains a voice signal, identify the phoneme sequence corresponding to the voice signal; For the wind noise feature in the sound signal, calculate the wind noise feature intensity at each moment; For the phoneme sequence For each phoneme, the exhalation intensity of the phoneme is obtained according to the pre-defined data model; the consistency of the wind noise feature and the phoneme sequence is analyzed based on the Gaussian Mixture Bayesian model. When the coincidence degree is higher than a certain threshold, it is determined that the sound signal contains The wind noise generated by the airflow generated by the user's speech hitting the microphone.

According to another aspect of the present application, there is provided an electronic device with a plurality of built-in microphones. The electronic device has a memory and a central processing unit. The memory stores computer-executable instructions. The computer-executable instructions can be executed by the central processing unit. Perform the following operations: analyze the sound signals collected by multiple microphones; determine whether the user is talking to the electronic device at close range; in response to determining that the user is talking to the electronic device at close range, the sound signal collected by the microphone is processed as the user's voice input .

Preferably, a plurality of microphones constitute a microphone array system.

Preferably, the judging whether the user is speaking into the electronic device at close range includes: calculating the position of the user's mouth relative to the microphone array by using the time difference between the sound signals arriving at each microphone on the array; When the distance is less than a certain threshold, it is determined that the user is talking to the electronic device at a close distance.

Preferably, the distance threshold is 10 cm.

Preferably, the processing the voice signal as the user's voice input includes: performing different processing on the user's voice input according to the distance between the speaker's mouth and the electronic device.

Preferably, the judging whether the user is speaking into the electronic device at close range includes: judging whether the sound signal collected by at least one microphone contains the voice signal of the user speaking; in response to determining that the sound signal collected by the at least one microphone contains the user Speaking voice signal, extract the voice signal from the voice signal collected by the microphone; determine whether the amplitude difference of the voice signal extracted from the voice signal collected by different microphones exceeds a predetermined threshold; in response to determining that the amplitude difference exceeds the predetermined threshold, confirm the user Talking to the electronic device at close range.

Preferably, the electronic device can also be operated to: define the microphone with the largest voice signal amplitude among the multiple microphones as the response microphone; and perform different processing on the user's voice input according to the different response microphones.

Preferably, the judging whether the user is speaking into the electronic device at close range includes: using a machine learning model trained in advance to process the sound signals of multiple microphones to judge whether the user is speaking into the electronic device at close range.

Preferably, the voice spoken by the user includes: the voice that the user speaks at a normal volume; the voice that the user speaks at a low volume; the voice that the user speaks in a non-voicing manner.

Preferably, the judged characteristics include volume, frequency spectrum characteristics, energy distribution, and the like.

Preferably, the electronic device is one of a smart phone, a smart watch, a smart ring, and a tablet computer.

According to another aspect of the present application, an electronic device with a built-in microphone has a memory and a central processing unit. The memory stores computer executable instructions. When the computer executable instructions are executed by the central processing unit, the following operations can be performed: Determine whether the voice signal collected by the microphone contains a voice signal; in response to confirming that the voice signal collected by the microphone contains a voice signal, determine whether the user is speaking in a low voice, that is, speaking at a lower than normal volume; in response to determining that the user is doing Speak in a low voice, and process the sound signal as a voice input without any wake-up operation.

Preferably, the whispered speech includes two ways of whispering without vocal cords and whispering with vocal cords.

Preferably, the electronic device is also operated: in response to determining that the user is speaking in a low voice; judging whether the user is speaking in a low voice without vocal cords or speaking in a low voice with vocal cords; Different treatment.

Preferably, the different processing is to activate different applications to respond to voice input.

Preferably, the signal characteristics used to determine whether the user is speaking in a low voice include volume, frequency spectrum characteristics, and energy distribution.

Preferably, the signal characteristics used when judging that the user is making a low-voiced speech without vocal cords or when making a low-voiced speech with vocal cords include volume, frequency spectrum characteristics, and energy distribution.

Preferably, the judging whether the user is speaking in a low voice includes: using a machine learning model to process a sound signal collected by a microphone to determine whether the user is speaking in a low voice.

Preferably, the machine learning model is a convolutional neural network model or a recurrent neural network model.

Preferably, the judging whether the user is doing a low-voiced speech without vocal cords or a low-voiced speech with vocal cords includes: using a machine learning model to process the sound signal collected by a microphone to determine that the user is doing low-voiced speech without vocal cords Or speaking in a low voice with vocal cords.

Preferably, the specific user is identified through voiceprint analysis, and only the voice signal containing the voice of the specific user is processed.

Advantages of this program:

1. The interaction is more natural. Putting the device in front of the mouth triggers voice input, which conforms to user habits and cognition.

2. The use efficiency is higher. It can be used with one hand. No need to switch between different user interfaces/applications, no need to hold down a button, just lift your hand to your mouth to use it.

3. The radio quality is high. The device's voice recorder is at the user's mouth, and the voice input signal received is clear, and it is less affected by environmental sounds.

4. High privacy and sociality. With the device in front of the mouth, the user only needs to make a relatively small sound to complete high-quality voice input, which has little interference to others. At the same time, the user's posture can include covering the mouth, which has better privacy protection.

Description of the drawings

Fig. 1 is a schematic flowchart of a voice input interaction method according to an embodiment of the present application;

FIG. 2 shows an overall flowchart of a method for triggering a voice input using differences in sound signals received by multiple microphones using an electronic device equipped with multiple microphones according to another embodiment of the present application;

FIG. 3 shows an overall flowchart of a voice input trigger method based on whispered speech recognition for an electronic device with a built-in microphone according to an embodiment of the present application;

Figure 4 depicts the overall flow chart of the voice input trigger method based on the distance judgment of the sound signal of the microphone;

FIG. 5 is a front schematic diagram of placing the upper microphone of the mobile phone close to the mouth in the triggering posture according to an embodiment of the present application;

6 is a schematic side view of the upper microphone of the mobile phone close to the mouth in the triggering posture according to an embodiment of the present application;

Fig. 7 is a schematic diagram of placing the lower end microphone of the mobile phone close to the mouth in the triggering posture according to an embodiment of the present application;

Fig. 8 is a schematic diagram of placing the smart watch microphone close to the mouth in the triggering posture according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the application, the application will be further described in detail below with reference to the drawings and specific implementations.

The present disclosure is directed to the voice input trigger of the smart electronic device, and determines whether to trigger the voice input application based on the inherent characteristics of the sound captured by the configured microphone. There is no need for traditional physical button triggering, interface element triggering, and wake-up word detection, and the interaction is more natural. Putting the device in front of the mouth triggers voice input, which conforms to user habits and cognition.

The present disclosure will be continued from the following aspects: 1. Voice input trigger based on the characteristics of wind noise when a human speaks, specifically, by recognizing the voice and wind noise sound when a human speaks to directly initiate the voice input and receive the sound signal As voice input processing; 2. Voice input trigger based on the difference of sound signals received by multiple microphones; 3. Voice input trigger based on whispered speech recognition; 4. Voice input trigger based on distance judgment of microphone's voice signal. .

1. Voice input trigger based on the characteristics of wind noise when human speaking

When the user speaks into the microphone at a close distance, even if the sound is small or does not trigger the vocal cords to speak, the sound signal collected by the microphone contains two sound components, one is the sound produced by the oral vibration of the human vocal cord, and the other is produced by human speech The wind noise generated by the airflow hitting the microphone. The voice input application of the electronic device can be triggered based on this characteristic.

Fig. 1 shows a schematic flowchart of a voice input interaction method 100 according to an embodiment of the present application.

In step S101, the sound signal collected by the microphone is analyzed to identify whether the sound signal contains human speech and whether it contains wind noise generated by the airflow generated by the human speech hitting the microphone,

In step S102, in response to determining that the sound signal contains the voice of a person's speech and the wind noise generated by the airflow generated by the user's speech hitting the microphone, the sound signal is processed as the user's voice input.

The voice input interaction method of the embodiment of the present application is particularly suitable for voice input without vocal cords when the privacy requirements are relatively high.

Here, the voice spoken by the user may include: the voice of the user speaking at a normal volume, the voice of the user speaking at a low volume, and the voice of the user speaking in a non-voicing manner.

In one example, the above-mentioned different ways of speaking can be recognized, and different feedbacks can be generated according to the recognition results. For example, normal speaking is to control the voice assistant of the mobile phone, whispered speaking is to control WeChat, and voiceless speaking is to make voice transcription notes.

As an example, the processing of using a sound signal as the user's voice input includes one or more of the following:

Store the sound signal to the storage medium on the electronic device;

Send the sound signal through the Internet;

Recognize the voice signal in the sound signal as text and store it in the storage medium on the electronic device;

Recognize the voice signal in the voice signal as text and send it out via the Internet;

Recognize the voice signal in the voice signal as text, understand the user's voice command, and perform the corresponding operation.

In one example, it also includes identifying a specific user through voiceprint analysis, and processing only the voice signal containing the voice of the specific user.

In an example, the electronic device is one of a smart phone, a smart watch, and a smart ring.

In one example, a neural network model is used to determine whether the voice signal of the user is included in the voice signal and the wind noise generated by the airflow hitting the microphone. This is just an example, other machine learning algorithms can be used.

In one example, recognizing whether the voice signal contains human speech and whether it contains wind noise generated by the airflow generated by human speech and hitting the microphone includes:

Identify whether the voice signal contains the user's voice;

In response to determining that the voice signal contains the voice spoken by the user, recognize the phonemes in the voice, and represent the voice signal as a phoneme sequence;

For each phoneme in the phoneme sequence, determine whether the phoneme is an exhaled phoneme, that is, when the user utters the phoneme, there is airflow out of the mouth;

Divide the sound signal into a sequence of sound segments according to a fixed window length;

Using frequency characteristics, identify whether each sound segment contains wind noise;

Compare the exhaled phoneme in the speech phoneme sequence with the segment identified as wind noise in the sound segment sequence, and compare the non-exhaled phoneme in the phoneme sequence with the wind noise segment. When the overlap between the exhaled phoneme and the wind noise segment is higher than a certain threshold , And when the degree of coincidence of the non-breathing phoneme and the non-wind noise segment is lower than a certain threshold, it is determined that the sound signal contains the wind noise sound generated by the airflow generated by the user's speech hitting the microphone.

In one example, recognizing whether the voice signal contains human speech and whether it contains wind noise generated by the airflow generated by human speech hitting the microphone includes:

Identify the sound characteristics of wind noise in the sound signal;

In response to determining that the sound signal contains wind noise, identifying that the sound signal contains a voice signal;

Recognizing the phoneme sequence corresponding to the voice signal in response to determining that the voice signal contains the voice signal;

According to the wind noise characteristics in the sound signal, calculate the wind noise characteristic intensity at each moment;

For each phoneme in the phoneme sequence, obtain the exhalation intensity of that phoneme according to a predefined data model;

By analyzing the consistency of wind noise features and phoneme sequence based on the Gaussian Mixture Bayesian model, when the coincidence degree is higher than a certain threshold, it is determined that the sound signal contains wind noise generated by the airflow generated by the user's speech hitting the microphone.

2. Voice input trigger based on the difference of sound signals received by multiple microphones

Fig. 2 shows an overall flow chart of a method for triggering a voice input using a difference in sound signals received by multiple microphones for an electronic device equipped with multiple microphones according to another embodiment of the present application.

An electronic device such as an electronic device with multiple microphones built in a mobile phone. The electronic device has a memory and a central processing unit. The memory stores computer executable instructions. When the computer executable instructions are executed by the central processing unit, the voice input trigger of this embodiment can be executed. method.

As shown in FIG. 2, in step S201, the sound signals collected by multiple microphones are analyzed.

In an example, the multiple microphones include at least 3 microphones to form a microphone array system, and the spatial position of the sound source relative to the smart device can be estimated by the time difference between the sound signal reaching each microphone.

The sound signal here includes, for example, the amplitude and frequency of the sound signal.

In step S202, based on the sound signals collected by multiple microphones, it is determined whether the user is speaking to the electronic device at a close range.

In one example, determining whether the user is speaking into the electronic device at close range includes:

Calculate the position of the user’s mouth relative to the microphone array using the time difference between the sound signals arriving at each microphone on the array,

When the distance between the user's mouth and the electronic device is less than a certain threshold, it is determined that the user is speaking to the electronic device at close range.

In an example, the distance threshold is 10 cm.

In step S203, in response to determining that the user is speaking to the electronic device at a close distance, the sound signal collected by the microphone is processed as the user's voice input.

In an example, processing the sound signal as the user's voice input includes:

According to the distance between the speaker's mouth and the electronic device, the user's voice input is processed differently. For example, when the distance is 0-3cm, the voice assistant is activated to respond to the user's voice input; when the distance is 3-10cm, the WeChat application is activated to respond to the user's voice input and send voice messages to friends;

Determine whether the voice signal collected by at least one microphone contains the voice signal of the user's speech,

In response to determining that the sound signal collected by at least one microphone contains the voice signal of the user's speaking, extracting the voice signal from the sound signal collected by the microphone,

When judging whether the amplitude difference of the voice signals extracted from the sound signals collected by different microphones exceeds a predetermined threshold,

In response to determining that the amplitude difference exceeds the predetermined threshold, it is confirmed that the user is speaking to the electronic device at close range.

In the above example, you can also include:

Define the microphone with the largest voice signal amplitude among multiple microphones as the response microphone,

Depending on the response microphone, the user's voice input is processed differently. For example, when the response microphone is the microphone at the bottom of the smartphone, activate the voice assistant on the smartphone; when the response microphone is the microphone at the top of the smartphone, activate the voice recorder function to record the user's voice to the storage device;

In one example, determining whether the user is speaking into the electronic device at close range includes: using a machine learning model trained in advance to process sound signals from multiple microphones to determine whether the user is speaking into the electronic device at close range. Generally, prepare training sample data, and then use the training sample data to train the selected machine learning model. In actual application (sometimes called testing), input the sound signals captured by multiple microphones (as test samples) into the machine learning model , The output obtained indicates whether the user is talking to the electronic device at close range. As an example, the machine learning model is, for example, deep learning neural network, support vector machine, decision tree, etc.

In an example, the voice spoken by the user includes: the user speaks at a normal volume, the user speaks at a low volume, and the user speaks in a silent manner.

In an example, the processing of using the sound signal as the user's voice input includes one or more of the following: storing the sound signal in a storage medium on the electronic device; sending the sound signal through the Internet; The voice signal is recognized as text and stored on the storage medium of the electronic device; the voice signal in the voice signal is recognized as text and sent out through the Internet; the voice signal in the voice signal is recognized as text, and the user’s voice command is understood, Take the appropriate action.

As an example, the electronic device is a smart phone, smart watch, smart ring, tablet computer, etc.

This embodiment uses the difference in sound signals between different built-in microphones to identify whether the user is speaking to the electronic device at a close distance, and then decides whether to start the voice input, which has the advantages of reliable recognition and simple calculation method.

3. Voice input trigger based on whispered speech recognition

Whispering refers to the way in which the speaking volume is less than the volume of normal speaking (such as a normal conversation with others). There are two ways to speak in a low voice. One is whispering without vibration of the vocal cords (commonly known as whispering), and the other is whispering where the vocal cords vibrate. In the low-voicing mode where the vocal cords do not vibrate, the sound produced mainly includes the sound of air passing through the throat, the sound of the mouth, and the sound of the tongue and teeth in the mouth. In the whispering mode of vocal cord vibration, the sound produced includes not only the sound produced in the whispering mode of vocal cord vibration without vibration, but also the sound produced by vocal cord vibration. However, compared to the way of speaking at a normal volume, the vibration of the vocal cords is less in the process of low-voice speech with vocal cord vibration, and the vocal cord vibration sound produced is smaller. The vocal cords do not vibrate The sound produced by whispering and the sound produced by vocal cord vibration have different frequency ranges and can be distinguished. Vocal cord vibration low voice speech and vocal cord vibration normal volume speech can be distinguished by the volume threshold. The specific threshold can be set in advance or by the user.

Example method: filter the voice signal collected by the microphone, and extract two parts of the signal, which are the sound component V1 generated by the vibration of the vocal cords and the sound V2 generated by the air through the throat, the mouth, and the tongue and teeth in the mouth. When the energy ratio of V1 and V2 is less than a certain threshold, it is determined that the user is speaking in a low voice.

Generally speaking, whispering can only be detected when the user is close to the microphone, such as when the distance is less than 30 cm. The definition of whispering in close-range situations as voice input is an interactive method that is easy to learn and understand and convenient for users to operate. It can avoid explicit wake-up operations, such as pressing a specific wake-up button or through voice Wake word. And this method will not be triggered by mistake in most practical use cases.

Fig. 3 shows an overall flow chart of a voice input trigger method based on whispered speech recognition for an electronic device equipped with a microphone according to an embodiment of the present application. An electronic device equipped with a microphone has a memory and a central processing unit, and computer executable instructions are stored on the memory. When the computer executable instructions are executed by the central processing unit, the voice input trigger method according to the embodiments of the present application can be executed.

As shown in FIG. 3, in step S301, it is determined whether the sound signal collected by the microphone contains a voice signal.

In step S302, in response to confirming that the voice signal collected by the microphone contains a voice signal, it is determined whether the user is speaking in a low voice, that is, speaking at a lower than normal volume.

In step S303, in response to determining that the user is speaking in a low voice, the sound signal is processed as a voice input without any wake-up operation.

Whispering can include two ways of whispering without vocal cords and whispering with vocal cords.

In one example, the voice input triggering method may further include: in response to determining that the user is speaking in a low voice, judging whether the user is making a low voice without vocal cords or a low voice speaking with vocal cords, according to different judgment results, The sound signal is processed differently.

As an example, different processing is to hand over voice input to different applications for processing. For example, normal speaking is to control the voice assistant of the mobile phone, speaking in a low voice is to control WeChat, and speaking without vocal cords is to make voice transcription notes.

As an example, the signal characteristics used to determine whether the user is speaking in a low voice may include volume, frequency spectrum characteristics, energy distribution, and so on.

As an example, it is determined that the signal characteristics used by the user when making a low-voiced speech without vocal cords or when making a low-voiced speech with vocal cords include volume, spectral characteristics, energy distribution, and the like.

As an example, determining whether the user is speaking in a low voice may include: using a machine learning model to process the sound signal collected by the microphone to determine whether the user is speaking in a low voice.

As an example, the machine learning model may be a convolutional neural network model or a recurrent neural network model.

As an example, judging whether the user is doing low-voicing without vocal cords or low-voicing with vocal cords includes: using a machine learning model to process the sound signals collected by the microphone to determine whether the user is doing low-voicing without vocal cords or Talk in a low voice with vocal cords.

Store the sound signal to the storage medium on the electronic device;

Send the sound signal through the Internet;

As an example, the voice input triggering method may also include: identifying a specific user through voiceprint analysis, and processing only the voice signal containing the voice of the specific user.

As an example, the electronic device may be a smart phone, smart watch, smart ring, etc.

Regarding the whispering mode and detection method, as an example, you can refer to the following references:

Zhang, Chi, and John HL Hansen. "Analysis and classification of speech mode: whispered through shouted. "Eighth Annual Conference of the International Speech Communication Association. 2007.

Meenakshi, G. Nisha, and Prasanta Kumar Ghosh. "Robust whisper activity detection using long-term log energy variation of sub-band signal." IEEE Signal Processing Letters 22.11 (2015): 1859-1863.

Fourth, the voice input trigger based on the distance judgment of the sound signal of the microphone

The following describes the overall flowchart of the voice input trigger method based on the distance judgment of the sound signal of the microphone with reference to FIG. 4.

As shown in FIG. 4, in step 401, the sound signal captured by the microphone is processed to determine whether there is a voice signal in the sound signal.

In step 402, in response to confirming that the voice signal exists in the voice signal, it is further determined whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold based on the voice signal collected by the microphone.

In step 403, in response to determining that the distance between the electronic device and the user's mouth is less than a predetermined threshold, the sound signal collected by the microphone is processed as a voice input.

In one example, the predetermined threshold is 10 cm.

The voice signal may include one or a combination of the following items: the sound produced by the user speaking at a normal volume; the sound produced by the user speaking in a low voice; the sound produced by the user's vocal cord without speaking.

In one example, the features used when judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold includes time domain features and frequency domain features in the sound signal, including volume and spectral energy.

In an example, the judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold includes: using a deep neural network model to process data collected by a microphone to determine whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold.

In an example, the judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold includes: recording the user's voice signal when the user is not making a voice input, and comparing the voice signal currently collected by the microphone with the voice when the user is not making a voice input. Signals are compared, and if it is determined that the volume of the voice signal currently collected by the microphone exceeds a certain threshold of the volume of the voice signal when no voice input is made, it is determined that the distance between the smart electronic device and the user's mouth is less than the predetermined threshold.

In an example, the voice input triggering further includes identifying a specific user through voiceprint analysis, and processing only the voice signal containing the voice of the specific user.

In one example, the electronic device is a smart phone, smart watch, smart ring, etc.

Figures 5 to 8 show a few cases where the user places the microphone of the smart electronic portable device close to the mouth, and the voice of the user will be used as the voice input. Among them, Figure 5 and Figure 6 are respectively the case where there is a microphone at the top of the mobile phone. In this case, when the user has voice interaction intentions, they can move the microphone of the mobile phone to 0-10 cm from the mouth, and speak directly as voice enter. Fig. 7 is the case where there is a microphone at the lower end of the mobile phone. Similar to the above-mentioned upper end with a microphone, the two postures are not mutually exclusive. If there are microphones at the upper and lower ends of the mobile phone, either posture can implement an interactive scheme. Fig. 8 shows the situation when the corresponding device is a smart watch, which is similar to the situation when the above-mentioned device is a mobile phone. The above description of the trigger gesture is exemplary, not exhaustive, and is not limited to the various devices and microphones disclosed.

As a specific implementation example of using a single microphone to receive voice input and trigger voice input, you can first analyze the voice input received by a single microphone to determine whether it is voice, and analyze the short-distance unique characteristics of the voice, such as microphone popping sound , Near-field wind noise, blowing sound, energy, spectrum characteristics, time domain characteristics, etc., to determine whether the distance between the electronic device itself and the user’s mouth is less than a given threshold, and to determine whether the voice input source belongs to a serviceable user through voiceprint recognition ，Combine the above points to determine whether to use the microphone signal as voice input.

As a specific implementation example of using dual microphones to receive sound input and trigger voice input, by analyzing the characteristic differences of the dual microphone input signals, such as energy characteristics and frequency spectrum characteristics, it is determined whether the sound position is close to one of the microphones, and the signal difference between the dual microphones In this way, the environmental noise is shielded, the voice is separated into the corresponding mono, and then through the feature analysis method of the single microphone, it is determined that the distance between the electronic device itself and the user’s mouth is less than a given threshold, and the voiceprint recognition is used to determine whether the voice input source belongs to It can serve the user, combining the above points to determine whether to use the signal as a voice input.

As a specific implementation example of using a multi-microphone array to receive voice input and trigger voice input, by comparing and analyzing the difference of the voice input signal received by different microphones, by separating the near-field voice signal from the environment, identifying and detecting whether the voice signal is Including voice, using multi-microphone array sound source localization technology to determine whether the distance between the position of the user’s mouth of the voice signal and the device is less than a predetermined threshold, and using voiceprint recognition to determine whether the voice input source belongs to a serviceable user, combining the above points To determine whether to use the signal as a voice input.

In one example, the smart electronic portable device analyzes the voice signal and detects that the pronunciation position is near itself, that is, the mobile device is located closer to the user's mouth. The smart electronic portable device uses the sound signal as a voice input, and according to the task and The context is different, combined with natural language processing technology to understand the user's voice input and complete the corresponding task.

The microphone is not limited to the foregoing examples, but may include one or a combination of the following items: a built-in single microphone in the device; a built-in dual microphone in the device; a built-in multi-microphone array in the device; an external wireless microphone; and an external wired microphone.

As mentioned above, the smart electronic portable device can be a mobile phone equipped with a binaural Bluetooth headset, a wired headset with a microphone, or other microphone sensors.

The smart electronic portable device can be a watch, a smart wearable device among smart rings and watches.

The smart electronic portable device is a head-mounted smart display device equipped with a microphone or a multi-microphone group.

In one example, after the electronic device activates the voice input application, a feedback output may be made, and the feedback output includes one of vibration, voice, and image, or a combination thereof.

The solutions of the various embodiments of the present application can provide one or more of the following advantages:

The embodiments of the present application have been described above, and the above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the described embodiments, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A smart electronic device equipped with a microphone is characterized in that the smart electronic portable device performs voice input-based interaction with a user as follows:

Process the sound signal captured by the microphone to determine whether there is a voice signal in the sound signal;

In response to confirming that there is a voice signal in the voice signal, based on the voice signal collected by the microphone, it is further determined whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold; and

In response to determining that the distance between the electronic device and the user's mouth is less than a predetermined threshold, the sound signal collected by the microphone is processed as a voice input.
The smart electronic device according to claim 1, wherein the predetermined threshold is 3 cm.
The smart electronic device according to claim 1, wherein the predetermined threshold is 1 cm.
The smart electronic device according to claim 1, wherein there is a proximity light sensor at the microphone of the electronic device, and the proximity light sensor determines whether an object is approaching the electronic device.
The intelligent electronic device according to claim 1, characterized in that there is a distance sensor at the microphone of the electronic device, and the distance between the electronic device and the user's mouth is directly measured by the distance sensor.
The smart electronic device according to claim 1, characterized in that it is determined whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold value based on the characteristics of the sound signal collected by the microphone.
The intelligent electronic device according to claim 1, wherein the voice signal includes one or a combination of the following items:

The sound made by the user speaking at a normal volume;

The user's whispered voice;

The sound produced by the user's vocal cords without speaking.
The intelligent electronic device according to claim 1, further comprising:

In response to determining that the user is speaking into the electronic device at close range,

Determine that the user is speaking in one of the following ways, including:

The user’s voice at normal volume,

The user’s voice at a low volume,

The sound made by the user when the vocal cord is silent; and

According to the different judgment results, the sound signal is processed differently.
The intelligent electronic device according to claim 8, wherein the different processing is activating different application programs to process voice input.
The intelligent electronic device according to claim 8, wherein the characteristics used in the judgment include volume, frequency spectrum characteristics, and energy distribution.
The smart electronic device according to claim 1, wherein the features used when judging whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold value includes time-domain features and frequency-domain features in the sound signal, including volume and spectral energy.
The smart electronic device according to claim 11, wherein said determining whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold comprises:

The voice signal collected from the microphone is passed through the filter to extract the voice signal,

Determine whether the energy of the voice signal exceeds a certain threshold,

In response to the voice signal strength exceeding a certain threshold,

It is determined that the distance between the electronic device and the user's mouth is less than a predetermined threshold.
The smart electronic device according to claim 1, wherein said determining whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold comprises:

The deep neural network model is used to process the data collected by the microphone to determine whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold.
The smart electronic device according to claim 1, wherein said determining whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold comprises:

Record the voice signal when the user is not making voice input,

Compare the voice signal currently collected by the microphone with the voice signal when no voice input is made,

If it is determined that the volume of the voice signal currently collected by the microphone exceeds a certain threshold of the volume of the voice signal when no voice input is made, it is determined that the distance between the smart electronic device and the user's mouth is less than the predetermined threshold.
The smart electronic device according to claim 1, wherein the processing of using the sound signal as the user's voice input includes one or more of the following:

Store the sound signal to the storage medium on the electronic device;

Send the sound signal through the Internet;

Recognize the voice signal in the sound signal as text and store it in the storage medium on the electronic device;

Recognize the voice signal in the voice signal as text and send it out via the Internet;

Recognize the voice signal in the voice signal as text, understand the user's voice command, and perform the corresponding operation.
The intelligent electronic device according to claim 1, further comprising identifying a specific user through voiceprint analysis, and processing only the voice signal containing the voice of the specific user.
The smart electronic device according to claim 1, wherein the electronic device is a smart phone, a smart watch or a smart ring.
A voice interactive wake-up method executed by a smart electronic device equipped with a microphone, which is characterized in that it includes the following operations for the smart electronic device to interact with a user based on voice input:

Process the sound signal captured by the microphone to determine whether there is a voice signal in the sound signal,

In response to confirming that there is a voice signal in the voice signal, further determining whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold based on the voice signal collected by the microphone;

In response to determining that the distance between the electronic device and the user's mouth is less than a predetermined threshold, the sound signal collected by the microphone is processed as a voice input.
A computer-readable medium, characterized in that computer-executable instructions are stored thereon, and when the computer-executable instructions are executed by a computer, a voice interactive wake-up method can be executed, the voice interactive wake-up method includes:

Process the sound signal captured by the microphone to determine whether there is a voice signal in the sound signal,

In response to confirming that there is a voice signal in the voice signal, further determining whether the distance between the smart electronic device and the user's mouth is less than a predetermined threshold based on the voice signal collected by the microphone;

In response to determining that the distance between the electronic device and the user's mouth is less than a predetermined threshold, the sound signal collected by the microphone is processed as a voice input.