WO2023006033A1 - Speech interaction method, electronic device, and medium - Google Patents

Abstract

A speech interaction method, an electronic device, and a medium. The speech interaction method comprises: where it is detected that the current speech interaction environment of an electronic device does not meet a speech recognition condition, determining whether the current interaction state of a user meets a lip reading recognition condition (S303); if so, acquiring a lip reading recognition result, which is obtained by recognizing, by means of lip reading recognition, a speech instruction of the user which is received by the electronic device (S304); and executing a function corresponding to the lip reading recognition result (S306). By means of the method, where it is determined that the speech instruction of the user has been difficult to recognize by means of lip reading recognition, whether lip reading recognition is used is further determined by determining whether the user is interacting with a voice assistant, such that the accuracy of speech instruction recognition can be effectively improved, thereby further improving the accuracy of executing the speech instruction of the user by the electronic device.

Description

Voice interaction method, electronic device and medium

This application claims the priority of the Chinese patent application with the application number 202110865871.X and the application title "Voice Interaction Method, Electronic Equipment and Media" filed with the China Patent Office on July 29, 2021, the entire contents of which are incorporated by reference in In this application.

technical field

The present application relates to the technical field of human-computer interaction, and in particular to a voice interaction method, electronic equipment and media.

Background technique

With the development of artificial intelligence technology, electronic devices such as robots have been widely used in catering, education, medical care, culture, smart home, finance, telecommunications and other industries, and can provide users with comprehensive artificial intelligence services.

Users can interact with electronic devices such as robots through touch screens, voice, remote control, etc. Among them, when the user uses voice to interact with the robot, the robot can recognize the user's voice command and execute it. For example, as shown in Figure 1, if the user wants to order the robot to perform the operation of "telling a story", he can issue a voice command of "telling a story". Story" operation.

But if it is in a noisy environment, when the user gives instructions to the robot through voice, the robot may not be able to receive or recognize the user's voice instructions and execute them. For example, because the current environment is too noisy, the robot cannot judge when the user's voice command ends, so it has been in a continuous listening state, or it cannot judge when the user's voice command starts, so it has been in a non-radio state, so it cannot listen to the user's voice Feedback such as command execution operations seriously affects user experience.

Contents of the invention

In order to solve the above-mentioned technical problem that the electronic device may not be able to receive or recognize the user's voice command and execute it in a noisy environment, the first aspect of the embodiment of the present application provides a voice interaction method, an electronic device and a medium. The method can be applied to electronic equipment, and the method includes:

When it is detected that the current voice interaction environment of the electronic device does not meet the voice recognition condition, determine whether the user's current interaction state meets the lip language recognition condition;

When it is determined that the user's current interaction state satisfies the lip language recognition condition, acquire the lip language recognition method obtained by identifying the user's mouth change characteristics acquired by the electronic device through the image acquisition device. Language recognition results;

Execute the function corresponding to the lip recognition result.

In the speech recognition method provided by the embodiment of the present application, if it is judged that it is difficult to recognize the user's voice command by using the speech recognition method, it further determines whether to use lip recognition by judging whether the user is interacting with the voice assistant; The method can effectively improve the accuracy rate of voice command recognition, thereby further improving the accuracy rate of the electronic device executing the user's voice command.

It can be understood that, in the embodiment of the present application, the noise value around the electronic device and the duration of the electronic device receiving sound, etc. all belong to the category of the voice interaction environment of the electronic device.

It can be understood that in the embodiment of the present application, the image acquisition device may be a camera device for collecting images, such as a camera.

It can be understood that in the embodiment of the present application, when the electronic device receives the user's voice instruction, it can simultaneously collect the user's voice and the user's mouth change features.

When it is determined that the current voice interaction environment of the electronic device satisfies the voice recognition condition, voice recognition is used to recognize the user's voice received by the electronic device to obtain a voice recognition result.

When it is determined that the current voice interaction environment of the electronic device does not meet the voice recognition conditions, but the user's current interaction state meets the lip recognition conditions, the user's mouth acquired by the image acquisition device of the electronic device can be detected by means of lip recognition. Identify the internal change features to obtain the lip language recognition results.

In a possible implementation of the first aspect above, the speech recognition conditions include:

The noise value around the electronic equipment is lower than the set noise value;

or;

In the case that the noise value around the electronic device is greater than or equal to the set noise value, the duration time for the electronic device to receive sound is greater than zero and less than the set time.

It can be understood that in some embodiments, judging whether the voice command of the user can be recognized by means of voice recognition can firstly determine whether the noise value around the electronic device is lower than the set noise value, and if so, it proves that the noise around the electronic device is lower than the set noise value. If the noise is small, the user’s voice command can be recognized through voice recognition; if not, it proves that the noise around the electronic device is already large, and the external environment is already in a noisy environment. At this time, it is further judged that the electronic device The continuous sound of the radio is greater than zero and less than the set time.

If the continuous sound received by the electronic device is greater than zero and less than the set time, it proves that the electronic device can still accurately determine the time point when the user's voice is cut off, and then confirm that the user's voice command can be recognized by voice recognition at this time. If the continuous sound received by the electronic device is greater than or equal to the first set value or less than zero, it means that the electronic device has been unable to accurately determine the time point when the user's voice is cut off, and then it is determined that it is difficult to use the voice recognition method at this time. identify.

In a possible implementation of the first aspect above, the voice recognition condition includes: the duration of the electronic device receiving sound is greater than zero and less than a set time.

It can be understood that, in some embodiments, the method of judging whether the voice command of the user can be recognized by voice recognition can be directly judging whether the duration of the electronic device receiving sound is greater than zero and less than the set time, and if so, prove The electronic device can still accurately determine the time point when the user's voice is cut off, and then determine that the user's voice command can be recognized by means of voice recognition at this time. If the continuous sound received by the electronic device is greater than or equal to the first set value or less than zero, it means that the electronic device has been unable to accurately determine the time point when the user's voice is cut off, and then infers that the external environment is too noisy, and determines that the voice recognition method is used at this time It has been difficult to recognize the user's voice commands.

In a possible implementation of the first aspect above, the lip recognition conditions include:

The user and the electronic device are in an interactive state within a set time.

It can be understood that when the user has been interacting with the electronic device, it means that the user still needs the electronic device to execute voice commands. If the user has not interacted with the electronic device, for example, has left, it means that the user no longer needs the electronic device The need for devices to execute voice commands.

In some embodiments, if the user has been interacting with the electronic device, the electronic device can capture the changes in the user's mouth more clearly, so that lip recognition can be used to identify the acquired changes in the user's mouth. Get the lip recognition result.

In a possible implementation of the first aspect above, the method for determining whether the user is in an interactive state with the electronic device within a set time includes:

determining whether a user interacting with the electronic device has changed within the set time;

Detecting whether the interaction intensity value between the user and the electronic device reaches a set intensity value;

When it is determined that the user who interacts with the electronic device within the set time has not changed, and the interaction intensity value between the user and the electronic device reaches the set intensity value, confirm the The user is in an interactive state with the electronic device;

Wherein, the interaction intensity value is related to the distance between the user and the electronic device and the face orientation of the user.

In some embodiments of the present application, before detecting the interaction intensity value between the user and the electronic device, it may be determined whether the user who interacts with the electronic device within the above-mentioned set time has changed. If no change has occurred, the current user may be As an object for subsequent detection of interaction strength values. When it is further determined that the interaction intensity value between the user and the electronic device reaches the set intensity value, it can be confirmed that the user is in an interactive state with the electronic device; Identify the changing features of the user's mouth to obtain lip recognition results.

It can be understood that in the embodiment of the present application, the interaction intensity value may be acquired based on the distance between the user's face and the electronic device, the orientation of the face, and the like within a set time. For example, if the distance between the user's face and the electronic device is relatively short within the set time, and the user's face faces the electronic device, the interaction intensity value is high, and vice versa.

It can be understood that the meaning of the interaction strength value mentioned in the embodiment of the present application is the same as that of the interaction willingness value, but the expression is different. And the set intensity value may be the second set value mentioned in the following embodiments.

In a possible implementation of the first aspect above, before performing the functional step corresponding to the lip recognition result, it further includes:

Confirm whether the lip recognition result is correct;

If it is confirmed that the lip recognition result is correct, execute the function corresponding to the lip recognition result.

In the embodiment of the present application, before executing the functional step corresponding to the lip word recognition result, it is confirmed again whether the lip word recognition result is correct, which can effectively improve the accuracy of the electronic device in executing the user's voice command.

In a possible implementation of the first aspect above, the method for confirming whether the lip recognition result is correct includes:

Ask the user whether it is necessary to perform the function corresponding to the lip recognition result;

When the user confirms that the function corresponding to the lip recognition result needs to be executed, it is confirmed that the lip recognition result is correct.

In a possible implementation of the first aspect above, at the same time as the step of confirming whether the lip language result is correct, it also includes:

Obtain the body movement characteristics of the user and the noise value around the electronic device.

In the embodiment of the present application, in the step of confirming whether the result of the lip language is correct, because the previous steps have confirmed that the current environment is noisy, the voice recognition method may not be able to accurately recognize the user's voice command, so when the voice assistant is performing When confirming, the visual recognition function can be turned on at the same time. The visual recognition function can obtain the characteristics of the user's body movements and facilitate the recognition of the user's reply through body movements. Indicates that the lip-reading result is confirmed to be correct.

In some embodiments, in order to further increase the accuracy of the electronic device's recognition of the user's voice command, it is also possible to enable the noise detection function in addition to the visual recognition function when performing voice confirmation to the user, so as to detect the surrounding environment noise in real time . When it is judged that the ambient noise is lower than the set value, it can be determined that the voice command of the user can be recognized by the voice recognition method at this time, and the user's confirmation command or other subsequent voices can be confirmed by voice recognition at this time. If the ambient noise is still higher than the set value, lip recognition or visual recognition or a combination of lip recognition and visual recognition is used to recognize the user's confirmation instruction or other subsequent voice instructions.

In a possible implementation of the foregoing first aspect, the electronic device is a robot.

In the speech recognition method provided by the first aspect of the embodiment of the present application, when it is judged that it is difficult to recognize the user's voice command by the speech recognition method, it further determines whether to use lip language by judging whether the user is interacting with the voice assistant. Recognition; the method can effectively improve the accuracy rate of voice command recognition, thereby further improving the accuracy rate of the electronic device for executing the user's voice command.

The second aspect of the embodiment of the present application provides an electronic device, including:

a memory for storing instructions to be executed by the one or more processors of the electronic device, and

The processor is one of the one or more processors of the electronic device, configured to execute the above voice interaction method.

The third aspect of the embodiments of the present application provides a computer-readable storage medium, where instructions are stored on the computer-readable storage medium, and when the instructions are executed, the computer executes the above voice interaction method.

The fourth aspect of the embodiments of the present application provides a computer program product, where the computer program product includes instructions, and when the instructions are executed, the computer executes the above voice interaction method.

Description of drawings

FIG. 1 shows a schematic diagram of a scene of a voice interaction method according to some embodiments of the present application;

Fig. 2 shows a schematic structural diagram of an electronic device according to some embodiments of the present application;

Fig. 3 shows a schematic flowchart of a voice interaction method according to some embodiments of the present application;

Fig. 4 shows a schematic scene diagram of a voice interaction method according to some embodiments of the present application;

Fig. 5 shows a schematic diagram of a scene of a voice interaction method according to some embodiments of the present application;

Fig. 6 shows a schematic flowchart of a voice interaction method according to some embodiments of the present application.

Detailed ways

The embodiment of the application discloses a voice interaction method, electronic equipment and media.

It can be understood that the electronic devices applicable to the embodiments of the present application may be various electronic devices with voice recognition functions, including but not limited to robots, laptop computers, desktop computers, tablet computers, smart phones, servers, wearable devices, Head-mounted displays, mobile email devices, portable game consoles, portable music players, reader devices, televisions with one or more processors embedded in or coupled to them, or other electronic devices with computing capabilities.

In addition, the speech recognition function of the above-mentioned electronic device can be implemented in the form of various applications, for example, appearing in the form of a voice assistant, or the speech recognition function is built into the application program of the electronic device, for example, for the application program , such as voice search in the Maps app.

For ease of description, the electronic device is used as a robot, and the speech recognition function is implemented as a robot's voice assistant as an example.

As mentioned above, users can control electronic devices such as robots through voice commands. However, in a noisy environment, if the voice assistant of the robot uses voice recognition to recognize the user's voice commands, the above-mentioned voice assistant may appear It is impossible to judge when the user's voice command ends, so it is always in the receiving state, or it cannot be judged when the user's voice command starts, so it has been in the non-radio state, and the user's voice command cannot be executed, which in turn affects the user's Use experience.

In order to solve the above problems, the embodiment of the present application provides a voice interaction method. When the voice assistant is awakened by the user, the voice assistant can detect the surrounding noise level through the noise detection function. If the noise level is higher than the set threshold, the current The voice recognition mode is switched to the lip language recognition mode, so that the electronic device can recognize the user's voice command through the lip language recognition technology, and execute the voice command.

For example, taking the scene shown in Figure 1 as an example, when user 001 sends out the voice command "tell a story", during the process of the user sending out the command, the voice assistant detects that the surrounding noise level is high through the noise detection function, which is higher than the set level. Threshold, for example, if the threshold is set to 70 decibels, and the noise value detected by the voice assistant through the noise detection function is 78 decibels, the voice assistant will switch the current voice recognition method to the lip recognition method, so that the voice assistant can use lip recognition. The technology recognizes the user's voice commands and executes the "storytelling" voice commands.

The above technology can recognize voice commands under certain circumstances, but in normal scenarios, the accuracy of lip recognition is generally lower than that of speech recognition. Therefore, in the above solution, the speech recognition method is converted to In the lip language recognition method, there may be scenarios where the voice recognition can still perform accurate recognition even though the surrounding environment is noisy. At this time, converting the voice recognition method to the lip language recognition method increases the risk of recognition errors.

Therefore, another voice interaction method is provided in the embodiment of the present application. This method does not directly switch the voice recognition mode to the lip language recognition mode after judging that the surrounding environment is too noisy through the noise detection function to obtain the lip language recognition result; It is first to judge whether it is determined that the speech recognition method cannot be used, and if it is determined that the speech recognition method cannot be used, it is judged whether the condition for adopting lip language recognition is satisfied. The lip recognition result is obtained only after it is judged that the lip recognition condition is met. Among them, judging whether the voice recognition method cannot be used is as follows:

In an implementable solution, it can be determined that the voice assistant may not be able to judge the end of the user's voice command because the external environment is too noisy by judging that the listening time of the voice assistant is too long, for example, exceeding the conventional setting value of the system The time point, or by judging that the radio cannot be received, it is possible to determine the start time point when the voice assistant cannot judge the user's voice command because the external environment is too noisy, and then determine that it is difficult to accurately recognize the user's voice command through voice recognition at this time, Confirm that the user cannot be recognized by voice recognition at this time.

In another practicable solution, the surrounding environment can be judged first through the noise detection function. If the noise value of the surrounding environment is less than the set noise value, it is directly determined that the speech recognition method can be used at this time. If the noise of the surrounding environment The value is greater than or equal to the set noise value. At this time, it is further judged whether the voice assistant has been listening for too long, for example, it has exceeded the system’s conventional setting value, or whether it is unable to listen. It is determined that the external environment is too noisy for the voice assistant to judge the user’s voice commands. The end time point or the start time point of the system, and then determine the user's voice instruction that is difficult to accurately recognize through voice recognition at this time.

After it is determined that it is difficult to recognize the user's voice command by voice recognition, determine whether the user's face is facing the camera within the set time period, whether the face is within the shooting range of the robot camera, etc., to confirm whether the user is in and The way the voice assistant interacts confirms whether the lip recognition result is used. If it is confirmed that the user is interacting with the voice assistant, it can be determined that the lip recognition method can relatively accurately recognize the user's voice command, and then obtain the lip recognition result , and execute the function corresponding to the lip recognition result according to the lip recognition result.

For example, take the scene shown in Figure 1 as an example. When the voice assistant is awakened, it enters the voice recognition mode and starts to listen. When user 001 sends out the voice command of "tell a story", the voice assistant does not detect the user's voice. The instruction has ended, and the audio has been continuously receiving. When the audio receiving time exceeds the system setting value, such as 10 seconds, the voice assistant can judge that the voice assistant cannot judge the end of user 001’s voice instruction because the external environment is too noisy. Time point, and then determine the user's voice instructions that are difficult to accurately recognize through voice recognition at this time. Then detect whether the face of the user 001 has been facing the camera within the set time during the radio reception just now, and whether the face has always been within the shooting range of the robot camera. If the detection result is yes, it is determined that the user 001 is interacting with the electronic device. Therefore, it can be determined whether the voice command of user 001 can be recognized relatively accurately by means of lip recognition, and then the voice command of user 001 can be recognized by lip recognition, and the voice command of "telling a story" can be executed.

The speech recognition method provided by the embodiment of the present application first judges that the voice assistant has received the sound for a long period of time, and judges that the voice assistant may not be able to judge when the voice command of user 001 has ended, thereby judging the external environment It is too noisy; furthermore, it can be more accurately judged that it is difficult to recognize the voice command of user 001 by using voice recognition in this case, and further determine whether to use lip recognition by judging whether user 001 is interacting with the voice assistant ; It can effectively avoid the situation that the accuracy rate of the recognition result is reduced by using the lip language recognition result when the speech recognition can be performed, and effectively improve the accuracy rate of the speech command recognition.

Before introducing another voice interaction method provided in the embodiment of the present application in detail, the electronic device provided in the embodiment of the present application will be introduced first.

For ease of introduction, the electronic device is robot 002 as an example for illustration. It should be understood that the robot 002 in the embodiment of the present application can also interact with the cloud server, and send the cloud command of the identified user 001 to the cloud server, and the cloud server can use the database to feed back the interaction content to the robot 002. The interaction content is as follows: Songs, stories, etc.

As shown in FIG. 2 , the robot 002 may include a processor 110 , a power module 140 , a memory 180 , a sensor module 190 , an audio module 150 , a camera 170 , an interface module 160 , buttons 101 , and a display screen 102 .

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the robot 002 . In other embodiments of the present application, the robot 002 may include more or fewer components than shown in the illustration, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, may include a central processing unit CPU (Central Processing Unit), an image processor GPU (Graphics Processing Unit), a digital signal processor DSP, a neural network processor (neural-network processing unit, NPU), microprocessor MCU (Micro-programmed Control Unit), AI (Artificial Intelligence, artificial intelligence) processor or programmable logic device FPGA (Field Programmable Gate Array) and other processing modules or processing circuits. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. A storage unit may be provided in the processor 110 for storing instructions and data. In some embodiments, the storage unit in processor 110 is cache memory 180 .

It can be understood that in the embodiment of the present application, the processor 110 may control a corresponding program to execute the voice interaction method provided in the embodiment of the present application. Specifically, an artificial intelligence processor can be used to recognize the received voice to obtain a recognition result; the above-mentioned image processor can be used to analyze the collected lip movements of user 001 to obtain a recognition result; meanwhile, the above-mentioned image processor can be used Recognize the collected body movements of user 001, and obtain the recognition result. In addition, the processor 110 may be used to detect the noise around the electronic device in real time, so as to select a more accurate identification method.

The power module 140 may include a power supply, power management components, and the like. The power source can be a battery. The power management component is used to manage the charging of the power supply and the power supply from the power supply to other modules. In some embodiments, the power management component includes a charge management module and a power management module. The charging management module is used to receive charging input from the charger; the power management module is used to connect the power supply, the charging management module and the processor 110 . The power management module receives power and/or input from the charging management module, and supplies power to the processor 110 , the display screen 102 , the camera 170 , and the wireless communication module 120 .

The wireless communication module 120 may include an antenna, and transmit and receive electromagnetic waves via the antenna. The wireless communication module 120 can provide applications on the robot 002 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system ( Global navigation satellite system (GNSS), frequency modulation (frequency modulation, FM), near field communication (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The robot 002 can communicate with the network and other devices through wireless communication technology. For example, the robot 002 can communicate with the cloud server through the wireless communication module 120 .

The display screen 102 is used for displaying human-computer interaction interfaces, images, videos, and the like. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In the embodiment of the present application, the display screen 102 may be used to display various application program interfaces of the robot 002 .

The sensor module 190 may include a proximity light sensor, a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

The audio module 150 is used to convert digital audio information into analog audio signal output, or convert analog audio input into digital audio signal. The audio module 150 may also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be set in the processor 110, or some functional modules of the audio module 150 can be set in the processor 110. In some embodiments, the audio module 150 may include a speaker, an earpiece, a microphone, and an earphone jack. In the embodiment of the present application, the audio module 150 can be used to receive voice instructions from the user 001. In practice, the audio module 150 can also be used to perform operations such as playing music and telling stories according to the voice instructions of the user 001.

Camera 170 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP (Image Signal Processing, Image Signal Processing) to convert it into a digital image signal. The robot 002 can realize the shooting function through ISP, camera 170, video codec, GPU (Graphic Processing Unit, graphics processor), display screen 102 and application processor. In the embodiment of the present application, the camera 170 can obtain user 001's face image, lip movement image, etc. of user 001.

The interface module 160 includes an external memory interface, a universal serial bus (universal serial bus, USB) interface, and the like. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the robot 002. The external memory card communicates with the processor 110 through the external memory interface to realize the data storage function. The universal serial bus interface is used for the robot 002 to communicate with other electronic devices 002 .

In some embodiments, the robot 002 further includes a button 101 . Wherein, the key 101 may include a volume key, an on/off key, and the like.

Another voice interaction method according to the embodiment of the present application will be described in detail below in conjunction with the above-mentioned robot 002 . FIG. 3 shows a schematic diagram of a voice interaction method, wherein the voice interaction method shown in FIG. 3 can be executed by a voice assistant of the robot 002 . As shown in Figure 3, the voice interaction method shown in Figure 3 includes:

S301: After detecting that the user 001 wakes up the voice assistant, enter the voice recognition mode. In the embodiment of the present application, the speech recognition mode is a mode for recognizing the speech instruction of the user 001 received by the robot 002 .

After the voice assistant is woken up, the voice assistant will turn on the sound recording and obtain the characteristics of user 001's mouth changes. Among them, turning on the sound recording can facilitate the determination in the subsequent steps that when the voice command of user 001 is recognized by voice, it can directly Recognize the received sound. Acquiring the mouth change features of the user 001 can facilitate the recognition of the acquired user 001's mouth change features when it is determined in the subsequent steps that the voice command of the user 001 is recognized by lip language.

In the embodiment of the present application, the receiving of sound can be realized based on the microphone of the robot 002, and the acquisition of the mouth change characteristics of the user 001 can be realized based on the camera device of the robot 002.

It can be understood that, in some embodiments, the voice assistant of the robot 002 needs to be woken up after receiving the wake-up word from the user 001, so as to enter the voice recognition mode. For example, if the wake-up word of the voice assistant is "Hi, Xiaoyi", when the user 001 speaks the wake-up word of "Hi, Xiaoyi", the voice assistant will enter the voice recognition mode, turn on the radio and obtain the user's mouth change characteristics, In order to receive voice instructions from user 001.

In the embodiment of the present application, during the operation of the voice recognition mode, the voice recognition function and the lip language recognition function can be selectively enabled. As described below:

In an implementable solution, the voice recognition function and the lip language recognition function can be turned on during the entire process of receiving sound, so that the received sound can be directly recognized by voice recognition in real time to obtain the voice recognition result, and the captured User 001's mouth features are directly recognized in real time by means of lip recognition to obtain lip recognition results. When the subsequent judgment result is that the speech recognition method is adopted, the speech recognition result can be obtained directly without performing speech recognition again. If the subsequent judgment result is that the lip recognition method is adopted, the lip recognition result can be obtained directly without performing lip recognition again, which effectively saves the time for recognizing the voice of user 001.

In the second implementable solution, after entering the speech recognition mode, it can be judged in the subsequent steps whether to use the lip recognition method or the speech recognition method to recognize the speech command of user 001, and then turn on the speech recognition function Or lip recognition function, this solution can effectively reduce unnecessary speech recognition or lip recognition calculations.

In the third implementable solution, the voice recognition function can be turned on during the whole process of radio reception, and the lip recognition function can be turned on when it is later judged that lip recognition is needed for recognition. The setting of this implementation is because most scenarios still use voice recognition to recognize the voice commands of user 001, so turning on the voice recognition function for a long time can effectively avoid multiple conversions of turning on and off the voice recognition function, reducing most common scenarios. The calculation amount of the processor 110 in the processor 110 is increased, and the running speed of the processor 110 is increased. However, the lip recognition method only occurs in a small number of scenes. Therefore, when the lip recognition method is determined to be used, the lip recognition function can be effectively reduced. The computational load of the speech recognition process.

It can be understood that in the embodiment of the present application, the speech recognition function can be implemented based on the artificial intelligence processor of the robot 002 . Wherein, the artificial intelligence processor can intelligently recognize the voice of user 001. The lip language recognition function can be realized based on the image processor of the robot 002, in which the image processor can continuously recognize faces from the images, judge the person who is speaking, and extract the person's continuous mouth shape change characteristics; The features of the lip language recognition model in the image processor are input to recognize the pronunciation corresponding to the speaker's population type; then, according to the recognized pronunciation, the most likely natural language sentence is obtained.

It is understandable that the recognition rate of the lip language recognition method is not high in general scenarios. However, in a vertical scenario (that is, a scenario where some keywords trained by the model are recognized), the recognition accuracy rate is relatively high, for example, it can reach more than 90%.

S302: Detect whether the condition for recognizing the user's voice instruction by means of voice recognition is satisfied. If it is satisfied, then it shows that the speech recognition method can be used to recognize the user's voice command, and then proceeds to S308 to obtain the speech recognition result; if not satisfied, it shows that the speech recognition method can no longer be used to accurately recognize the user's voice command, then Go to S303, and check whether the condition for entering the lip recognition mode is met.

In the embodiment of the present application, the voice assistant can collect sound by controlling the microphone, and can judge the time point when user 001's voice starts or is cut off by using voice boundary detection (Voice activity detection, Vad) technology.

In a practicable solution, the condition for recognizing the voice command of user 001 by means of voice recognition may be that the voice assistant has already received the voice and the continuous voice is less than the first set value.

Among them, the first setting value can be determined according to the relevant performance of the device and the regular voice duration of user 001. For example, although the voice device exceeding 10s can still be recognized, the dialogue system of the device can no longer give an effective answer; or the user 001 generally does not take more than 10 seconds to issue voice commands, and there is usually a pause within 10 seconds. When the continuous radio reception exceeds 10 seconds, the VAD technology of the device can no longer accurately identify the truncation time point of the human voice in the audio. Therefore, the first set value can be set to 10 seconds.

It can be understood that in the embodiment of the present application, in one case, when the voice assistant continues to listen for more than the first set value, it can be judged that the use of voice boundary detection technology has been unable to accurately determine the time point when the voice of user 001 ends, so the voice assistant Just kept on listening. Therefore, it is determined that the external environment is too noisy, and it can be determined that it is difficult to recognize the voice instruction of the user 001 through voice recognition at this time.

It can be understood that in the embodiment of the present application, turning on the sound collection may refer to turning on the sound collection function and allowing sound collection, but in some cases, it may still be impossible to receive sound. For example, the use of voice boundary detection technology has been unable to accurately determine the time point when user 001's voice started, so the voice assistant may not be able to receive the voice.

Therefore, in another case, when the voice assistant has been unable to receive sound, it can be judged that the voice boundary detection technology has been unable to accurately determine the time point when the voice of user 001 starts, so the voice assistant has been unable to receive sound. Therefore, it can be determined that the external environment is too noisy, and it can be determined that it is difficult to recognize the voice instruction of the user 001 through voice recognition at this time.

The above solution is to confirm that the voice assistant has started to receive sound and the continuous sound is less than or equal to the first set value, so as to confirm that the use of VAD technology can still accurately judge the time point when the user 001's voice starts and is cut off, and then determine that it can be used at this time. The speech recognition mode recognizes the speech instruction of user 001.

In another practicable solution, the condition for recognizing the voice command of the user 001 by means of voice recognition may be that the noise value of the surrounding environment is less than a set value.

It should be noted that when the noise value of the surrounding environment is greater than the set value, it is judged that the external environment is noisy at that time, but at this time it is not directly judged that the voice recognition method cannot be used, but to further judge whether the duration of the voice assistant’s radio reception is less than The first set value, as mentioned above, indicates that the VAD technology can still accurately determine the time point when the voice of user 001 is cut off, and then determine that the voice command of user 001 can be recognized by voice recognition at this time. If the duration of the voice assistant’s sound collection is greater than or equal to the first set value, as mentioned above, it indicates that the VAD technology has been unable to accurately determine the time point when the voice of user 001 is cut off, and then it is determined that the speech recognition method can be used at this time and the user 001 cannot be accurately recognized. voice commands.

S303: Detect whether the condition for recognizing the user's voice command by means of lip recognition is satisfied.

If satisfied, it indicates that the user's voice command can be recognized by using the lip language recognition method, and then go to S304 to obtain the lip language recognition result; if not satisfied, it indicates that the lip language recognition method can not be used to recognize the user's voice command. , go to S307, and prompt the user that the identification fails.

It can be understood that, in some embodiments of the present application, after the voice assistant detects that the user 001 has kept interacting with the voice assistant, for example, the user 001 before the robot 002 has remained unchanged, and the face has been facing the camera device of the robot 002 and Only when the distance between the robots 002 and 002 is within the set range, it is considered that the conditions for entering the lip recognition mode are met. Because if the user 001 in front of the robot 002 is always the same user and the user’s face is always facing the camera device of the robot 002 and the distance between the user and the robot 002 is within the set range, the movement of the user’s mouth can be accurately captured at this time, At this time, the voice command of the user 001 can be recognized more accurately by using the lip recognition method.

S304: Obtain a lip recognition result.

In this embodiment of the present application, the lip recognition result may be the instruction keyword included in the user 001 instruction. Among them, the above-mentioned keywords can be the conventional instruction keywords that have been stored in the voice assistant and have been trained by the model, for example, tell a story, read a picture book, play music, tell a joke, exit, return, etc. It is understandable that because the above-mentioned keywords have been Stored in the voice assistant, therefore, the above-mentioned instruction keywords can be accurately identified by using the lip recognition method.

In some embodiments, in order to avoid misrecognition when command words are mixed into long sentences, the keyword may be used as the result of lip recognition only when there is a pause before and after the keyword is recognized.

For example, if user 001 sends the voice command of "tell a story" first, and then says the voice content of "I heard that this robot 002 also has the function of playing music", user 001 actually wants robot 002 to tell the story. Stories, not music. At this time, if the voice assistant uses the two keywords of "telling a story" and "playing music" as the result of lip recognition, it will fail to confirm that the lip recognition of "telling a story" and "playing music" should be performed. Which lip recognition result in the result corresponds to the function, or directly executes the function corresponding to the obtained second lip recognition result "play music". And if the voice assistant can only have a pause interval before and after the keyword is recognized, the keyword is used as the lip language recognition result. Then, "telling a story" can be directly used as the lip recognition result, so as to execute the function corresponding to the lip recognition result of "telling a story". This scheme can effectively avoid the misrecognition that may be caused by command words appearing in long sentences.

S305: confirm whether the lip language recognition result is correct, if the result is yes, it shows that the lip language recognition is accurate, then go to S305; if the result is no, it shows that the lip language recognition is incorrect, then go to S306;

In order to further confirm whether the lip recognition result is accurate, the voice recognition method provided in the embodiment of the present application may include a step of confirming the voice command to the user 001 . The confirmation of the voice command to the user 001 may be to ask the user 001 whether he wants the voice assistant to perform the function corresponding to the lip recognition result. For example, as shown in Figure 4, if the recognized keyword is "telling a story", the way to confirm the voice command to user 001 can be to ask user 001, wherein the content of the inquiry can be: "Do you want to Shall I tell you a story?" and other inquiries.

When the user 001 confirms the voice command, as shown in FIG. 5 , the confirmation may be indicated by voice answering "Yes".

In the embodiment of this application, because the previous steps have confirmed that the current environment is noisy, the voice recognition method may not be able to accurately recognize the voice command of user 001. Therefore, when the voice assistant confirms to user 001, the visual recognition function can be turned on at the same time. It is convenient to recognize the confirmation performed by the user 001 through body movements, for example, the user 001 may express confirmation through a nodding action or an ok gesture.

The visual recognition function can be a function that can detect the body movements of the user 001, and the visual recognition function can be realized based on the image processor of the robot 002. Body movements are parsed to obtain visual recognition results. For example, after the nodding action image of user 001 is collected by the image processor, the nodding action image can be analyzed, and the recognition result obtained can be the text corresponding to the nodding action, such as "confirm", "yes", etc. .

In some embodiments, in order to further increase the accuracy of the voice assistant for user 001's voice command recognition, the voice assistant can also enable the noise detection function in addition to the visual recognition function when performing voice confirmation to user 001, so that real-time Detect ambient noise. When it is judged that the surrounding environment noise is lower than the set value, it can be determined that the voice command of user 001 can be recognized by the voice recognition method at this time, and then the voice command of user 001 or the subsequent confirmation command can be confirmed by voice recognition. Other voice commands are recognized; if the ambient noise is still higher than the set value, lip recognition or visual recognition or a combination of lip recognition and visual recognition is used to confirm the user 001's command or other subsequent voice commands to identify.

S306: Execute a function corresponding to the lip recognition result based on the lip recognition result.

In the implementation of this application, as shown in Figure 5, if the lip recognition result is "telling a story", after user 001 confirms that the lip recognition result is correct, the voice assistant can execute the function corresponding to "telling a story".

In some embodiments, during the process of executing the task corresponding to the voice command of the user 001, the visual recognition function can be continuously turned on to keep the recognition result of the body movement of the user 001 by the voice assistant.

For example, in the process of performing the task of "telling a story", the lip recognition function can be continuously turned on to obtain the lip recognition results of user 001. Body action recognition results. For example, the user 001 makes a five-finger gesture within the range captured by the camera device of the robot 002 to indicate that the task of storytelling is stopped, and the voice assistant can recognize the gesture and stop the execution of the task.

It can be understood that in the embodiment of the present application, when the visual recognition function is turned on, the voice recognition function and the lip language recognition function are both turned on at the same time.

In other embodiments, in order to further increase the accuracy of the voice assistant for user 001's voice command recognition, in the process of performing the task corresponding to user 001's voice command, in addition to turning on the visual recognition function, you can also turn on the noise as described above. Detection function for real-time detection of ambient noise. After judging that the surrounding environment noise is lower than the set value, it can be determined that the voice command of user 001 can be recognized by using the voice recognition mode at this time, and then the voice recognition mode can be used at this time; if the surrounding environment noise is still higher than the set value value, the lip recognition result method or the visual recognition method or the combination of the lip recognition method and the visual recognition method is used to accurately recognize other voice commands of user 001.

For example, when the voice assistant performs the task of "telling a story", the voice assistant simultaneously turns on the visual recognition function, the lip language recognition function and the noise detection function. At a certain moment, the voice assistant detects that the surrounding environment noise is lower than the set value Finally, if it is determined that the voice command of user 001 can be recognized by the voice recognition method at this time, then the voice command of user 001 in the task process is recognized by the voice recognition method. For example, user 001 has switched to using the The voice recognition method recognizes the voice command of user 001 during the task, and issues the command "stop telling the story". ", and execute the function corresponding to the voice recognition result of "stop telling the story".

It can be understood that in the embodiment of the present application, the voice recognition function and the lip language recognition function are always on during the process of the above-mentioned voice assistant executing the voice command of user 001.

S307: Remind the user that the identification of 001 fails.

In some embodiments, the way of reminding the user 001 of the recognition failure may be to display prompt information such as "recognition error" and "unrecognizable" on the screen of the robot 002 .

In some embodiments, the method of reminding the user 001 of the recognition failure may also be to prompt the user 001 through voice messages such as "recognition error" and "unrecognizable".

In some embodiments, after reminding the user 001 that the recognition fails, the user 001 can also be reminded to face the camera, raise the voice, etc. to remind the user 001 to issue the voice command again.

S308: Obtain a voice recognition result.

In the embodiment of the present application, after the speech recognition result is acquired, a function corresponding to the speech recognition result may be executed based on the speech recognition result.

In the embodiment of the present application, the speech recognition method provided in FIG. 3 first judges that the voice assistant has received the sound for a long period of time, and judges that the voice assistant may not be able to judge when the voice command of user 001 has ended, thereby It is determined that the external environment is too noisy, and it is further determined that it is difficult to recognize the voice command of the user 001 by means of voice recognition under such circumstances. Further, by judging whether the user 001 is interacting with the voice assistant to determine whether to use the lip recognition method; it can effectively avoid the situation that the accuracy of the recognition result is reduced by using the lip recognition result when speech recognition is possible, Effectively improve the accuracy of voice command recognition. In addition, the user 001 can be asked again after the lip language recognition result is obtained, which can effectively ensure the accuracy of the recognition result.

In the embodiment of the present application, judging in step 303 whether the user 001 has been maintaining the interactive state with the voice assistant can be judged from the following aspects:

First, whether the user 001 interacting with the voice assistant remains unchanged during the listening process.

Wherein, if the voice assistant detects that the user 001 who interacts with the voice assistant during the listening process has not changed, the possibility that the user 001 is interacting with the voice assistant is relatively high.

If the user 001 who interacts with the voice assistant has changed, the user 001 who may have issued the voice command has left. At this time, in some embodiments, it can be directly determined that the received voice is invalid. In some other embodiments, it is also possible to detect the voice command of the last user 001 who interacted with the voice assistant during the listening process. For example, the voice assistant detects that the object interacting with the voice assistant has been changed once during the listening process, that is During the sound collection process, two users 001 have interacted with the voice assistant, and the voice instruction of the second user 001 who interacts with the voice assistant during the sound collection process is detected.

In the embodiment of the present application, it is possible to detect whether the user 001 interacting with the voice assistant has changed through the face tracking technology.

In some embodiments, if some electronic devices do not have face tracking technology and do not have the ability to track faces, the user interacting with the voice assistant can be detected by detecting whether the face of user 001 in front of the voice assistant has changed 001 has changed.

Second, whether the interaction willingness value between the user 001 and the voice assistant reaches the second set value.

In some embodiments, the interaction willingness value may be calculated based on the distance between the face of the user 001 and the voice assistant, the orientation of the face, and the like within a period of time. For example, if the distance between the face of the user 001 and the voice assistant is relatively short within a period of time, and the face of the user 001 is facing the voice assistant, the interaction willingness value is higher, and vice versa.

Specifically, in a possible implementation, the voice assistant can obtain the face angle of user 001 and the distance between user 001 and robot 002 by collecting images of user 001 within a period of time, and then according to the user 001's The face angle and the distance between the user 001 and the smart device are used to obtain the interaction willingness value of the user 001 through the interaction willingness value model. The higher the interaction willingness value, the greater the interaction intensity between user 001 and the voice assistant.

Among them, in the interaction willingness value model, it can be defined that different face angles correspond to different values, the distance between user 001 and robot 002 corresponds to different values, and the value corresponding to the face angle is the same as the distance between user 001 and robot 002. The value corresponding to the distance can be assigned different weights. For example, the angle of the face can better reflect whether the user 001 is interacting with the voice assistant, and the weight corresponding to the angle of the face can account for 60%. User 001 and robot 002 The weight corresponding to the distance between them may be 40%.

It can be understood that when the interaction willingness between user 001 and the electronic device is weak, the distance between user 001 and the electronic device is relatively long, and the face angle deviates to a certain extent from the face angle facing the electronic device. 001's lip movements are accurately captured and recognized. Therefore, it is difficult to recognize user 001's voice commands by lip language recognition at this time. Conversely, when user 001 has a strong willingness to interact with the electronic device, the distance between user 001 and the electronic device is relatively close, and the face angle is close to or equal to the angle facing the electronic device. Lip movement is accurately captured and recognized, therefore, at this time, the voice command of user 001 can be accurately recognized by means of lip recognition.

In some embodiments, in order to judge more accurately whether lip recognition is used to obtain recognition results, the judgment conditions in steps 302 and 303 in FIG. 3 can be sorted and supplemented, wherein the specific speech recognition method is shown in As shown in 6, step 301 and steps 304-308 refer to the foregoing, and will not be repeated here. The following describes steps 302-303 in detail. Specifically, steps 302-303 can be adjusted as follows:

S302A: Determine whether there is a human voice in the received sound.

If the judgment result is yes, it means that user 001 has issued a voice command, go to S302B; if the judgment result is no, it shows that there is no user 001 to send a voice command, then go to S302C, start combining and receiving sound again, and re-test .

In the embodiment of the present application, the human voice detection model in the artificial intelligence processor can be used to detect whether there is a human voice in the received sound. If there is a human voice, execute S302B to further determine whether the voice assistant detects that the duration of sound collection is less than the first set value. If there is no human voice, then after the interval setting time, go to S302C, start receiving sound again, and recalculate the duration of sound collection. For example, the interval setting time can be 200ms.

S302B: Judging whether the condition for recognizing the user's voice command by voice recognition is met;

If the judgment result is yes, then it shows that the speech recognition method can be used to identify the user's voice command, and then proceed to S308 to obtain the speech recognition result; If it is recognized, go to S303A to check whether the conditions for entering the lip recognition mode are met.

Wherein, the conditions for voice command recognition are as described in step S302 in FIG. 3 , and will not be repeated here.

S302C: Start receiving sound again, and recalculate the duration of receiving sound.

S303A: Determine whether the user 001 of face tracking has not changed.

If the judgment result is yes, it means that the user who interacts with the voice assistant during the sound collection process is always the same user, then go to S303B, and use the user as the user who interacts with the voice assistant; if the judgment result is no, it means that during the sound collection process If the user interacting with the voice assistant has been changed, go to S303C, and use the last user 001 collected by the camera device as the user 001 interacting with the voice assistant.

It can be understood that when judging whether the user is interacting with the voice assistant, it is necessary to first determine the user who is interacting with the voice assistant, because during a period of radio recording, because the environment is too noisy, the VAD technology is satisfied with the time point of judging the human voice, so it is possible It appears that multiple users interacted with the voice assistant during this listening process, that is, the users who interacted with the voice assistant may have changed. For example, two users may have interacted with the voice assistant in turn during this listening process, and The user before the replacement may have left, so the second user after the replacement can be used as the user interacting with the voice assistant, and lip recognition is performed on the mouth features of the second user.

S303B: The current user is used as the user interacting with the voice assistant. Wherein, the current user is a user who has been interacting with the voice assistant during the listening process.

S303C: Use the last user 001 captured by the camera device as the user 001 interacting with the voice assistant.

S303D: Determine whether the value of the interaction willingness of the user 001 who interacts with the voice assistant reaches the first set value.

If the judgment result is yes, it indicates that the user 001 is interacting with the device, then go to S305 to obtain the lip recognition result; if the judgment result is no, it indicates that the user 001 is not willing to interact with the voice assistant, and the lip recognition is adopted If the method is also difficult to recognize the user's voice command, go to step S307 and prompt the user that the recognition fails.

The speech recognition method shown in FIG. 6 of the present application can sort several judgment conditions for using lip recognition, and can more accurately judge the timing of using lip recognition. In addition, when it is judged that there is no human voice in the radio, the current process can be ended in advance, and the next round of detection can be started, which avoids adding unnecessary subsequent recognition steps and effectively improves the recognition efficiency.

To sum up, the voice recognition method provided by the embodiment of the present application first judges that the voice assistant has received voice for a long period of time, and judges that the voice assistant may not be able to judge when the voice command of user 001 has ended, so as to determine It is found that the external environment is too noisy, and it is judged that it is difficult to recognize the voice command of the user 001 by means of voice recognition under such circumstances. Further, by judging whether the user 001 is interacting with the voice assistant to determine whether to use the lip recognition method; it can effectively avoid the situation that the accuracy of the recognition result is reduced by using the lip recognition result when speech recognition is possible, Effectively improve the accuracy of voice command recognition.

In addition, the user 001 can be asked again after the lip language recognition result is obtained, which can effectively ensure the accuracy of the recognition result.

Secondly, in the speech recognition method provided by the embodiment of the present application, the visual recognition function and the noise detection function can be turned on while confirming to the user 001. On the one hand, the voice assistant can keep the recognition result of the body movements of the user 001, and on the other hand, according to the surrounding environment As the noise changes, adjust the way of recognizing user 001 voice commands in time to increase the accuracy of voice command recognition.

The embodiment of the present application also provides a voice interaction device, including:

The detection module is configured to control the electronic device to enter the voice recognition mode after detecting that the user 001 wakes up the voice assistant.

The recognition control module is configured to control the electronic device to recognize the voice command of the user 001 by voice recognition to obtain a voice recognition result if it detects that the current voice interaction environment of the electronic device satisfies the voice recognition condition.

If it is detected that the current voice interaction environment of the electronic device does not meet the voice recognition condition, then it is detected whether the current interaction state of the user 001 meets the lip recognition condition; when it is determined that the current interaction state of the user 001 meets the lip recognition condition, control The electronic device uses a lip recognition method to recognize the mouth change characteristics of the user 001 acquired by the electronic device through the image acquisition device to obtain a lip recognition result.

The execution module is configured to control the electronic device to execute the function corresponding to the recognition result according to the recognition result acquired by the electronic device. For example, if the electronic device obtains the result of lip recognition, the electronic device is controlled to execute the function corresponding to the lip recognition result; if the electronic device obtains the result of speech recognition, the electronic device is controlled to execute the function corresponding to the speech recognition result.

The various embodiments disclosed in this application may be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of the present application may be implemented as a computer program or program code executed on a programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements) , at least one input device, and at least one output device.

Program code can be applied to input instructions to perform the functions described herein and to generate output information. The output information may be applied to one or more output devices in known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), microcontroller, application specific integrated circuit (ASIC), or microprocessor.

The program code can be implemented in a high-level procedural language or an object-oriented programming language to communicate with the processing system. Program code can also be implemented in assembly or machine language, if desired. In fact, the mechanisms described in this application are not limited in scope to any particular programming language. In either case, the language may be a compiled or interpreted language.

In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which can be executed by one or more processors read and execute. For example, instructions may be distributed over a network or via other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magnetic Optical discs, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or A tangible, machine-readable memory used to transmit information (eg, carrier waves, infrared signals, digital signals, etc.) by electrical, optical, acoustic, or other forms of propagating signals using the Internet. Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

The embodiment of the present application also provides a computer program or a computer program product including the computer program. When the computer program is executed on a computer, it will enable the computer to implement the above voice command execution method. In an implementable form, the computer program product may include instructions, and the instructions are used to implement the above voice interaction method.

In the drawings, some structural or methodological features may be shown in a particular arrangement and/or order. However, it should be understood that such specific arrangements and/or orderings may not be required. Rather, in some embodiments, these features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments these features may not be included or may be combined with other features.

It should be noted that each unit/module mentioned in each device embodiment of this application is a logical unit/module. Physically, a logical unit/module can be a physical unit/module, or a physical unit/module. A part of the module can also be realized with a combination of multiple physical units/modules, the physical implementation of these logical units/modules is not the most important, the combination of functions realized by these logical units/modules is the solution The key to the technical issues raised. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units/modules that are not closely related to solving the technical problems proposed by this application, which does not mean that the above-mentioned device embodiments do not exist other units/modules.

It should be noted that in the examples and descriptions of this patent, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply There is no such actual relationship or order between these entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Although this application has been shown and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and details may be made therein without departing from this disclosure. the scope of the application.

Claims (12)

1. A voice interaction method applied to electronic equipment, characterized in that the method includes:

   When it is detected that the current voice interaction environment of the electronic device does not meet the voice recognition condition, determine whether the user's current interaction state meets the lip language recognition condition;

   When it is determined that the user's current interaction state satisfies the lip language recognition condition, acquire the lip language recognition method obtained by identifying the user's mouth change characteristics acquired by the electronic device through the image acquisition device. Language recognition results;

   Execute the function corresponding to the lip recognition result.
2. The method according to claim 1, wherein the speech recognition conditions include:

   The noise value around the electronic equipment is lower than the set noise value;

   or;

   In the case that the noise value around the electronic device is greater than or equal to the set noise value, the duration time for the electronic device to receive sound is greater than zero and less than the set time.
3. The method according to claim 1, wherein the voice recognition condition includes: the duration of the electronic device receiving sound is greater than zero and less than a set time.
4. The method according to any one of claims 1-3, wherein the lip recognition conditions include:

   The user and the electronic device are in an interactive state within a set time.
5. The method according to claim 4, wherein the method for determining whether the user is in an interactive state with the electronic device within a set time includes:

   determining whether a user interacting with the electronic device has changed within the set time;

   Detecting whether the interaction intensity value between the user and the electronic device reaches a set intensity value;

   When it is determined that the user who interacts with the electronic device within the set time has not changed, and the interaction intensity value between the user and the electronic device reaches the set intensity value, confirm the The user is in an interactive state with the electronic device;

   Wherein, the interaction intensity value is related to the distance between the user and the electronic device and the face orientation of the user.
6. The method according to any one of claims 1-5, characterized in that, before performing the functional step corresponding to the lip recognition result, further comprising:

   Confirm whether the lip recognition result is correct;

   If it is confirmed that the lip recognition result is correct, execute the function corresponding to the lip recognition result.
7. The method according to claim 6, wherein the method for confirming whether the lip recognition result is correct comprises:

   Ask the user whether it is necessary to perform the function corresponding to the lip recognition result;

   When the user confirms that the function corresponding to the lip recognition result needs to be executed, it is confirmed that the lip recognition result is correct.
8. The method according to claim 6 or 7, characterized in that, while the step of confirming whether the lip language result is correct, it also includes:

   Obtain the body movement characteristics of the user and the noise value around the electronic device.
9. The method according to any one of claims 1-8, wherein the electronic device is a robot.
10. An electronic device, characterized in that it comprises:

    a memory for storing instructions to be executed by the one or more processors of the electronic device, and

    The processor is one of the one or more processors of the electronic device, configured to execute the voice interaction method according to any one of claims 1-9.
11. A computer-readable storage medium, characterized in that instructions are stored on the computer-readable storage medium, and when the instructions are executed, the computer executes the voice interaction method according to any one of claims 1 to 9.
12. A computer program product, characterized in that the computer program product includes instructions, and the instructions cause a computer to execute the voice interaction method according to any one of claims 1 to 9 when executed.

Images