FR3075427A1 - VOICE ASSISTANT - Google Patents

Abstract

 An assistance device (1) comprising:  - at least one processor (3) operatively coupled with a memory (5),  - at least a first input (10) connected to the processor (3) and capable of receiving video data from at least one video sensor (11), and  - at least a second input (20) connected to the processor (3) and able to receive audio data from at least one microphone (21).  The processor (3) is arranged for:  - analyze the video data from the first input (10),  - identify at least one reference human gesture in the video data, and  - trigger an analysis of audio data only if said at least one reference human gesture in the video data is detected.

Description

Voice assistant

The invention relates to the field of service provision, in particular by voice command.

The development of so-called "connected" objects tends to facilitate machine-machine interactions and the compatibility of devices with each other. For example, a mobile phone can be used as an interface to control a wireless speaker or TV from another manufacturer / designer.

In addition, household appliances, particularly in the multimedia and high fidelity (“Hi-Fi”) fields, have human-machine interfaces whose nature is changing. Voice-activated interfaces tend to replace touch screens, which themselves replaced physical button remote controls. Such voice-activated interfaces are, in particular, the basis of the rise of "voice assistants" such as systems known as "Google Home" (Google), "Siri" (Apple) or "Alexa" (Amazon).

To avoid nuisance triggers, voice assistants are generally intended to activate only when a keyword or a key phrase is spoken by the user. It is also theoretically possible to limit activation by recognizing only the voices of presumed legitimate users. However, such precautions are imperfect, in particular when the perceived sound quality does not allow a good analysis of sounds, for example in a noisy environment. The keyword or key phrase may not be picked up by the microphone or may not be recognized among all the sounds picked up. In such cases, triggering is impossible or erratic.

The invention improves the situation.

An assistance device is proposed comprising:

- at least one processor operatively coupled with a memory,

- at least a first input connected to the processor and capable of receiving video data from at least one video sensor, and

- at least one second input connected to the processor and capable of receiving audio data from at least one microphone, the processor being arranged for:

- analyze the video data from the first entry,

- identify at least one reference human gesture in the video data, and

- trigger an analysis of audio data only if said at least one reference human gesture in the video data is detected.

According to another aspect, an assistance system is proposed comprising such a device and at least one of the following organs:

- a video sensor connected or connectable to the first input;

- a microphone connected or connectable to the second input;

- a speaker connected or connectable to an output of the device.

According to another aspect, an assistance method is proposed, implemented by computer means, comprising:

- analyze video data from a first entry,

- identify at least one reference human gesture in the video data and,

- trigger an analysis of audio data only if said at least one reference human gesture in the video data is detected.

According to another aspect of the invention, there is provided a computer program comprising instructions for implementing the method as defined herein when this program is executed by a processor. According to another aspect of the invention, there is provided a non-transient recording medium, readable by a computer, on which such a program is recorded.

Such objects allow a user to trigger the implementation of a voice command process by making a gesture, for example with the hand. Thus, untimely triggers and absence of triggers, usually resulting from a malfunction of the speech recognition process, are avoided. In particular, the triggering of the voice command process is insensitive to ambient noise and involuntary voice commands. Gesture-based interfaces are less common than voice-activated interfaces, especially since it is considered less natural or less instinctive to address a machine by gestures than by voice. Consequently, the use of gestural commands is reserved for specific contexts rather than for so-called "general public" and "domestic" uses. Such objects are particularly advantageous when combined with voice assistants. Gesture recognition for triggering voice recognition can be combined with triggering by voice recognition (pronunciation of keyword (s)). In this case, the user can either make a gesture or say a keyword (s) to activate the voice assistant. Alternatively, triggering by gesture recognition replaces triggering by voice recognition. In this case, the efficiency is further improved. This also makes it possible to neutralize the microphones outside the assistant activation periods, either by switching them off or by disconnecting them. The risks of microphones being used for unintended purposes are reduced, for example by a third party taking undue control of such voice assistants.

The following features can optionally be implemented. They can be implemented independently of each other or in combination with each other:

- The device may further comprise an output controlled by the processor and capable of transmitting commands to a sound broadcasting system. The processor can, moreover, be arranged to transmit a command to reduce the sound volume or to interrupt the sound broadcast in the event of said at least one reference human gesture being detected in the video data. This reduces ambient noise and therefore facilitates subsequent operations of audio analysis, in particular speech recognition, and therefore improves the relevance and operation of services based on audio analysis.

- Analysis of audio data may include recognition of voice commands. This makes it possible to provide interactive services to the user, in particular of the voice assistance type.

- The device can further comprise an output controlled by the processor and able to transmit commands to a third-party device. The processor can, moreover, be arranged to transmit a command on said output, the command being selected according to the results of the recognition of voice commands. Such a device allows voice control of third party devices in an improved manner.

- The processor can, moreover, be arranged to trigger the emission of a visual and / or audible indicator perceptible by a user in the event of detection of said at least one reference human gesture in the video data. This allows the user to speak words / phrases to certain devices only when they know that the audio analysis is effective, thus avoiding unnecessary repetition of certain commands.

- The triggering of the emission of an indicator can include:

- the lighting of a device indicator,

- the emission of a predetermined sound on an output of the device, and / or

- the emission of a predetermined word or a predetermined series of words on an output of the device.

This allows you to adapt to many situations, especially when the environment is noisy or when a light is not visible to a user.

The aforementioned optional characteristics can be transposed, independently of each other or in combination with each other, to non-transient computer-readable devices, systems, methods, computer programs and / or recording media.

Other characteristics, details and advantages will appear on reading the detailed description below, and on analysis of the appended drawings, in which:

- Figure 1 shows a non-limiting example of a device proposed according to one or more embodiments, and

- Figure 2 shows a non-limiting example of interactions implemented according to one or more embodiments.

In the following detailed description of embodiments, many specific details are presented to provide a more complete understanding. Nevertheless, the person skilled in the art can realize that embodiments can be put into practice without these specific details. In other cases, well-known features are not described in detail to avoid unnecessarily complicating the description.

In the following, it is a question of detecting at least one human gesture. The term "gesture" is used here in its broadest sense, that is to say as well referring to movements (dynamic) and postures (static) of at least one member of the human body, typically a hand.

FIG. 1 represents a device 1 for assistance available to a user 100. The device 1 comprises:

at least one processor 3 operatively coupled with a memory 5,

at least one first input 10 connected to processor 3, and

- at least one second input 20 connected to processor 3.

The first input 10 is capable of receiving video data from at least one video sensor 11, for example a camera or a webcam. The first input 10 forms an interface between the video sensor and the device 1 and takes, for example, the form of an HDMI type connector (for “High-Definition Multimedia Interface”), As a variant, other types of video input can be provided, in addition to or in replacement of the HDMI connector. For example, the device 1 may comprise a plurality of first inputs 10, in the form of several connectors of the same type or of different types. Thus, the processor 3 can receive as input several video streams. This allows, for example, to capture images in different rooms of a building or from different angles. The device 1 can, moreover, be made compatible with a variety of video sensors 11.

The second input 20 is capable of receiving audio data from at least one microphone 21. The second input 20 forms an interface between the microphone and the device 1 and takes, for example, the form of a connector of coaxial type ( for example says "jack"). Alternatively, other types of audio input may be provided, in addition to or as a replacement for the coaxial connector. In particular, the first input 10 and the second input 20 may have a common connector, capable of receiving both a video stream and an audio stream. HDMI connectors are, for example, connectors with this possibility. The HDMI connectors also have the advantage of being widespread on existing devices, notably televisions. Thus, a single HDMI connector can allow the device 1 to be connected to a television equipped with both a microphone and a camera. These third-party equipment can then be used to supply respectively a first input 10 and a second input 20 of the device 1.

For example, the device 1 can also include a plurality of second inputs 20, in the form of several connectors of the same type or of different types. Thus, the processor 3 can receive as input several audio streams, for example from several microphones distributed in a room, which makes it possible to improve the subsequent speech recognition by signal processing methods known as such. The device 1 can also be made compatible with a variety of microphones 21.

In the nonlimiting example shown in FIG. 1, the device 1 also comprises:

an output 30 connected to processor 3 and controlled by processor 3.

The output 30 is capable of transmitting commands to a sound diffusion system 50, for example a connected speaker, a high-fidelity installation (“Hi-Fi”), a television set, a computer (or “smartphone”), a tablet or even a computer. The sound diffusion system 50 comprises at least one speaker 51.

In the nonlimiting example shown in FIG. 1, the device 1 also comprises:

an output 40 connected to processor 3 and controlled by processor 3.

The output 40 is capable of transmitting commands to at least one third party device 60, for example a connected speaker, a Hi-Fi installation, a television set, a computer (or “smartphone”), a tablet or even a computer.

The outputs 30, 40 can, for example, take the form of connectors of various types preferably selected to be compatible with third-party equipment. The connector of one of the outputs 30, 40 may, for example, be common with the connector of one of the inputs. For example, the HDMI connectors allow the implementation of two-way audio transmissions (technology known under the acronym "ARC" for "Audio Return Charnel"). Thus, a second input 20 and an output 30 may have a common connector connected to equipment, such as a television, including both a microphone 21 and speakers 51.

For example, the device 1 can also comprise a single output or even more than two outputs in the form of several connectors of the same type or of different types. Thus, the processor 3 can transmit several commands as an output, for example to control several third-party devices distinctly.

So far, inputs 10, 20 and outputs 30, 40 have been presented as taking the form of one or more mechanical connectors. In other words, the device 1 can be connected to third-party equipment by cables. As a variant, at least some of the inputs / outputs can take the form of a communication module without fd. In such cases, the device 1 also comprises at least one wireless communication module, so that the device 1 can be wirelessly connected to remote third-party devices, including devices as shown in the example above. . The wireless communication modules are then connected to processor 3 and controlled by processor 3.

The communication modules can, for example, include a short-distance communication module, for example based on radio waves such as those of the Wifi type. Wireless local area networks, particularly domestic networks, are often implemented using a Wi-Fi network. Thus, the device 1 can be integrated into an existing environment, in particular the so-called "home automation" networks.

The communication modules can, for example, include a short distance communication module, for example of the Bluetooth® type. Communication means compatible with Bluetooth® type technology equip a large part of recent equipment, in particular smartphones and so-called “portable” speakers.

The communication modules can, for example, include a near field communication module (or NFC for "Near Field Communication"). In such cases, the communication being effective only at distances of a few centimeters, the device 1 must be placed in the immediate vicinity of relays or third-party equipment to which one wishes to connect.

In the nonlimiting example shown in FIG. 1, the video sensor 11, the microphone 21 and the speaker 51 of the sound broadcasting system 50 are third-party equipment (not integrated into the device 1). This equipment can be connected to the processor 3 of the device 1 while being integrated into other devices, together or separately from each other. Such third-party devices include, for example, a television, a smartphone, a tablet or even a computer. This equipment can also be connected to the processor 3 of the device 1 while being equipment independent of any other device. In the embodiments for which at least some of the aforementioned equipment is absent from the device 1, in particular the video sensor 11 and the microphone 21, the device 1 can be considered as a multimedia box, or auxiliary device, intended to come connect or pair with at least one third-party device, such as a TV. In this case, such a multimedia box is only operational once connected to such a third-party device. Such a multimedia box can be included in a TV decoder (designated by the acronym STB for "Set Top Box") or even in a game console.

As a variant, at least some of the aforementioned equipment may be integrated into the device 1. In the latter case, the device 1 also comprises:

- at least one video sensor 11 connected to a first input 10;

- at least one microphone 21 connected to a second input 20; and or

- at least one speaker 51 connected to an output 30 of the device 1.

Alternatively, the device 1 comprises a combination of integrated equipment and inputs / outputs intended to connect to third-party devices and devoid of corresponding integrated equipment.

In variants, the device 1 further comprises at least one visual indicator, for example one or more indicator lights. Such an indicator, controlled by the processor 3, can be activated so as to inform the user 100 about a state of the device 1. The state of such an indicator can vary, for example during pairing operations with equipment. third party and / or in the event of activation or deactivation of the device 1 as will be described in more detail below.

In the embodiments for which at least some of the aforementioned equipment is integrated into the device 1, in particular at least one video sensor 11 and at least one microphone 21, the device 1 can be considered as a device at least partially autonomous. In particular, the method described below and with reference to Figure 2 can be implemented by the device 1 without the need to connect or pair it with third-party devices.

The device 1 further comprises a power source, not shown, for example a power cord for connection to the mains and / or a battery.

In the examples described here, the device 1 comprises a single processor 3. As a variant, several processors can cooperate to implement the operations described herein.

The processor 3, or data processing unit (CPU), is associated with the memory 5. The memory 5 comprises for example a random access memory (RAM), a read only memory (ROM), a cache memory and / or a flash memory , or any other storage medium capable of storing software code in the form of instructions executable by a processor or data structures accessible by a processor.

The processor 3 is arranged for:

- analyze the video data from at least one first input 10,

- identify at least one reference human gesture in the video data, and

- trigger an analysis of audio data only if said at least one reference human gesture in the video data is detected.

The reference gesture or the reference gestures can be, for example, stored in the form of determination / identification criteria in the memory 5 and which the processor 3 calls upon during the analysis of the video data. Such criteria can be set by default. As a variant, such criteria can be modified by software updates and / or by learning from the user 100 himself. Thus, the user 100 can select the key gestures or reference gestures allowing the triggering of the analysis of the audio data.

In the examples described here, both the triggering of the audio data analysis and the audio analysis itself are implemented by the device 1 (via a second input 20 and the processor 3) . As a variant, the triggering is implemented by the device 1 while the audio analysis is implemented by a third-party device to which the device 1 is connected. In other words, the device 1 can operate in a so-called “autonomous” mode at meaning that the device 1 itself performs audio analysis and optionally subsequent operations. Such a device 1 can advantageously replace a voice assistant. The device 1 can also operate in a “backup” mode in the sense that the device 1 triggers the audio analysis by a third-party device, for example by transmitting an activation signal to the third-party device, such as those referenced 60 and connected at exit 40.

In other words, the processor 3 can, optionally, be arranged to carry out the analysis of the audio data in addition to the triggering.

Whether the device 1 operates in “autonomous” or “backup” mode, the triggering of the audio analysis by detection of a gesture can be combined with a triggering of the audio analysis by the voice (pronunciation of one or multiple keywords). Thus, the audio analysis and the services which result from it can remain activatable, in parallel, by the voice alone independently of the gestures (detected by a third party device) as well as by the gestures independently of the voice (detected by the device 1). . Triggering can also be conditioned by the detection of a combination of the voice and the use of a reference gesture, simultaneously or successively.

As a variant, the triggering of the audio analysis by detection of a gesture may be exclusive of a triggering of the audio analysis by the voice. In other words, the device 1 can be arranged to make the voices, including that of the user 100, inoperative before the triggering of the audio analysis by a gesture. Thus, a device 1 in autonomous mode, or a system combining a backup device 1 with a third-party device, can prohibit the triggering of audio analysis by voice.

Analysis of audio data may include recognition of voice commands. Techniques for recognizing voice commands are known as such, in particular in the context of voice assistants.

FIG. 2 represents the interactions between different elements during the implementation of a method according to one embodiment.

User 100 performs a gesture (static or dynamic). The gesture is captured by a video sensor 11 connected to a first output 10 of a device 1. The processor 3 of the device 1 receives a video stream (or video data) including the capture of the reference gesture. The processor 3 can receive a substantially continuous video stream or, for example, only when motion is detected.

The processor 3 implements an analysis operation of the video data received. The operations include attempts to identify one or more reference human gestures. If no reference gesture is detected, then the rest of the process is not triggered. Device 1 remains on standby.

If the reference gesture made by the user 100 is detected, then the rest of the method is implemented. In FIG. 2, the implementation of two optional and independent operations from one another are shown in dashed lines:

- an operation aimed at reducing ambient noise before implementing audio analysis, and

- an operation aimed at confirming to user 100 that the audio analysis has been triggered or is about to be triggered.

In the embodiments comprising a combination of these two optional operations, they can be implemented one after the other or concomitantly.

In embodiments, the processor 3 is therefore, in addition, arranged to transmit a command to reduce the sound volume or to interrupt the sound broadcasting in the event of detection of at least one reference human gesture in the video data. . The command is, for example, transmitted via the output 30 and intended for the sound diffusion system 50 including a loudspeaker 51 as shown in FIG. 2. The transmission of such a command can be carried out, in replacement or in complement, via other outputs of the device 1 such as the output 40 and intended for third party equipment 60.

The processor 3 is, in addition, arranged to trigger the emission of a visual and / or audible indicator perceptible by the user 100 in the event of detection of at least one reference human gesture in the video data. The sending of the indicator is represented by the sending of an "OK" in Figure 2. For example, the triggering of the emission of an indicator can include:

- the lighting of a device 1 indicator;

- the emission of a predetermined sound on an output of the device 1, for example the outputs 30 and / or 40 of the embodiment of FIG. 1; and or

- the emission of a predetermined word or of a predetermined series of words on an output of the device 1, for example the outputs 30 and / or 40 of the embodiment of FIG. 1.

In the embodiment shown in FIG. 2, once the analysis of the audio data has started, the processor 3 is arranged to receive audio data to be analyzed, in particular via a second output 20 and the microphone 21. The audio data comprises, for example example, a voice command pronounced by the user 100. In nonlimiting examples, the processor 3 can, in addition, be arranged to carry out an audio analysis including recognition of voice commands, then to transmit a command selected according to results of the recognition of voice commands, in particular via outputs 30 and / or 40, and intended respectively for the sound broadcasting system 50 and / or a third-party device 60.

The variety of voice commands that can be translated by the device 1 into commands that can be interpreted by computer by third-party devices includes, for example, usual commands of a Hi-fi system, of the “increase in volume”, “decrease in volume” type, "Change of track" or "change of source".

Hitherto, reference has been made to embodiments and variants of a device 1. The person skilled in the art will understand without effort that the various combinations of operations described as implemented by the processor 3 can generally be heard. as forming a method of assistance (of the user 100) implemented by computer means. Such a process can also take the form of a computer program or of a medium on which such a program is recorded.

The device 1 has been presented in a functional state. The person skilled in the art will further understand that, in practice, the device 1 can take a temporarily inactive form, such as a system including various parts intended to cooperate with each other. Such a system can, for example, include a device 1 and at least one of a video sensor connectable to the first input 10, a microphone connectable to the second input 20 and a speaker 51 connectable to an output 30 of the device 1.

Optionally, the device 1 can be provided with a processing device including an operating system and programs, components, modules and / or applications in the form of software executed by the processor 3, which can be stored in a nonvolatile memory such as memory 5.

Depending on the embodiments chosen, certain acts, actions, events or functions of each of the methods and processes described in this document may be performed or occur in a different order from that in which they have been described, or may be added , merged or not be performed or not occur, as the case may be. In addition, in certain embodiments, certain acts, actions or events are performed or occur concurrently and not successively or vice versa.

Although described through a certain number of detailed exemplary embodiments, the proposed methods and the systems and devices for implementing the methods include different variants, modifications and improvements which will be apparent to the person skilled in the art, it being understood that these different variants, modifications and improvements are part of the scope of the invention, as defined by the protection sought. In addition, different aspects and characteristics described above can be implemented together, or separately, or substituted for each other, and all of the different combinations and sub combinations of aspects and characteristics are part of the scope of the 'invention. In addition, some systems and equipment described above may not incorporate all of the modules and functions described for the preferred embodiments.

The invention is not limited to the examples of devices, systems, methods, recording media and programs described above, only by way of example, but it encompasses all the variants that the person skilled in the art can envisage in the protection sought.

Claims (10)

claims 1. Assistance device (1) comprising: - at least one processor (3) operatively coupled with a memory (5), - at least a first input (10) connected to the processor (3) and capable of receiving video data from at least one video sensor (11), and - at least a second input (20) connected to the processor (3) and capable of receiving audio data coming from at least one microphone (21), the processor (3) being arranged for: - analyze the video data from the first input (10), - identify at least one reference human gesture in the video data, and - trigger an analysis of audio data only if said at least one reference human gesture in the video data is detected. 2. Device according to claim 1 further comprising an output (30) controlled by the processor (3) and capable of transmitting commands to a sound distribution system (50), the processor (3) being further arranged to transmit a command to reduce the sound volume or to interrupt the sound diffusion in the event of said at least one reference human gesture being detected in the video data. 3. Device according to one of the preceding claims, in which the analysis of audio data includes recognition of voice commands. 4. Device according to claim 3, further comprising an output (30, 40) controlled by the processor (3) and capable of transmitting commands to a third party device (50, 60), the processor (3) being further arranged for transmitting a command on said output (30, 40), the command being selected according to the results of the recognition of voice commands. 5. Device according to one of the preceding claims in which the processor (3) is further arranged to trigger the emission of a visual and / or audible indicator perceptible by a user in the event of detection of said at least one human gesture of reference in video data. 6. Device according to claim 5, in which the triggering of the emission of an indicator includes: - the lighting of a device indicator, - the emission of a predetermined sound on an output (30, 40) of the device (1), and / or - the emission of a predetermined word or of a predetermined series of words on an output (30, 40) of the device (1). 7. Assistance system comprising a device (1) according to one of the preceding claims and at least one of the following organs: - a video sensor (11) connected or connectable to the first input (10); - a microphone (21) connected or connectable to the second input (20); - a loudspeaker (51) connected or connectable to an output (30) of the device (1). 8. Assistance process, implemented by IT means, comprising: - analyze video data from a first input (10), - identify at least one reference human gesture in the video data and, - trigger an analysis of audio data only if said at least one reference human gesture in the video data is detected. 9. Non-transient recording medium readable by a computer on which a program is recorded comprising instructions for implementing the method according to the preceding claim. 10. Computer program comprising instructions for implementing the method according to claim 8 when this program is executed by a processor (3).