FR3102603A1 - Method and device for evaluating a voice recognition system - Google Patents

Abstract

The invention relates to a method and a device (10) for evaluating a voice recognition system (11). To this end, several voice commands are given to test the reliability or the performance of the voice recognition system (11). Each response of the speech recognition system (11) to a question is analyzed to determine a score representing the reliability or performance of the speech recognition system (11). The score is advantageously a function of the number of correct answers to the questions asked but also of the number of erroneous questions to the questions asked, an incorrect answer corresponding to an off-topic answer, unrelated to the question asked. Figure for the abstract: Figure 1

Description

Method and device for evaluating a voice recognition system

The invention relates to a method and a device for evaluating a voice recognition system, in particular a voice recognition system of a vehicle, for example an automobile.

Technology background

Contemporary vehicles, or at least some of them, carry equipment or systems with which the driver and/or passengers can interact. Interaction with on-board systems provided for this purpose requires one or more human-machine interfaces, known as HMIs. These HMIs can take several forms, in particular through voice interfaces. A voice interface requires that voice commands spoken by a user are received, interpreted and understood by the HMI. To this end, some vehicles today carry an automatic speech recognition system (also called a voice recognition system) which makes it possible to analyze the human voice acquired by a microphone in order to transcribe it into a form that can be used by a machine. In a vehicle, such a voice recognition system makes it possible, for example, to control the multimedia system of the vehicle, the navigation system and sometimes even to control the vehicle, in particular when the latter is in autonomous mode.

It is important for the user-friendliness of the system and the safety of the vehicle and its passengers that the voice recognition system is reliable and efficient, that is to say that it understands and correctly interprets the voice commands received.

There are certain methods to test the performance, efficiency and/or reliability of speech recognition systems, such as the word error rate for example, called WER (from the English "Word Error Rate"). According to this method, the lower the rate, the better the word recognition. However, this method is based on word recognition but does not guarantee that the response to a command is correct.

An object of the present invention is to improve the evaluation of a voice recognition system.

According to a first aspect, the invention relates to a method for evaluating a voice recognition system of a vehicle, the method being implemented by an evaluation device comprising at least one processor, the method comprising the following steps :

- rendering of a plurality of voice commands;

- analyzing each response from the voice recognition system to each voice command of the plurality of voice commands;

- determination of a score representative of a reliability of the voice recognition system, the score being a function of a number of correct responses and of a number of incorrect responses to the plurality of voice commands.

According to a variant, the plurality of voice commands comprises at least one voice command relating to a domain not learned by the voice recognition system.

According to another variant, a positive mark is assigned to a correct answer and a negative mark is assigned to an incorrect answer.

According to an additional variant, a mark equal to 0 is assigned to a representative response of non-understanding of a question by the voice recognition system.

According to an additional variant, the plurality of voice commands are rendered by voice synthesis.

According to yet another variant, the method further comprises a step of unlocking the vehicle from the connected object.

According to an additional variant, the analysis implements machine learning.

According to yet another variant, each answer is compared with a list of answers in a correspondence table associating questions and expected answers

According to a second aspect, the invention relates to a device for evaluating a voice recognition system of a vehicle, the device comprising a memory associated with a processor configured for the implementation of the steps of the method according to the first aspect of the invention.

According to a variant, the device also comprises voice synthesis means, at least one microphone and at least one loudspeaker.

According to a third aspect, the invention relates to a computer program which comprises instructions adapted for the execution of the steps of the method according to the first aspect of the invention, this in particular when the computer program is executed by at least one processor.

Such a computer program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

According to a fourth aspect, the invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for the execution of the steps of the method according to the first aspect of the invention.

On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a ROM memory of the microelectronic circuit type, or even a magnetic recording means or a hard disk.

On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other ways. The computer program according to the invention can in particular be downloaded from an Internet-type network.

Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the method in question.

Brief description of figures

Other characteristics and advantages of the invention will emerge from the description of the non-limiting embodiments of the invention below, with reference to the appended figures 1 to 3, in which:

schematically illustrates a voice recognition system in communication with a device for evaluating this voice recognition system, according to a particular embodiment of the present invention;

schematically illustrates the device for evaluating the voice recognition system of FIG. 1, according to a particular embodiment of the present invention;

illustrates a flowchart of the different steps of a method for evaluating a voice recognition system of a vehicle, according to a particular embodiment of the present invention.

A method and a device for evaluating a voice recognition system, for example on board a vehicle, will now be described in the following with reference in conjunction with FIGS. 1 to 3. The same elements are identified with the same signs reference throughout the following description.

According to a particular and non-limiting embodiment of the invention, a method for evaluating a voice recognition system of a vehicle comprises the rendering, for example by voice synthesis, of several voice commands to test the reliability or the performance of the voice recognition system. Each voice recognition system response to a question is analyzed to determine a score representing the reliability or performance of the voice recognition system. The score is advantageously a function of the number of correct answers to the questions asked but also of the number of erroneous questions to the questions asked, an erroneous answer corresponding to an off-topic answer, unrelated to the question asked.

Taking erroneous answers into account to establish a reliability score makes it possible to judge the real performance or relevance of the voice recognition system, and not its simple ability to recognize the words spoken in voice commands. The evaluation of the system is thus improved in that it is more complete in judging the relevance of the responses given to voice commands.

schematically illustrates an evaluation device in communication with a voice recognition system 11, according to a particular and non-limiting embodiment of the present invention.

The evaluation device 10 and the voice recognition system are said to be in communication in that it exchanges questions and answers, the evaluation device 10 for example posing a set of test voice commands, for example by voice synthesis , and the voice recognition system providing answers in return, also by voice synthesis for example.

The evaluation device 10 corresponds for example to a computer equipped with one or more microphones and one or more loudspeakers for rendering voice commands. A more detailed description of such a device is presented in the text below next to Figure 2.

The voice recognition system 11 corresponds for example to a computer equipped with one or more microphones to pick up the electroacoustic signals corresponding to the voice commands given by the device 10. The voice recognition system also comprises one or more loudspeakers to make responses to voice commands received.

The voice recognition system 11 for example implements acoustic processing to extract from the acoustic signals received one or more acoustic images, the signal being for example digitized and parameterized by frequency analysis, using for example the Fourier transform. Automatic learning associating elementary segments of speech and lexical elements is also implemented, for example based on statistical modeling and/or on artificial neural networks. The models resulting from the preceding modeling are then for example decoded to reconstitute the most probable voice command.

System 11 is based, for example, on language, pronunciation and acoustico-phonetic models.

The voice recognition system 11 is for example configured to respond to voice commands relating to one or more determined domains, for example voice commands relating to navigation, voice commands relating to the multimedia system, voice commands relating to meteorology, etc The domain or domains managed by the voice recognition system 11 have thus been the subject of learning. One or more domains are not managed by the voice recognition system 11, such as the domain of meteorology in the following example, this example being of course purely illustrative and not limiting.

It is through these exchanges of voice commands and responses that the evaluation device 10 tests and evaluates the level of performance or reliability of the voice recognition system 11.

In a first operation, several voice commands are given by the evaluation device, for example by voice synthesis, these commands being given through one or more loudspeakers fitted to the evaluation device or connected to the latter, for example via a wired or wireless connection (for example a Bluetooth® connection). Each voice command corresponds for example to a determined question posed to the voice recognition system 11.

The set of rendered voice commands advantageously comprises one or more questions relating to one or more domains controlled and learned by the voice recognition system, for example questions relating to navigation. This set of rendered voice commands advantageously comprises one or more questions relating to one or more areas not controlled and not learned by the voice recognition system, for example one or more questions relating to the meteorology and/or to the multimedia system (for example the radio).

The table below shows some examples of voice commands asked by the evaluation device 10 and the associated domain. This table shows 2 commands relating to an area managed and learned by system 11 (i.e. the navigation area) and 2 voice commands relating to 2 areas not managed and not learned by system 11 (i.e. music/multimedia d on the one hand and meteorology on the other).

Voice command Domain Navigate to Paris Navigation Navigate to Route de Gizy à Vélizy Navigation Play Beyonce Music / Multimedia What will the weather be like on arrival in Paris Meteorology

Of course, the number of voice commands is not limited to 4 but extends to any number, for example 10, 20, 30, 50 or 100 voice commands.

The rendered voice commands are for example extracted from a list of voice commands recorded in the detection device 10 and output at random, depending on the number of voice commands desired. This number and/or the question fields correspond for example to parameters determined and entered by a user, for example through an HMI (for example a graphic HMI displayed on a screen, for example touchscreen). According to a variant, this number and/or the domains are chosen randomly by the device or in a deterministic manner according to one or more determined rules.

In a second operation, the device 10 analyzes the responses provided by the system 11 to the voice commands given to the previous operation. These responses are for example provided in the form of voice responses by the system 11 and captured by one or more microphones of the device 10. This or these microphones are for example integrated into the device 10 or connected to the device 10, for example via a wired connection or wireless (for example a Bluetooth® connection).

The table below shows the System 11 responses to the voice commands in Table 1.

Voice command Answer Navigate to Paris Navigation to Paris launched Navigate to Route de Gizy à Vélizy Navigation to Vélizy Route de Gizy launched Play Beyonce I did not understand the question What will the weather be like on arrival in Paris 10 minutes left to reach destination

An analysis of each of the responses to the voice commands is advantageously implemented by the device 10 to determine. For this purpose, each response is for example classified into one of the following categories:

- correct answer, i.e. the answer is related to the domain of the voice command asked and corresponds to the voice command asked;

- erroneous answer, that is to say that the answer is unrelated to the domain of the voice command asked (for example the answer corresponds to a domain managed and learned by the system 11 but does not correspond to the domain of the command voice) and does not match the spoken command;

- command response not understood, i.e. the response states that the system did not understand the voice command.

Returning to the example of Table 2, the table below associates each response with a category.

Answer Category Navigation to Paris launched Correct answer Navigation to Vélizy Route de Gizy launched Correct answer I did not understand the question Response to command not included 10 minutes left to reach destination Wrong answer

The analysis uses, for example, a machine learning method to classify the responses. According to a variant embodiment, each answer is compared to a list of answers in a correspondence table, called LUT (from the English "Look-Up Table") associating questions and expected answers, thus making it possible to classify the answers. This correspondence table is for example enriched via a machine learning method.

In a third operation, the device 10 determines a score representing the reliability and/or the performance of the voice recognition system 11 from the responses to the voice commands, in particular from the number of correct responses and the number of incorrect responses. Thus, a score is for example assigned to each response to determine a final score in the system 11. A positive score (for example +1) is for example assigned to each correct response and a negative score (for example -1) is for example attributed to each incorrect answer. A neutral score (for example 0) is for example assigned to each command response not understood. Such a notation makes it possible to take into account erroneous notes which are more annoying for a user (because they are false and could mislead the user) than a response to a command not understood and stated as such (a response announcing that the command n was not understood is generally better accepted than a false or erroneous answer).

The table below assigns a score to each response.

Answer Rating or rating Navigation to Paris launched +1 Navigation to Vélizy Route de Gizy launched +1 I did not understand the question 0 10 minutes left to reach destination -1

The final mark assigned to System 11 corresponds to the average of the marks assigned to each answer, i.e. the sum of each mark divided by the number of questions (or equivalently the number of answers).

According to the example in Figure 1, the score assigned to system 11 is 0.25, or 25% correct answer.

Of course, another notation could be adopted, for example +2 for each correct answer, +1 for an answer indicating that the command was not understood and -2 or -3 for an incorrect answer.

As opposed to an evaluation according to which only the domain or domains managed or learned by the system 11 are tested (where a score or rating of 100% would have been obtained), the evaluation of the present invention is more representative of an actual use. of a vehicle navigation system where the user can ask any question, even questions relating to areas not learned by the system 11. According to the evaluation method of the invention, the erroneous or false answers are sanctioned by assigning a negative rating, which makes the evaluation more representative of real use and customer satisfaction with the use of the voice recognition system 11.

schematically illustrates a device 10 configured to evaluate a voice recognition system, for example for a vehicle, according to a particular and non-limiting embodiment of the present invention.

The device 10 is for example configured for the implementation of the operations described with regard to FIG. 1 and/or the steps of the method described with regard to FIG. 3. Examples of such a device 10 comprise, without being limited thereto , computer, smart phone, tablet, calculator. The elements of device 10, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and/or in discrete components. The device 10 can be made in the form of electronic circuits or software (or computer) modules or else a combination of electronic circuits and software modules.

The device 10 comprises one (or more) processor(s) 20 configured to execute instructions for carrying out the steps of the method and/or for executing the instructions of the software or software embedded in the device 10. The processor 20 can include integrated memory, an input/output interface, and various circuits known to those skilled in the art. The device 10 further comprises at least one memory 21 corresponding for example to a volatile and/or non-volatile memory and/or comprises a memory storage device which can comprise volatile and/or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

The computer code of the on-board software or software comprising the instructions to be loaded and executed by the processor is for example stored on the first memory 21.

According to a particular and non-limiting embodiment, the device 10 comprises a block 22 of interface elements for communicating with external devices, for example a remote server or the "cloud". Block 22 interface elements include one or more of the following interfaces:

- RF radiofrequency interface, for example of the Bluetooth® or Wi-Fi® type, LTE (from English "Long-Term Evolution" or in French "Evolution à long terme"), LTE-Advanced (or in French LTE-avanced );

- USB interface (from English "Universal Serial Bus" or "Bus Universel en Série" in French);

- HDMI interface (from the English "High Definition Multimedia Interface", or "Interface Multimedia Haute Definition" in French).

According to another particular embodiment, the device 10 comprises a communication interface 23 which makes it possible to establish communication with other devices via a communication channel 230. The communication interface 23 corresponds for example to a transmitter configured to transmit and receive information and/or data via the communication channel 230.

According to an additional particular embodiment, the device 10 can supply and/or receive output signals to one or more external devices, such as one or more microphones 240, one or more loudspeakers 250 and/or a screen 260 respectively via input/output interfaces 24, 25 and 26. According to a variant, one or the other of the external devices is integrated into the device 10. The display screen 240 corresponds for example to the screen, touch or not.

illustrates a flowchart of the different steps of a method for evaluating a voice recognition system, according to a particular and non-limiting embodiment of the present invention. The method is advantageously implemented in the device 10.

In a first step 31, several voice commands are rendered.

In a second step 32, each response from the voice recognition system to each voice command is analyzed.

In a third step 33, a score representative of a reliability of the voice recognition system is determined. This score is advantageously a function of a number of correct responses and a number of incorrect responses to the voice commands given in the first step.

Steps 31 to 33 are for example reiterated during an update of the voice recognition system, for example when a new domain is learned by the voice recognition system.

Claims (10)

Method for evaluating a speech recognition system (11), said method being implemented by an evaluation device (10) comprising at least one processor, said method comprising the following steps:
- rendering (31) of a plurality of voice commands;
- analysis (32) of each response of said voice recognition system (11) to each voice command of said plurality of voice commands;
- determination (33) of a score representative of a reliability of said voice recognition system (11), said score being a function of a number of correct responses and of a number of incorrect responses to said plurality of voice commands. A method according to claim 1, wherein said plurality of voice commands includes at least one voice command relating to a domain not learned by said voice recognition system (11). A method according to claim 1 or 2, wherein a positive mark is assigned to a correct answer and a negative mark is assigned to an incorrect answer. Method according to one of Claims 1 to 3, for which a mark equal to 0 is assigned to an answer representative of non-understanding of a question by said voice recognition system (11). Method according to one of Claims 1 to 4, for which the said plurality of voice commands is rendered by voice synthesis. Method according to one of claims 1 to 5, for which said analysis (32) implements machine learning. Method according to any one of Claims 1 to 6, for which each answer is compared with a list of answers in a correspondence table associating questions and expected answers. Device (10) for evaluating a voice recognition system of a vehicle, said device (10) comprising a memory (21) associated with at least one processor (20) configured for the implementation of the steps of the method according to any of claims 1 to 7. Device (10) according to claim 8, further comprising voice synthesis means, at least one microphone (240) and at least one loudspeaker (250). Computer program product comprising instructions adapted for the execution of the steps of the method according to one of Claims 1 to 7, when the computer program is executed by at least one processor.