EP4377953A1

EP4377953A1 - A method for providing a functionality of a digital assistant in a vehicle, a system, a vehicle, an electronic control device and a computer program product

Info

Publication number: EP4377953A1
Application number: EP21773372.4A
Authority: EP
Inventors: Markus-Julian CHUR; Sebastian Gehrling; Jan Pingel; Prathvirani TOLETY; Jan WELZBACHER; Adnan YURBAS
Original assignee: Volkswagen AG
Current assignee: Cariad SE
Priority date: 2021-09-07
Filing date: 2021-09-07
Publication date: 2024-06-05
Also published as: WO2023036401A1

Abstract

The invention relates to a method for providing a functionality of a digital assistant (DA) in a vehicle (10), the method comprising the following: providing a recording device (11 ) installed in the vehicle (10) for recording a voice request (VR) of a user, providing a special proxy server (DAP) for the digital assistant (DA) in an edge gateway (GW) of an electronic control device (ECU) of the vehicle (10), transmitting the voice request (VR) of the user from the proxy server (DAP) to a digital assistant server (DAS) in a backend device (20) for performing a voice analysis and providing an answer (AW) as a result of the analysis, receiving the answer (AW) to the voice request (VR) from a digital assistant server (DAS) at the proxy server (DAP), transmitting the answer (AW) from the proxy server (DAP) to a play back device (12) installed in the vehicle (10).

Description

A method for providing a functionality of a digital assistant in a vehicle, a system, a vehicle, an electronic control device and a computer program product

The invention is related to a method for providing a functionality of a digital assistant in a vehicle according to the independent method claim. Further, the invention is related to a system for a vehicle for providing a functionality of a digital assistant in the vehicle according to the independent system claim. Furthermore, the invention is related to a vehicle comprising a corresponding system according to the independent device claim. Moreover, the invention is related to an electronic control device according to the further independent device claim as well as to a computer program product according to the independent product claim.

Intelligent personal assistants are basically known. An Intelligent personal assistant is as a voice assistant that enables information to be retrieved, dialogs to be conducted and assistance services to be provided by means of communication in natural, human language by performing a speech analysis for speech recognition, interpreting this semantically, processing it logically and, as a result, formulating an answer through speech synthesis. Such applications are used especially on mobile devices like smartphones. However, using a smartphone while driving a vehicle can have disadvantages. Mobile devices do not have access to the applications running in a vehicle like a built-in application has. Voice control of in-vehicle facilities like the climatization is not possible. A mobile device often is not at hand (e.g. it is in the pocket) when the user wants to access their digital assistant while driving. A mobile device does not have an optimized microphone setup in the vehicle yielding to an error prone activation word recognition by the proxy and speech command recognition by the digital assistant. The response of the digital assistant is played back on the mobile device’s loudspeaker which is low in volume and interferes with the vehicle’s audio system. A mobile device has inferior radio signal reception and transmission characteristics compared to the vehicle’s radio signal transmission/reception characteristics. The user is distracted by handling the mobile device when using the digital assistant while driving.

The aim of the present invention is to provide a method for providing an extended functionality of a digital assistant in a vehicle, with improved acoustic features, convenient use and improved safety during the use, preferably in an easy, simple and intuitive way. Also, the aim of the invention is to provide an improved system for a vehicle for providing a functionality of a digital assistant in the vehicle. Besides, the aim of the invention is to provide a vehicle comprising a corresponding system. Further, the aim of the invention is to provide an electronic control device and a computer program product for a corresponding method.

According to the first aspect, embodiments of the invention provide a method for providing a functionality of a digital assistant in a vehicle with the features of the independent method claim. According to the second aspect, embodiments of the invention provide a system for a vehicle for providing a functionality of a digital assistant in the vehicle with the features of the independent system claim. According to the third aspect, embodiments of the invention provide a vehicle comprising a corresponding system with the features of the independent device claim. According to the fourth aspect, embodiments of the invention provide a corresponding electronic control device for a corresponding method with the features of the second independent device claim. According to the fifth aspect, embodiments of the invention provide a computer program product for a corresponding method with the features of the independent product claim. Details and features disclosed on individual aspects of the invention also apply to the other aspects of the invention and vice versa.

According to the first aspect, embodiments of the invention provide a method for providing a functionality of a digital assistant in a vehicle, the method comprising: providing a recording device installed in the vehicle for recording a voice request (or command) of a user, providing a (special) proxy server for the digital assistant in an edge gateway of an electronic control device of the vehicle, transmitting the voice request of the user from the proxy server to a digital assistant server (in a backend device) for performing a voice analysis and providing an answer (or reply) as a result of the analysis, receiving the answer to the voice request from digital assistant server at the proxy server, transmitting the answer from the proxy server to a play back device installed in the vehicle.

Embodiments of the method may be carried out in the given order or in a modified order. Individual actions of the method may be carried out simultaneously and/or repeatedly to allow a flowing process.

A backend device within the meaning of the invention is a computer or a group of computers providing a high computational power, a large memory and easy access to updates as well as to a digital assistant software that may be used in the vehicle. The backend device within the meaning of the invention is the representative of the cloud. The idea is to provide a proxy server (or simply a proxy) that interfaces with a digital assistant in the cloud. The invention proposes to allocate the proxy to the edge gateway of an electronic control device of the vehicle and within the edge gateway of an electronic control device especially to the loT Edge runtime for the sake of easy update possibilities which is paramount if new skills are to be added to the proxy.

Currently the loT Edge runtime is designed to support loT data traffic. The loT traffic is bulky, there are packets of huge size that can block transmission of voice commands for seconds. Therefore, it may be preferably provided that the traffic from the loT Edge runtime to the cloud shall permit to suspend the transmission of large loT information packets in favor of higher priority audio command data.

Advantageously, the audio stream coming from the vehicle may be aggregated to larger packets by the proxy in the loT Edge runtime.

Further, it may be advantageous if the speech command reply may be directly sent to the element which plays back the speech command reply.

Furthermore, it may be advantageous if the action command (so called “raw action command”) may be sent to the proxy and within the proxy further detailed according to the available skills and vehicle capabilities.

With the help of the invention important advantages may be achieved:

• For the voice control systems the latency between spoken command and reaction played back on the audio system is a key performance parameter. The edge gateway of an electronic control device aggregates all incoming and outgoing data traffic streams and thus the proxy arranged there may prioritize the audio data over e.g. bulk data transfer.

• Speech data comes usually in small to very small data chunks, for example a coded audio data stream with 16kbit/s and a typical audio frame length of 20ms. Such a frame has 320 data bits/40 data bytes. However, the overhead of sending such a packet is already larger than the 40 data bytes. And response time matters but not to such extend of some 10s of ms. The invention may allow that said audio frames encompassing 100ms of speech are aggregated into one transmit frame.

• The speech reply from the digital assistant may be directly sent to the playback instance (usually the head device) installed in the vehicle, having improved sound characteristics. • The action command (“raw action command”) may be further translated into a vehicle specific command sequence, for example machine language.

Besides, the location of the proxy in the edge gateway of an electronic control device promotes faster access to all components in the vehicle. In addition, it is possible to easily load new skills into the proxy server due to the easy module update capability of the loT Edge runtime.

In some embodiments, during a communication between the vehicle and the backend device: high priority is assigned to audio data of the voice request and/or of the answer among different data to be transmitted between the vehicle and the cloud.

Thus, the voice requests of the user who wants to gain access to the digital assistant may be transmitted from the vehicle to the cloud prior to bulk data packets for other services which cause a network traffic jam between the vehicle and the backend device. Also, the answers of the digital assistant server may be transmitted from the backend device to the vehicle prior to other data coming from the backend device.

In some embodiments, for transmitting the voice request of the user from the proxy server to a digital assistant server audio data of the voice request are aggregated into: a defined transmit frame, a determined length and/or a defined transmit format.

Thus, improved audio data aggregation may be achieved.

In some embodiments, the method comprises at least one of the following: play back of the answer for the user by the play back device, especially by a head device of the vehicle, and/or visualize the answer for the user by a display device installed in the vehicle.

Thus, the play back device of the vehicle with improved acoustic characteristics may be used for the reply to the user command.

In some embodiments, the method comprises at least one of the following: providing an action command in the vehicle according to the answer from the digital assistant server, preferably by the proxy server for the digital assistant and/or by the digital assistant server. Thus, the extended services and/or improved skills of the vehicle may be controlled using digital assistant functionality in an easy and convenient way by voice requests. An action command may be provided in the backend device and/or in the vehicle itself.

In some embodiments, for providing an action command in the vehicle according to the answer from the digital assistant server the action command is transformed to: a specific vehicle command sequence, preferably by the proxy server for the digital assistant.

Thus, an action command provided by the backend device may be translated into the language, which the corresponding device of the vehicle understands for controlling functions.

In some embodiments, the method comprises at least one of the following: verbalize an action command according to the answer from the digital assistant server for the user by the play back device, especially by a head device of the vehicle, and/or visualize of an action command according to the answer from the digital assistant server for the user by a display device installed in the vehicle.

Thus, intuitive and convenient possibilities may be provided on how to show the user the result of his voice command. The user may then easily verify, if his voice command was interpreted correctly and will be processed in a desired way.

In some embodiments, the method comprises at least one of the following: providing a confirmation possibility for the user to approve an action command in the vehicle according to the answer from the digital assistant server through using the recording device and/or a display device and/or an input device installed in the vehicle.

Thus, the user may then intervene before an action command will be executed.

In some embodiments, the method comprises at least one of the following: execute an action command in the vehicle according to the answer from the digital assistant server, especially upon a confirmation from the user of the action command, preferably by an electronic control device of the vehicle. Thus, it may be ensured that the voice commands of the user lead to the desired function in the vehicle.

In some embodiments, the method comprises at least one of the following: provide at least one alternative action command according to the answer from the digital assistant server, especially upon a rejection from the user of the action command, preferably by the proxy server for the digital assistant, and/or invite the user to provide a new voice request by the play back device and/or a display device through using the recording device and/or a display device and/or an input device installed in the vehicle, especially upon a rejection from the user of the action command.

Thus, fail back positions may be provided for the case that the user voice request was not interpreted correctly.

According to a further possible use of the method, the method may be executed for synchronization of a digital calendar of the user in the backend device 20, in the vehicle 10 and/or in a mobile device 30 of a user. Thus, enhanced use of the method may be provided.

According to the second aspect, embodiments of the invention provide a system for a vehicle for providing a functionality of a digital assistant in the vehicle, comprising: a recording device installed in the vehicle for recording a voice request of a user, a special proxy server for the digital assistant in an edge gateway (of an electronic control device) of the vehicle, wherein the proxy server is configured for transmitting the voice request of the user from the vehicle to a digital assistant server (in a backend device) for performing a voice analysis and providing an answer as a result of the analysis, wherein the proxy server is configured for receiving the answer to the voice request from digital assistant server at the vehicle, and wherein the proxy server is configured for transmitting the answer from the proxy server to a play back device installed in the vehicle, and the play back device for verbalizing the answer for the user. With the help of the inventive system, the same advantages may be achieved as described above in the context of the inventive method. Full reference is made to these advantages in the present case.

In some embodiments, the proxy server is configured for providing: an action command in the vehicle according to the answer from the digital assistant server, and/or a transformation of an action command in the vehicle according to the answer from the digital assistant server into a specific vehicle command sequence, at least one alternative action command according to the answer from the digital assistant server, and/or especially upon a rejection from the user of the action command, an invitation for the user to provide a new voice request, especially upon a rejection from the user of the action command.

Advantageously the proxy server may be configured for synchronizing a digital calendar of the user in the backend device, in the vehicle and/or in a mobile device of a user, such as a smartphone.

Thus, improved functionality of the digital assistant may be provided in the vehicle using different services and features of the vehicle.

Embodiments of the invention provide, according to the third aspect, a vehicle comprising a corresponding system. With such a vehicle the same advantages may be achieved as with the system described above. Full reference is made to these advantages in the present case.

According to the fourth aspect, embodiments of the invention provide an electronic control device comprising: a memory, in which a program code is stored, and a computing device, wherein when carrying out the program code by the computing device, a method is performed as described above. With embodiments of such an inventive electronic control device the same advantages may be achieved as with corresponding embodiments of the method described above. Full reference is made to these advantages in the present case.

According to the fourth aspect, the invention provides a computer program product comprising a program code for carrying out a method as described above. In other words, the computer program product comprises instructions which, when the program is executed by a computer, cause the computer to carry out embodiments of the method described above. With embodiments of such a computer program product, the same advantages may be achieved as with corresponding embodiments of the method described above. Full reference is made to these advantages in the present case.

Embodiments of the invention and its further developments as well as its advantages will be explained in more detail below using the figure. The figure shows schematically:

Fig. 1 a schematic design of a system for a vehicle within the meaning of the invention configured for providing a functionality of a digital assistant (DA) in the vehicle.

Fig. 1 shows a system 100 for a vehicle 10 for providing a functionality of a digital assistant DA in the vehicle 10, a recording device 11 installed in the vehicle 10 for recording a voice request VR of a user, a special proxy server DAP for the digital assistant DA in an edge gateway GW of an electronic control device ECU of the vehicle 10, wherein the proxy server DAP is configured for transmitting the voice request VR of the user from the vehicle 10 to a digital assistant server DAS in a backend device 20 for performing a voice analysis and providing an answer AW as a result of the analysis, wherein the proxy server DAP is configured for receiving the answer AW to the voice request VR from digital assistant server DAS at the vehicle 10, and wherein the proxy server DAP is configured for transmitting the answer AW from the proxy server DAP to a play back device 12 installed in the vehicle 10, and the play back device 12 for verbalizing the answer AW for the user.

Furthermore, the proxy server DAP may be configured for providing: an action command ACT in the vehicle 10 according to the answer AW from the digital assistant server DAS, and/or a transformation of an action command ACT in the vehicle 10 according to the answer AW from the digital assistant server DAS into a specific vehicle command sequence, at least one alternative action command AACT according to the answer AW from the digital assistant server DAS, and/or especially upon a rejection from the user of the action command ACT, an invitation for the user to provide a new voice request VR, especially upon a rejection from the user of the action command ACT.

Preferably, the proxy server DAP may be configured for synchronizing a digital calendar of the user in the backend device 20, in the vehicle 10 and/or in a mobile device 30 of a user, such as a smartphone. A backend device 20 may be a cloud computer providing a high computational power, a large memory and easy access to updates as well as to a digital assistant software that may be used in the vehicle. The backend device 20 is thus the representative of the cloud.

The idea is to provide a proxy server DAP (or simply a proxy) that interfaces with the digital assistant DA in the backend device 20 arranged in the cloud. The invention proposes to allocate the proxy to the edge gateway GW of an electronic control device ECU of the vehicle 10 and within the edge gateway GW of an electronic control device ECU especially to the loT Edge runtime for the sake of easy update possibilities which is paramount if new skills are to be added to the proxy.

Since the loT traffic can be bulky, it may be preferably provided according to some embodiments of the invention that the traffic from the loT Edge runtime to the cloud suspends the transmission of large loT information packets in favor of higher priority audio data of the user voice request or commands.

Advantageously, the audio stream coming from the vehicle may be aggregated to larger packets by the proxy server DAP in the loT Edge runtime.

Further, it may be advantageous if the voice request reply (that is answer from the backend device 20) may be directly sent to the element installed in the vehicle 10 which plays back the answer with better sound quality, such as head device as a possible play back device 12 of the vehicle 10.

Furthermore, an action command (so called “raw action command”) corresponding to the answer to the voice request may be sent to the proxy server DAP and within the proxy server DAP further detailed according to the available skills and vehicle capabilities.

With the help of the inventive system 100 and the corresponding method, important advantages may be achieved:

• The latency between spoken command and reaction played back on the audio system within the vehicle may be drastically reduced. The performance of the digital assistant functions may be preferably increased. The edge gateway GW of an electronic control device ECU aggregates all incoming and outgoing data traffic streams and thus the proxy arranged there may easily prioritize the audio data over e.g. bulk data transfer.

• The invention may allow that audio data may be aggregated into one transmit frame for better throughput of the audio data. • The speech reply from the digital assistant may be directly sent to the playback instance (usually the head device) installed in the vehicle, having improved sound properties.

• The action command (“raw action command”) may be further translated into a vehicle specific command sequence, for providing extended functionality of the vehicle with the help of the digital assistant DA.

Furthermore, the location of the proxy server DAP in the edge gateway GW of the electronic control device ECU provides faster access to all components in the vehicle. In addition, it is possible to easily load new skills into the proxy server DAP due to the easy module update capability of the loT Edge runtime.

The vehicle 10 comprising a corresponding system 100 provides the third aspect of the invention.

An electronic control device ECU comprising a memory, in which a program code is stored, and a computing device, wherein when carrying out the program code by the computing device, a method is performed as described above, provides the fourth aspect of the invention. In other words, the electronic control device ECU is configured in such a way to execute embodiments of the method as described above.

Also, a computer program product comprising a program code for carrying out a method as described above provides an aspect of the invention.

The above description of the figure describes the present invention only in the context of examples. Of course, individual features of the embodiments may be combined with each other, provided it is technically reasonable, without leaving the scope of the invention.

Reference signs

100 system

10 vehicle

11 recording device

12 play back device

14 display device

15 input device

20 backend device

30 mobile device

AW answer

ACT action command

AACT alternative action command

DA digital assistant

DAP proxy server

DAS digital assistant server

DAMP digital assistant server on mobile device

ECU electronic control device

GW edge gateway

VR voice request

Claims

Claims A method for providing a functionality of a digital assistant (DA) in a vehicle (10), the method comprising the following: providing a recording device (11 ) installed in the vehicle (10) for recording a voice request (VR) of a user, providing a special proxy server (DAP) for the digital assistant (DA) in an edge gateway (GW) of an electronic control device (ECU) of the vehicle (10), transmitting the voice request (VR) of the user from the proxy server (DAP) to a digital assistant server (DAS) in a backend device (20) for performing a voice analysis and providing an answer (AW) as a result of the analysis, receiving the answer (AW) to the voice request (VR) from a digital assistant server (DAS) at the proxy server (DAP), and transmitting the answer (AW) from the proxy server (DAP) to a play back device (12) installed in the vehicle (10). The method according to claim 1 , wherein by a communication between the vehicle (10) and the backend device (20): high priority is assigned to audio data of the voice request (VR) and/or of the answer (AW) among different data to be transmitted between the vehicle (10) and the cloud (20). The method according to any one of the preceding claims 1 or 2, wherein for transmitting the voice request (VR) of the user from the proxy server (DAP) to a digital assistant server (DAS) audio data of the voice request (VR) are aggregated into: a defined transmit frame, a determined length and/or a defined transmit format. The method according to any one of the preceding claims, wherein the method comprises at least one of the following: play back of the answer (AW) for the user by the play back device (12), especially by a head device of the vehicle (10), and/or visualize the answer (AW) for the user by a display device (14) installed in the vehicle (10). The method according to any one of the preceding claims, wherein the method comprises at least one of the following: providing an action command (ACT) in the vehicle (10) according to the answer (AW) from the digital assistant server (DAS), preferably by the proxy server (DAP) for the digital assistant (DA) and/or by the digital assistant server (DAS). The method according to any one of the preceding claims, wherein for providing an action command (ACT) in the vehicle (10) according to the answer (AW) from the digital assistant server (DAS) the action command (ACT) is transformed to: a specific vehicle command sequence, preferably by the proxy server (DAP) for the digital assistant (DA). The method according to any one of the preceding claims, wherein the method comprises at least one of the following: verbalize an action command (ACT) according to the answer (AW) from the digital assistant server (DAS) for the user by the play back device (12), especially by a head device of the vehicle (10), and/or visualize of an action command (ACT) according to the answer (AW) from the digital assistant server (DAS) for the user by a display device (14) installed in the vehicle (10). The method according to any one of the preceding claims, wherein the method comprises at least one of the following: providing a confirmation possibility for the user to approve an action command (ACT) in the vehicle (10) according to the answer (AW) from the digital assistant server (DAS) through using the recording device (11 ) and/or a display device (14) and/or an input device (15) installed in the vehicle (10).

14 The method according to any one of the preceding claims, wherein the method comprises at least one of the following: execute an action command (ACT) in the vehicle (10) according to the answer (AW) from the digital assistant server (DAS), especially upon a confirmation from the user of the action command (ACT), preferably by an electronic control device (ECU) of the vehicle (10), and/or the method is executed for synchronization of a digital calendar of the user in the backend device (20), in the vehicle (10) and/or in a mobile device (30) of a user. The method according to any one of the preceding claims, wherein the method comprises at least one of the following: provide at least one alternative action command (AACT) according to the answer (AW) from the digital assistant server (DAS), especially upon a rejection from the user of the action command (ACT), preferably by the proxy server (DAP) for the digital assistant (DA), and/or invite the user to provide a new voice request (VR) by the play back device (12) and/or a display device (14) through using the recording device (11 ) and/or a display device (14) and/or an input device (15) installed in the vehicle (10), especially upon a rejection from the user of the action command (ACT). A System (100) for a vehicle (10) for providing a functionality of a digital assistant (DA) in the vehicle (10), comprising: a recording device (11 ) installed in the vehicle (10) for recording a voice request (VR) of a user, a special proxy server (DAP) for the digital assistant (DA) in an edge gateway (GW) of an electronic control device (ECU) of the vehicle (10), wherein the proxy server (DAP) is configured for transmitting the voice request (VR) of the user from the vehicle (10) to a digital assistant server (DAS) in a backend device (20) for performing a voice analysis and providing an answer (AW) as a result of the analysis, wherein the proxy server (DAP) is configured for receiving the answer (AW) to the voice request (VR) from digital assistant server (DAS) at the vehicle (10), and wherein the proxy server (DAP) is configured for transmitting the answer (AW) from the proxy server (DAP) to a play back device (12) installed in the vehicle (10), and the play back device (12) for verbalizing the answer (AW) for the user. 15 The System (100) according to the preceding claim, wherein the proxy server (DAP) is configured for providing: an action command (ACT) in the vehicle (10) according to the answer (AW) from the digital assistant server (DAS), and/or a transformation of an action command (ACT) in the vehicle (10) according to the answer (AW) from the digital assistant server (DAS) into a specific vehicle command sequence, at least one alternative action command (AACT) according to the answer (AW) from the digital assistant server (DAS), and/or especially upon a rejection from the user of the action command (ACT), an invitation for the user to provide a new voice request (VR), especially upon a rejection from the user of the action command (ACT), and/or the proxy server (DAP) is configured for synchronizing a digital calendar of the user in the backend device (20), in the vehicle (10) and/or in a mobile device (30) of a user. A vehicle (10) comprising a system (100) according to any one of the preceding claims 11 or 12. An electronic control device (ECU) comprising: a memory, in which a program code is stored, and a computing device, wherein when carrying out the program code by the computing device, a method according to any one of the proceeding claims 1 to 10 is performed. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 10.