DE102018128006A1 - NATURAL LANGUAGE GENERATION BASED ON THE USER LANGUAGE STYLE - Google Patents

Abstract

A system and method for generating natural language generation (NLG) outputs, the method comprising: receiving speech signals from a user at a microphone of a client device; Determining a requested communication destination and at least one input communication value based on the received speech signals; Determining the use of a static natural language generation (NLG) template or a NLG dynamic template to produce an NLG output, determining whether to use a static NLG template or a dynamic NLG template using a selection process for NLG templates for neural networks are done; Selecting an NLG template after determining whether to use a static NLG template or a dynamic NLG template; and generating an NLG output based on the selected NLG template.

Description

INTRODUCTION

The present invention relates to generating Natural Language Generation (NLG) output based on characteristics or style of a user's language.

Using automatic speech recognition (ASR), various device functions can be controlled on a client device. Vehicles include z. For example, hardware and software capable of receiving speech from a vehicle occupant, processing that language to understand speech content, and then performing an action based on the speech content. The vehicles or other client devices may use the hardware and software to process the speech received at the vehicle. A response to an ASR input may be generated at the vehicle or at the client device.

SUMMARY

According to one aspect of the invention, there is provided a method of generating natural language generation (NLG) outputs, the method comprising: receiving speech signals from a user at a microphone of a client device; Determining a requested communication destination and at least one input communication value based on the received speech signals; Determining the use of a static natural language generation (NLG) template or a NLG dynamic template to produce an NLG output, determining whether to use a static NLG template or a dynamic NLG template using a selection process for NLG templates for neural networks are done; Selecting an NLG template after determining whether to use a static NLG template or a dynamic NLG template; and generating an NLG output based on the selected NLG template.

• • den Schritt zum Identifizieren eines Benutzers oder eines wahrscheinlichen Benutzers, der die Sprachsignale übertragen hat;
• • den Auswahlprozess für NLG-Vorlagen für neuronale Netzwerke wird basierend auf einer Historie des identifizierten Benutzers oder des wahrscheinlichen Benutzers und/oder einem Profil des identifizierten Benutzers oder des wahrscheinlichen Benutzers durchgeführt;
• • die ausgewählte NLG-Vorlage wird basierend auf der Historie des identifizierten Benutzers oder des wahrscheinlichen Benutzers und/oder dem Profil des identifizierten Benutzers oder des wahrscheinlichen Benutzers ausgewählt;
• • den Schritt zum Identifizieren eines Stils basierend auf den empfangenen Sprachsignalen, der Historie des identifizierten Benutzers oder wahrscheinlichen Benutzers und/oder dem Profil des identifizierten Benutzers oder wahrscheinlichen Benutzers, und wobei der identifizierte Stil in dem Auswahlprozess für NLG-Vorlagen für neuronale Netzwerke verwendet wird;
• • den Schritt, wenn bestimmt wird, dass eine dynamische NLG-Vorlage verwendet werden soll, um die NLG-Ausgabe zu erzeugen, und dann Erzeugen der dynamischen NLG-Vorlage unter Verwendung eines Erzeugungsprozesses für NLG- Vorlagen;
• • die dynamische NLG-Vorlage wird basierend auf dem Kommunikationsziel und einem oder mehreren erhaltenen Kommunikationswerten erzeugt;
• • den Schritt zum Identifizieren eines Benutzers oder eines wahrscheinlichen Benutzers, der die Sprachsignale übertragen hat, wobei die dynamische NLG-Vorlage basierend auf Benutzerinformationen erzeugt wird, die dem Benutzer oder wahrscheinlichen Benutzer zugeordnet sind, und wobei die Benutzerinformationen entweder Benutzerhistorieninformationen oder Benutzerprofilinformationen oder beide dieser beinhalten;
• • der Erzeugungsprozesses für NLG-Vorlagen ist ein regelbasierter Erzeugungsprozesses für NLG-Vorlagen;
• • der Erzeugungsprozesses für NLG-Vorlagen ist ein Erzeugungsprozesses für NLG-Vorlagen neuronaler Netzwerke;
• • die Schritte: Senden einer Kommunikationswertanforderung an einen entfernten Server, wobei die Kommunikationswertanforderung eine Anforderung ist, um reaktive Kommunikationswerte zu erhalten, die in der NLG-Ausgabe zu verwenden sind und die auf dem Kommunikationsziel und den eingegebenen Kommunikationswerten basieren; und Empfangen einer Kommunikationswertreaktion von dem entfernten Server, wobei die Kommunikationswertantwort reaktive Kommunikationswerte beinhaltet, die beim Erzeugen der NLG-Ausgabe zu verwenden sind;
• • den Schritt des Anpassens des Auswahlprozesses für NLG-Vorlagen neuronaler Netzwerke basierend auf früheren Iterationen des Auswahlprozesses für NLG-Vorlagen neuronaler Netzwerke; und/oder
• • der Anpassungsschritt umfasst ferner Anpassen des Auswahlprozesses für NLG-Vorlagen für neuronale Netzwerke basierend auf einem Satz von Eingaben, der NLG-Ausgabe und einem gemessenen Erfolg des Auswahlprozesses für NLG-Vorlagen neuronaler Netzwerke, wobei der Satz von Eingaben das Kommunikationsziel, die eingegebenen Kommunikationswerte und Informationen bezüglich eines Benutzers oder eines wahrscheinlichen Benutzers, der die Sprachsignale übertragen hat, beinhaltet.

Gemäß einem weiteren Aspekt der Erfindung ist ein Verfahren zum Erzeugen von Ausgaben natürlichsprachlicher Generierung (NLG) vorgesehen, wobei das Verfahren Folgendes beinhaltet: Empfangen von Sprachsignalen von einem Benutzer an einem Mikrofon einer Clientvorrichtung; Identifizieren eines Benutzers oder eines wahrscheinlichen Benutzers, der die Sprachsignale übertragen hat; Bestimmen eines Kommunikationsziels und mindestens eines eingegebenen Kommunikationswertes basierend auf den empfangenen Sprachsignalen; Bestimmen, ob eine statische NLG-Vorlage oder eine dynamische NLG-Vorlage zum Verwenden beim Erzeugen einer NLG-Ausgabe verwendet werden soll, wobei Bestimmen, ob eine statische NLG-Vorlage oder eine dynamische NLG-Vorlage verwendet werden soll, unter Verwendung eines Auswahlprozesses für NLG-Vorlagen neuronaler Netzwerke erfolgt, wobei der Auswahlprozess für NLG-Vorlagen neuronaler Netzwerke ein künstliches neuronales Netzwerk verwendet, um einen Satz von Eingaben zu lösen, um auszuwählen, ob eine statische NLG-Vorlage oder eine dynamische NLG-Vorlage zum Verwenden beim Erzeugen einer NLG-Ausgabe verwendet werden soll, und wobei der Satz von Eingaben das Kommunikationsziel, die eingegebenen Kommunikationswerte und entweder eine Benutzerhistorie, die dem Benutzer zugeordnet ist, oder ein Benutzerprofil, das dem Benutzer zugeordnet ist, beinhaltet;According to various embodiments, this method may further include one of the following features or a technically possible combination of these features:

• The step of identifying a user or a likely user who has transmitted the voice signals;
• The NLG template selection process for neural networks is performed based on a history of the identified user or the probable user and / or a profile of the identified user or the probable user;
• The selected NLG template is selected based on the history of the identified user or the probable user and / or the profile of the identified user or the probable user;
• The step of identifying a style based on the received speech signals, the history of the identified user or probable user, and / or the profile of the identified user or probable user, and wherein the identified style is used in the neural network NLG template selection process ;
• The step of determining that an NLG dynamic template is to be used to generate the NLG output and then generating the NLG dynamic template using a NLG template generation process;
• The dynamic NLG template is generated based on the communication destination and one or more received communication values;
• The step of identifying a user or a probable user who has transmitted the voice signals, wherein the dynamic NLG template is generated based on user information associated with the user or likely user, and wherein the user information is either user history information or user profile information, or both of these include;
• • the NLG template production process is a rule-based production process for NLG templates;
• • the NLG template generation process is a generation process for NLG templates of neural networks;
• The steps of: sending a communication value request to a remote server, the communication value request being a request to obtain reactive communication values to be used in the NLG output based on the communication destination and the entered communication values; and receiving a communication value response from the remote server, the communication value response including reactive communication values to be used in generating the NLG output;
• The step of adapting the selection process for NLG templates of neural networks based on previous iterations of the selection process for NLG templates of neural networks; and or
• The adaptation step further comprises adjusting the neural network NLG template selection process based on a set of inputs, the NLG output, and a measured success of the neural network NLG template selection process, wherein the set of inputs is the communication destination, the input communication values and information regarding a user or a probable user who has transmitted the voice signals.

According to another aspect of the invention, there is provided a method of generating outputs of natural language generation (NLG), the method comprising: receiving speech signals from a user at a microphone of a client device; Identifying a user or a likely user who has transmitted the voice signals; Determining a communication destination and at least one input communication value based on the received speech signals; Determine whether to use a static NLG template or an NLG dynamic template for use in generating NLG output, determining whether to use a NLG static template or a NLG dynamic template using a selection process for NLG templates of neural networks, where the neural network NLG template selection process uses an artificial neural network to solve a set of inputs to select whether to use a NLG static template or a NLG dynamic template for use in creating a NLG template NLG output, and wherein the set of inputs includes the communication destination, the input communication values, and either a user history associated with the user or a user profile associated with the user;

• • der Satz von Eingaben beinhaltet sowohl die Benutzerhistorie, die dem Benutzer zugeordnet ist, als auch das Benutzerprofil, das dem Benutzer zugeordnet ist;
• • der Auswahlprozess für NLG-Vorlagen neuronaler Netzwerke beinhaltet einen Satz von potenziellen Ausgaben, die einen ersten Ausgabewert für NLG-Vorlagen und einen zweiten Ausgabewert für NLG-Vorlagen beinhalten, wobei der erste Ausgabewert für NLG-Vorlagen einer Wahrscheinlichkeit entspricht, dass eine statische NLG-Vorlage zum Erzeugen der NLG-Ausgabe verwendet wird, und wobei der zweite Ausgabewert für NLG-Vorlagen einer Wahrscheinlichkeit entspricht, dass eine dynamische NLG-Vorlage zum Erzeugen der NLG-Ausgabe verwendet wird;
• • der Identifizierungsschritt beinhaltet Verwenden der empfangenen Sprachsignale zum Identifizieren des Benutzers, wobei bestimmt wird, dass eine statische NLG-Vorlage oder eine dynamische NLG-Vorlage basierend auf einem identifizierten Stil, einer Emotion und/oder einem Dialekt der empfangenen Sprachsignale verwendet wird;
• • die dynamische NLG-Vorlage wird erzeugt und/oder die statische NLG-Vorlage wird basierend auf dem identifizierten Stil, der Emotion und/oder dem Dialekt der empfangenen Sprachsignale ausgewählt;
• • die dynamische NLG-Vorlage wird unter Verwendung eines maschinell lernenden Erzeugungsprozesses für NLG-Vorlagen erzeugt, wobei der maschinelle lernende Erzeugungsprozesses für NLG-Vorlagen eine andere maschinelle Lernmethode verwendet als die, die im Auswahlprozess für NLG-Vorlagen neuronaler Netzwerke verwendet wird; und/oder
• • die unterschiedliche maschinelle Lernmethode ist ein regelbasierter Ansatz, der einen vordefinierten Satz von Regeln beinhaltet, die beim Erstellen der dynamischen NLG-Vorlage eingehalten werden.

According to various embodiments, this method may further include one of the following features or a technically possible combination of these features:

• The set of inputs includes both the user history associated with the user and the user profile associated with the user;
• • The neural network NLG template selection process includes a set of potential outputs that include a first output value for NLG templates and a second output value for NLG templates, where the first output value for NLG templates corresponds to a probability that a static NLG Template is used for generating the NLG output, and wherein the second output value for NLG templates corresponds to a probability that an NLG dynamic template is used for generating the NLG output;
• The identifying step includes using the received speech signals to identify the user, determining that a NLG static template or a NLG dynamic template is used based on an identified style, emotion, and / or dialect of the received speech signals;
• The NLG dynamic template is generated and / or the NLG static template is selected based on the identified style, emotion and / or dialect of the received speech signals;
• The dynamic NLG template is generated using a machine learning generation process for NLG templates, the NLG template machine learning generation process using a different machine learning method than that used in the selection process for NLG templates of neural networks; and or
• • The different machine learning method is a rule-based approach that includes a predefined set of rules that are followed when creating the NLG dynamic template.

list of figures

• 1 ist ein Blockdiagramm, das eine Ausführungsform eines Kommunikationssystems abbildet, das in der Lage ist, das hierin offenbarte Verfahren zu nutzen;
• 2 ist ein Blockdiagramm, das eine Ausführungsform eines automatischen Spracherkennungssystems (ASR) abbildet;
• 3 ist ein Flussdiagramm einer Ausführungsform eines Verfahrens zum Erzeugen einer Ausgabe natürlichsprachlicher Generierung (NLG); und
• 4 ist ein Blockdiagramm, das eine Ausführungsform eines neuronalen Netzwerkmodells abbildet, das in einem maschinell lernenden NLG-Prozess oder einer maschinell lernenden NLG-Anwendung verwendet werden kann.

One or more embodiments of the invention will be described below in conjunction with the accompanying drawings, wherein like designations denote like elements, and wherein:

• 1 Figure 10 is a block diagram depicting one embodiment of a communication system capable of utilizing the method disclosed herein;
• 2 Fig. 10 is a block diagram depicting one embodiment of an Automatic Speech Recognition System (ASR);
• 3 Fig. 10 is a flowchart of one embodiment of a method for generating a Natural Language Generation (NLG) output; and
• 4 Figure 10 is a block diagram depicting one embodiment of a neural network model that may be used in a machine learning NLG process or a machine learning NLG application.

DETAILED DESCRIPTION

The system and method described below provides for generating Natural Language Generation (NLG) output using a machine learning NLG process, which may, at least in some embodiments, choose between using a static (or predefined) NLG template and using a NLG dynamic template , The NLG template can be selected to tailor the selected NLG template to a particular user, ASR input, and / or a combination thereof. Natural language generation (NLG) refers to the process of generating a natural language output that conveys a statement, request or question or other intent from a machine to a user. In many cases, NLG templates are used to produce NLG output. A machine learning selection process for NLG templates can be performed as part of the machine learning NLG process. The selection process for NLG templates is the process of selecting to use a static or dynamic NLG template using machine-learning methods. The NLG template may be selected based on specific user information, a communication destination, or a communication intent or various other information, as discussed below.

As used herein, an "NLG template" is a collection of predefined or predefined phrases, sentences, and / or statements that include one or more input fields in which a particular value or word is placed based on the desired or intended output can. In addition, as used herein, an "NLG output" is any output that is based on a natural language and that, at least in some cases, can be audibly transmitted using a loudspeaker (ie, an audible NLG output) or visualized , such as B. Presenting a textual representation containing text characters (eg, alphanumeric characters). And, as used herein, a machine learning NLG process is the process of using machine learning methods to select an NLG template, generate an NLG template, and / or produce an NLG output, and therefore, in at least some embodiments, can machine learning selection process for NLG templates. If a neural network model is used, the NLG process can be referred to as a neural network NLG process (and the same goes for the NLG machine learning subprocesses).

As explained in more detail below, the machine learning NLG process may include various processes (or "sub-processes"), including a selection process for NLG templates, a NLG template generation process and / or a NLG output generation process. The selection process for NLG templates is the process of selecting an NLG template to be used in generating the NLG output, choosing between a static and a dynamic template, based on a communication destination, a style of the received language, a user profile , a user history or various other factors. The NLG generation process may be executed when a dynamic template for generating the NLG output is determined. The NLG output generation process is the process for generating the NLG output using the selected and / or generated NLG template.

In some scenarios, the NLG template selection process may determine that a static NLG template is used, which is an NLG template predefined to map specific communication destinations to pre-defined NLG template outputs. And in other embodiments, the NLG template selection process may determine that to use a NLG dynamic template that is an NLG template that is at least partially generated using the NLG template automatic learning generation process at a time after the selection process for NLG templates NLG templates, and may be based, at least in some embodiments, on deriving certain template-related outputs from different predefined or static NLG templates. If a static NLG template is selected, the NLG output process (that is, the process of using NLG templates to produce NLG output) can be performed immediately. If it is determined that a NLG dynamic template should be used in the NLG output generation process, a NLG template generation process can be used to generate the NLG dynamic template. The dynamic NLG template may be generated based on a communication destination that identifies from received speech signals, a style (or other attributes) of the received speech signals, an emotion of the received speech signals, a persona of the received speech, or based on a user profile or history becomes. This way a can dynamic NLG template can be customized, which can be adapted to the communication style in order to improve the user-friendliness.

For example, the NLG dynamic template may be tailored to a particular user and / or generated, updated, or changed based on ASR input from a particular user. And, in certain embodiments, when the same communication destination and the same or similar style or emotion is repeatedly used by a user to query the ASR system, a dynamic NLG template may be stored to speed future NLG processes. For example, instead of creating the same NLG dynamic template, the NLG dynamic template can be stored and later retrieved from memory when it is designated for use, thereby avoiding the need to recreate the NLG dynamic template. It should be noted that the dynamic NLG template is saved to become a static NLG template. In addition, the machine learning NLG process (and sub-processes) can use a variety of machine learning methods, including artificial neural networks, to select between NLG templates and / or to update or modify NLG templates.

In one embodiment, a natural language generation (NLG) neural network (NLG) selection process may be used to select an NLG template based on a style (or emotion) of received speech signals and a communication destination identified based on the received speech signals. In one scenario, a first user may speak in a verbose and extravagant manner, and so a neural network NLG template selection process can be used to select a verbose NLG template that, when used for NLG, gives a verbose and extravagant response. For example, if a user or a user's language is recognized as talkative or verbose, an NLG template may be used to produce NLG output resulting in a talkative or verbose output; For example, if a user says, "Hello ASR system, what is the weather forecast for my current location," the system can answer, "Hello user 1 , today's weather is cloudy. The maximum temperature is 43 degrees Fahrenheit and the minimum temperature is 31 degrees Fahrenheit. "In another scenario, a second user can use very few words, and so an NLG template tailored to a more direct and concise style can be used in the NLG selection process Templates of neural networks are selected. Here you can select an NLG template that produces the following output: "Cloudy, high temperature 43 , Low temperature 31 . "The neural network NLG template selection process may select a static template based on the style or emotion of the user's language or may select using a NLG dynamic template. In the latter case, a NLG template generation process can be performed to generate a NLG dynamic template based on the style or emotion of the user's language.

Continuing to refer to the previous two examples, the first NLG template (the detailed NLG template) used a predefined set tailored to provide weather information: "Today's weather is <weather_condition>. The maximum temperature is <high_temp> degrees Fahrenheit and the minimum is <low_temp> degrees Fahrenheit. "The second NLG template (the direct NLG template) used a template phrase that was also tailored to provide weather information:"<weather_condition> , High temperature <high_temp>, low temperature <low_temp>". Here "<weather_condition>" is a placeholder or input field for the overall weather situation, such as: Cloudy, sunny, partly cloudy, etc., "<high_temp>" is the daily high temperature and "<low_temp>" is the daily low temperature. In addition, the first user said "Hello ASR system", and so the NLG engine learning process appended a "hello" statement to the beginning of the NLG output using the first NLG template to match the ASR input , The placeholders or input fields may provide a location within the phrase or phrase into which a reactive communication value may be inserted. For example, the reactive communication value for low temp " 31 "And using one of the templates, the reaction communication value can be inserted into the input field. In one embodiment, the first template may be a static template configured to be used for use when the communication destination is the provision of local weather and when the user or the user's language is verbose. In another embodiment, the NLG process may determine that there is no suitable static template for the response to the first user, and so may a dynamic template based on the verbosity of the received speech signals of the first user and / or on other information, such as For example, the communication goal to be developed.

The machine learning process NLG, including the subprocesses described above, may be implemented in a computer-executable application that may be executed on a variety of different client devices, including hand-held mobile devices (eg, smartphones), home automation devices (e.g., smart personal computers) Assistants such as Amazon ™ Alexa ™ and Google ™ Home), vehicles, and / or any device that receives voice, can connect to a remote computer network, and can be configured to implement the machine learning NLG process or application described herein , In a particular embodiment, a vehicle may be used as a client device to receive and perform the machine learning NLG process. In other embodiments, the machine learning NLG process may be performed at a remote device that may have access to multiple resources (eg, multiple data) and / or higher processing capabilities. The NLG machine-learning process, including the NLG template machine-learning selection process, may be based on numerous inputs, including user history, user profile, communication destination, and communication value.

With reference to 1 , an operating environment is represented, which is a communication system 10 and which may be used to implement the method disclosed herein. The communication system 10 generally includes a vehicle 12 with a body control module (BCM) 26 and a wireless communication device 30 , a constellation of satellites of the Global Navigation Satellite System (GNSS) 60 , one or more wireless carrier systems 70 , a land communication network 76 , a computer 78 , a remote facility 80 and a personal mobile device 90 , It should be understood that the disclosed method may be used with any number of different systems and is not specifically limited to the operating environment shown herein. Also the architecture, construction, configuration and general operation of the system 10 and its individual components are well known in the art. Thus, the following paragraphs merely provide a brief overview of such a communication system 10 ready; but other systems not shown here could employ the disclosed methods.

The wireless carrier system 70 can be any suitable mobile phone system. The carrier system 70 is with a mobile mast 72 shown; however, the carrier system can 70 include one or more of the following components (eg, depending on the mobile technology): cell towers, base transmitting stations, mobile switching centers, base station controllers, developed nodes (e.g., eNodeBs), mobility management units (MMEs), serving and PGN gateways, etc and all other network components that are required to support the wireless carrier system 70 with the landline 76 or the wireless carrier system with the user equipment (UEs, eg the telematics equipment in the vehicle 12 may involve) to connect. The carrier system 70 can implement any suitable communication technique, including GSM / GPRS technology, CDMA or CDMA2000 technology, LTE technology, etc. Generally, there are wireless carrier systems 70 , their components, the arrangement of their components, the interaction of the components, etc., widely known in the art.

Apart from using the wireless carrier system 70 For example, a different wireless carrier system may be used in the form of satellite communications to provide unidirectional or bidirectional communication with the vehicle. This may be done using one or more communication satellites (not shown) and an uplink transmitting station (not shown). The unidirectional communication may be, for example, satellite radio services wherein programmed content data (news, music, etc.) is received from the uplink transmitting station, packed for uploading, and then sent to the satellite, which transmits programming to the subscribers. Bidirectional communication may be, for example, satellite telephony services using the one or more communication satellites to facilitate telephone communications between the vehicle 12 and the uplink station. In use, this satellite telephony can be either in addition to or instead of the wireless carrier system 70 be used.

The landline 76 may be a conventional land-based telecommunications network connected to one or more landline telephones and the wireless carrier system 70 with the remote location 80 combines. For example, the landline 76 a telephone network (PSTN) as used to provide landline telephony, packet-switched data communications and the Internet infrastructure. One or more segments of the fixed network 76 could be through the use of a standard landline, a fiber optic network or other fiber optic network, a cable network, through the use of power lines, other wireless networks, such as local wireless networks (WLAN) or networks that provide wireless broadband access (BWA) or any combination thereof.

The computer 78 (only one shown) may be some of a number of computers accessible over a private or public network, such as the Internet. Each of these computers 78 may be a client device that can perform speech processing and / or at least part of the machine learning NLG process. For other such accessible computers 78 For example, it may be: a computer at a service center where diagnostic information and other vehicle data may be uploaded from the vehicle; a client computer used by the vehicle owner or other subscriber for such purposes as accessing or receiving vehicle data or for setting or configuring subscriber preferences or controlling vehicle functions; a car sharing server that coordinates reservations and / or registrations of multiple users requesting the use of a vehicle as part of a car sharing service; or a third-party storage location, the vehicle data or other information either by communicating with the vehicle 12 or the remote facility 80 or both. A computer 78 may also be used for providing Internet connectivity, such as DNS services, or as a network address server using DHCP or other suitable protocol to the vehicle 12 assign an IP address.

The remote facility 80 can be designed to handle the vehicle electronics 20 and the mobile device 90 provides a variety of system backend capabilities using one or more electronic servers. For example, the remote device 80 partially used to allow or coordinate information between the vehicle 12 and one or more client devices, such as the mobile device 90 or the computer 78 to be sent. In one embodiment, the remote device 80 Provide voice recognition services that may include receiving voice signals from a customer device and processing the received voice signals using a voice recognition system. Additionally or alternatively, the remote device may 80 include one or more network switches, servers, databases, live advisors, and an automated voice response system (VRS), all of which are known in the art. The remote location 80 may include any or all of the various components, wherein all of the various components are preferably coupled together via a wired or wireless local area network. The remote location 80 receives and transmits data via a landline 76 connected modem.

The remote facility 80 may also include one or more databases that may store account information such as subscriber authentication information, vehicle identifiers, profile records, behavior patterns, speech recognition, and / or NLG information (including NLG templates in a NLG template database) and other relevant subscriber information. As used herein, speech recognition and / or NLG information includes information that may be used to train a speech recognition or NLG system, such as the machine learning NLG process discussed in detail below. In some embodiments, the remote device may 80 perform one or more steps of the NLG process and / or supplement the NLG process by, for example, providing useful information to the client device performing the NLG process. Data transmissions can also be made by wireless systems, such. Eg IEEE 802.11x , GPRS and the like, take place. Professionals in the field will recognize that although only a remote facility 80 and a computer 78 in the illustrated embodiment, numerous remote devices 80 and / or computer 78 can be used.

The personal mobile device 90 is a mobile device and may include hardware, software, and / or firmware that facilitate cellular and short-range wireless communication (SRWC) and other mobile device applications. As used herein, a personal mobile device is a mobile device that is SRWC-enabled and that is portable by a user and that the portability of the device is at least partially dependent on the user, such as a portable device (e.g. Smartwatch), an implantable device, or a handheld device (for example, a smartphone, a tablet, a laptop). As used herein, a short-range wireless communication (SRWC) device is an SRWC-enabled device. The personal mobile device 90 may be a client device and may include a processor and memory (eg, a non-transitory computer-readable medium configured to operate with the processor) for storing the software, firmware, and so on. The personal mobile devices and the memory may be various software applications 92 allow the user (or manufacturer) to pre-install or install (for example, with a software application or graphical user interface (GUI)).

An implementation of an application 92 for mobile devices may enable receiving speech and processing the received speech using speech recognition techniques, some of which may involve generating a Natural Language Generation (NLG) response. In some embodiments, the mobile device may 90 a client device performing the machine learning NLG process. For example, the mobile device 90 include a microphone that allows reception of speech waves generated by one or more users. In a particular embodiment, the selection of NLG templates may be performed on the mobile device according to the NLG process of neural networks discussed below. In some embodiments, the application 92 or another mobile device application may include a graphical user interface (GUI) that allows a user to enter credentials, transmit credentials for authorization and / or authentication, connect to the vehicle 12 to enter vehicle status information, to request vehicle functions, to configure functions to be performed and / or to configure one or more vehicle settings. The mobile device 90 can with the wireless communication device 30 according to one or more SRWC technologies or wired connections, such as a connection via a Universal Serial Bus (USB) cable. Although a single mobile device 90 is displayed, the communication can 10 a variety of mobile devices 90 include.

The vehicle 12 is illustrated as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), ships, aircraft, etc., may also be used. Part of the vehicle electronics 20 is generally in 1 and includes a Global Navigation Satellite System (GNSS) module 22 , an engine control unit (ECU) 24 , a Body Control Module (BCM) 26 , a wireless communication device 30 that can be configured to perform a machine learning NLG process and / or voice recognition processing, other vehicle system modules (VSMs) 42 and numerous other components and devices. A part or the entire vehicle electronics can communicate with each other via one or more communication buses, such. Bus 44 , get connected. communication 44 provides vehicle electronics with network connections using one or more network protocols. Examples of suitable network connections include a Controller Area Network (CAN), a Media Oriented System Transfer (MOST), a Local Area Network (LIN), a Local Area Network (LAN), and other suitable connections, such as: Ethernet, or others that conform to known ISO, SAE and IEEE standards and specifications, among others.

The vehicle 12 can use numerous vehicle system modules (VSMs) as part of vehicle electronics 20 include, for example, the GNSS module 22 , ECU 24 , BCM 26 , the wireless communication device 30 and the vehicle user interfaces 52 -58, as described in more detail below. The vehicle 12 can also use other VSMs 42 in the form of electronic hardware components residing throughout the vehicle receiving an input from one or more sensors and using the sensed inputs to perform diagnostic, monitoring, control, reporting and / or other functions. Each of the VSMs 42 can over the communication bus 44 can be connected to the other VSMs and can be programmed to perform vehicle system tests and subsystem diagnostics. One or more VSMs 42 may update their software or firmware periodically or occasionally and, in some embodiments, such vehicle updates may be over-the-air (OTA) updates made by a computer 78 or a remote location 80 over the landline 76 and communication devices 30 be received. Those skilled in the art will recognize that the aforementioned VSMs are only examples of some of the modules used in the vehicle 12 can be used, but numerous other modules are also possible.

The Global Navigation Satellite System (GNSS) module 22 receives radio signals from a constellation of GNSS satellites. In one embodiment, the GNSS module 22 a GPS module (Global Positioning System), the GPS signals from a constellation of GPS satellites 60 can receive. These signals enable the module 22 Determine the vehicle position that may allow the vehicle to determine if it is in a known location, such as home or work. In addition, the GNSS module can 22 this location data (eg, geographic coordinates) to the wireless communication device 30 then use this data to identify known locations, such as B. the apartment or the workplace of a driver. In addition, the GNSS module 22 used to guide the driver Provide navigation and other location related services. Navigation information can be viewed on the display 58 (or other display within the vehicle) or presented in verbal form, as is the case for example with route description navigation. The navigation services may be operated using an associated vehicle navigation module (the part of the GNSS module 22 or some or all of the navigation services may be provided via a telematics unit installed in the vehicle, wherein the location information is sent to a remote location for the purpose of providing the vehicle with navigation maps, map annotations (sights, restaurants, etc.), route calculations, and the like , The location information may be sent to a remote facility 80 or another remote computer system, such as the computer 78 , for other purposes, such as B. Fleet management and / or for use in a car sharing service. In addition, new or updated map data can be obtained from a remote telematics unit via a vehicle-mounted telematics unit 80 into the GNSS module 22 be downloaded.

The vehicle electronics 20 Also includes a number of vehicle user interfaces that provide vehicle occupants with means for providing and / or receiving information, including pushbutton (s). 52 , Audio system 54 , Microphone 56 and visual display 58 , As used herein, the term "vehicle user interface" broadly encompasses any suitable form of electronic device that includes both on-vehicle hardware and software components and allows a vehicle user to communicate with or through a component of the vehicle. The pushbutton (s) 52 allow manual user input to the communication device 30 to provide additional data, responses or control inputs. The audio system 54 provides audio output to a vehicle occupant and may be an associated stand-alone system or part of the primary vehicle audio system. In accordance with the particular embodiment shown here, the audio system is 54 operational both with the vehicle bus 44 and also coupled to an entertainment bus (not shown) and can provide AM, FM and satellite radio, CD, DVD and other multimedia functionality. This functionality may be provided independently of or in connection with the infotainment module. The microphone 56 provides an audio input for the wireless communication device 30 to allow the driver or other occupant voice commands and / or handsfree over the wireless carrier system 70 which can be processed with speech recognition techniques, as explained in more detail below. The microphone 56 can be connected to an integrated automated language processing unit utilizing the human-machine interface (HMI) known in the art. The optical display or the touch screen 58 Preferably, it is a graphic display, such as a touch screen on the dashboard or a warning light, which is reflected off the windshield and can be used to provide a variety of input and output functions. Various other vehicle user interfaces can also be used because the interfaces of 1 merely serve as an example of a particular implementation.

The Body Control Module (BCM) 26 in the exemplary embodiment 1 as with the communication bus 44 electrically connected, shown. In some embodiments, the BCM 26 integrated with or part of a center stack module (CSM) and / or with a wireless communication device 30 to get integrated. Alternatively, the BCM and the CSM are separate devices that communicate via bus 44 connected to each other. The BCM 26 may include a processor and / or memory associated with the processor 36 and the memory 38 the wireless communication device 30 may be similar, as explained below. The BCM 26 can with the wireless communication device 30 and / or one or more vehicle system modules, such as GNSS 22 , Audio system 54 or other VSMs 42 , communicate. The processor and memory of the BCM 26 may be used to control or execute one or more vehicle operations including, for example, central locking control, air conditioning, exterior mirrors, vehicle ignition control or primary movers (eg, engine, prime mover), and / or the control of various other vehicle modules. The BCM 26 can data from the wireless communication device 30 and then send the data to one or more vehicle modules.

In addition, the BCM 26 Provide information that matches the vehicle condition or specific vehicle components or systems. For example, the BCM of the wireless communication device 30 Provide information indicating whether the vehicle's ignition is on, what gear the vehicle is currently in (ie in gear), and / or other vehicle related information. The BCM 26 may be used to determine one or more vehicle conditions, such as whether the vehicle is on, the battery charge of a vehicle battery, and / or other vehicle conditions. These Different vehicle conditions can be used as a wireless communication device 30 received and used as input for the machine-learning NLG process. For example, the vehicle may have certain mobile devices 90 can detect within or near the vehicle, and based on previous user settings or configurations, the vehicle may determine that the presence of a particular mobile device indicates a high probability that a particular user will be in the vicinity of the vehicle or on the vehicle located. This information can be used to identify the respective user so that an NLG template can be created, selected, and / or used by the vehicle to generate NLG output.

The wireless communication device 30 is able to transmit data over the short-range wireless communication (SRWC) and, in some embodiments, may transmit data over the cellular network communication. As in the exemplary embodiment of 1 shown includes the wireless communication device 30 an SRWC circuit 32 , a cellular chipset 34 , a processor 36 , Storage 38 and the antennas 40 and 50 , In some embodiments, the wireless communication device 30 specifically configured to carry out at least part of the method disclosed herein. In an embodiment, the wireless communication device 30 be a stand-alone module or, in other embodiments, the device 30 as part of one or more other vehicle system modules, such as a Center Stack Module (CSM), BCMs 26 , an infotainment module, a telematics unit, a main unit and / or a gateway module, integrated or be included. In some embodiments, the device may 30 as an OEM-installed (embedded) or as an after-market device installed in the vehicle.

The wireless communication device 30 may be for wireless communication in accordance with one or more wireless protocols including wireless local area network communication (SRWC), such as one of the IEEE 802.11 protocols, Wi-Fi ™, WiMAX, ZigBee ™, Wi-Fi ™ Direct, Bluetooth ™, Bluetooth ™ Low Energy (BLE) or Near Field Communication (NFC). As used herein, Bluetooth ™ refers to any of the Bluetooth ™ technologies, such as Bluetooth Low Energy ™ (BLE), Bluetooth ™ 4.1 , Bluetooth ™ 4.2 , Bluetooth ™ 5.0 and other Bluetooth ™ technologies that can be developed. As used herein, Wi-Fi ™ or Wi-Fi ™ technology refers to any of the Wi-Fi ™ technologies, such as IEEE 802.11b / g / n / ac or any other IEEE 802.11 technology. The short-range wireless communication circuit 32 allows the wireless communication device 30 To send and receive SRWC signals, such as BLE signals. The SRWC circuit can be the connection between the device 30 and another SRWC device. In addition, in some embodiments, the wireless communication device may be a mobile phone chipset 34 which allows the device to communicate over one or more mobile radio protocols, such as the mobile radio bearer system 70 be used.

The wireless communication device 30 it can be the vehicle 12 allow to communicate with one or more remote networks via packet-switched data communication. This packet-switched data communication can be achieved by using a non-vehicle-based wireless access point that is connected to a landline network via a router or modem. When the data transmission device 30 For packet-switched data communication, such as TCP / IP, it can be configured or set up with a static IP address to obtain an assigned IP address from another device in the network, such as a network device. As a router or a network address server to automatically receive.

Packet-switched data communication may also be via the use of a cellular network that is responsible for the device 30 can be accessible. The communication device 30 can have a cellular chipset 34 Data via the wireless carrier system 70 transfer. In such an embodiment, radio transmissions may be used to connect a communication channel, such as a voice channel and / or a data channel, to the wireless carrier system 70 set up so that voice and / or data transmissions can be sent and received over the channel. Data may be sent either over a data link, such as via packet data transmission over a data channel, or over a voice channel using techniques known in the art. For combined services involving both voice and data communications, the system may use a single call over a voice channel and switch between voice and data transmission over the voice channel as needed, again employing techniques known to those skilled in the art. It should be noted that the mobile device 90 may include a cellular chipset and / or other communication means that may be used for packet-switched data communication.

The processor 36 can be any type of device capable of handling electronic commands, including microprocessors, microcontrollers, host processors, controllers, vehicle communication processors, and application specific integrated circuits (ASICs). He can be a special for the data transfer device 30 be provided processor or it can be shared with other vehicle systems. The processor 36 Executes various types of digitally stored commands, such as software or firmware programs stored in memory 38 are stored, which the device 30 to provide a wide variety of services. For example, the processor 36 in at least one embodiment, execute programs or process data to perform at least part of the method described herein, which may include selecting and / or creating NLG templates using a neural network model. The memory 38 may include a volatile RAM memory or other temporarily operated memory, as well as a non-transitory computer-readable medium (e.g., EEPROM) or any other electronic computer medium storing some or all of the software for performing the various external device functions described herein.

In an embodiment, the wireless communication device 30 be operated both in the on state and in the off state. As used herein, an "on-state" is a state of the vehicle in which the ignition or prime mover system of the vehicle is on and, as used herein, a "disabled state" is a state of the vehicle in which the ignition or prime mover system the vehicle is not switched on. The operation or state of the wireless communication device 30 can by another vehicle system module, such as by the BCM 26 or controlled by an infotainment module. When switched on, the wireless communication device 30 always remain "on" or powered by a vehicle battery or other power source. When turned off, the wireless communication device 30 be kept in a low power mode or periodically energized, so that the device 30 can be activated and perform functions.

Now referring to 2 , becomes an illustrative architecture for an ASR system 210 which can be used to implement the method disclosed herein. Although the ASR system 210 hereafter with reference to the wireless communication device 30 of the vehicle 12 is explained, the ASR system 210 into any client device such as those described above, including the mobile device 90 and the computer 78 , An ASR system that conforms to the ASR system 210 may be integrated into one or more remote speech processing servers, including one or more at the remote facility 80 located server. In general, a vehicle occupant interacts via his voice with an Automatic Speech Recognition System (ASR) for one or more of the following reasons: training the system to the specific voice of the person in the vehicle; Storing word-for-word terms, such as a spoken contact, or a spoken control word, such as a reference number or a keyword; or recognizing the language of the person in the vehicle for any suitable purpose, such as voice dialing, menu navigation, transcription, service calls, control of devices and their function in the vehicle or the like. Generally, ASR draws acoustic data from human speech, compares and contrasts the acoustic data with stored data of subwords, makes a selection of a suitable subword that can be concatenated with other selected subwords, and passes the linked subwords for post processing, such as dictation or Transcription, dialing from the address book, storing, training the ASR models, or adjusting the parameters, etc. Additionally, in at least some embodiments, the ASR system may be used to perform at least part of the NLG machine learning process, including selection of an NLG Template and / or the generation of a NLG output based on a selected NLG template.

ASR systems are generally known to those skilled in the art and 2 only illustrates one specific illustrative ASR system 210 , The system 210 includes a device for receiving speech, such as the vehicle microphone 56 , and an acoustic interface 33 , such as a sound card of the wireless communication device 30 with an analog-to-digital converter to digitize the voice into acoustic data. The system 210 Also includes a memory, such as the vehicle memory 38 for storing the acoustic data and for storing speech recognition software and databases, and a processor such as the vehicle processor 36 for processing the acoustic data. The processor operates on the memory and in conjunction with the following modules: one or more front-end processor (s) or preprocessor software modules 212 for decomposing acoustic data of speech output in parametric representations, such as acoustic properties; one or more decoder software module (s) 214 for decoding the acoustic properties in digital output data in the form of (sub) words corresponding to the input utterances; and one or more post-processor software modules 276 to use the output data from the decoder module (s) 214 for all suitable purposes.

The system 210 can also record voice from other suitable audio sources 31 received, either in direct communication with the / the modules of the preprocessor software 212 as seen by the solid line, or indirectly through the acoustic interface 33 , To the audio sources 31 For example, a telephone source such as a voicemail system or any other telephone-based service may be included.

One or more modules or models may act as input to the decoder module (s) 214 be used. First, grammar and / or lexicon models 278 Provide rules that determine which words can logically follow other words to form valid sentences. Roughly speaking, a grammar can define a vocabulary universe that the system 210 expected at any given time in each ASR mode. Is the system located? 210 For example, in training mode to learn commands, then the grammar model (s) can / do 278 Include commands to the system 210 are known and used by this. Is the system located? 210 in another example in main menu mode, then the active grammar model (s) may / may 278 Include all the main menu commands that the system has 210 expected, such as call, dial, leave, delete, directory or the like. Furthermore, by the / the acoustic models 280 a support in choosing the most likely partial word according to the input of the preprocessor module 212 done. Third, word models 222 and sentence / language models 224 Provide rules, syntax, and / or semantics in classifying the selected sub-words or words into the context of the words or sentences. Furthermore, can / can sentence and language models 224 create a universe of sentences that the system 210 at any given time in each ASR mode, and / or provide rules that determine which sentences can logically follow other sentences to form an extended, valid speech output.

According to an alternative illustrative embodiment, some or all of the ASR system may 210 on a computing equipment at one of the vehicle 12 remote location, such as the computer 78 or the remote facility 80 , housed and processed. For example, grammar models, acoustic models and the like may be stored in the memory of one of the servers and / or databases in the remote facility 80 stored and to the wireless vehicle communication device 30 be transmitted to the vehicle's own speech processing. Likewise, the speech recognition software may be using processors of one of the remote servers in the remote device 80 are processed. In other words, the ASR system 210 in the wireless communication device 30 passing through the computer 78 / the remote device 80 and the vehicle 12 distributed in any way, be housed, and / or on the computer 78 or the remote facility 80 be housed.

First, acoustic data is extracted from the human speech that a vehicle occupant uses via the microphone 56 has entered, which converts the sounds into electrical signals and to the acoustic interface 33 passes. A sound-sensitive element in the microphone 56 captures the occupant's statements as variations in air pressure and converts the utterances into corresponding variations of an analog electrical signal, such as DC or voltage. The acoustic interface 33 receives the analog electrical signals that are initially sampled so that the values of the analog signal are detected at certain points of time, then quantized such that the amplitudes of the analog signals are converted into a continuous stream of digital speech signals at each sampling instant , In other words, the acoustic interface 33 converts the analog, electrical signals into digital, electronic signals. The digital data is binary bits stored in memory 38 the wireless communication device 30 buffered and then from the processor 36 the wireless communication device 30 can be processed or processed as soon as they are processed by the processor 36 be received in real time.

Then, the continuous stream of digital speech signals is processed by the preprocessor module (s) 212 converted into discrete sequences of acoustic parameters. More precisely, the processor performs 36 the preprocessor module (s) 212 to segment the digital voice data into overlapping phonetic or acoustic frames of, for example, 10-30 milliseconds (ms) duration. The frames correspond to acoustic subwords, such as syllables, demi-syllables, phones, diphones, phonemes, etc. The preprocessor module (s) 212 also performs phonetic analysis for each frame for extracting acoustic parameters from the speech output of the occupant, such as time-varying feature vectors. Utterances in the language of the occupant can be represented as sequences of these feature vectors. As is well known to those skilled in the art, feature vectors may be extracted and include, for example: pitch, energy profile, spectral Features and / or cepstral coefficients by Fourier transforms the frames and decorrelate the acoustic spectra by means of a cosine transform. Acoustic frames and corresponding parameters that cover a specific speech duration are linked to unknown test patterns for decoding.

As a third step, the processor leaves the decoder module (s) 214 to process the incoming feature vectors of each test pattern. The decoder module (s) 214 are also known as a recognition engine or classifier and use stored speech reference patterns. Like the test patterns, the reference patterns are defined as a concatenation of related acoustic frames and corresponding parameters. The decoder module 214 compares and contrasts the acoustic feature vectors of a subtype test pattern for comparison with stored reference patterns for subtotals, examines the degree of difference or similarity therebetween, and finally applies decision logic for selecting the most likely matching partial word as the recognized partial word. In general, the most consistent subword is that which is the least deviant stored sample, or most likely to coincide, the test pattern is determined by one of several techniques known to those skilled in the art for parsing and recognizing subwords. Such techniques may include: dynamic time-warping classifiers, AI techniques, neural networks, free phoneme recognizers, and / or probabilistic pattern matchers, such as a Hidden Markov Model (HMM) engine.

The HMM is known to those skilled in the art of creating many model hypotheses for speech recognition of an acoustic input. The hypotheses are considered in the final identification and selection of the output, which is the most likely correct decoding of the acoustic input due to the feature analysis of the speech. More specifically, an HMM engine generates statistical models in the form of an "N-best" list of hypotheses for subword models that are graded on the basis of HMM calculated confidence values or probabilities of an observed sequence of acoustic data the Bayes' theorem.

A Bayesian HMM method identifies a best hypothesis corresponding to the most probable utterance, or subword sequence, for a given set of acoustic feature vectors whose confidence values depend on a variety of factors, including the signal to noise ratio of the incoming acoustic signal. The HMM may also include a statistical distribution known as a mixture of diagonal Gaussian values containing a probability value for each detected feature vector of each subword, the values may be used to rearrange the N-best list of hypotheses. The HMM engine can also be used to identify and select a subword whose value is highest in the probabilistic model.

Similarly, individual HMMs can be concatenated for a sequence of subwords to form HMM for single or multiple words. Thereafter, an N-best list of word reference patterns and associated parameter values for single or multiple words may be generated and further evaluated.

In one example, the speech recognition decoder processes 214 the feature vectors using the appropriate acoustic models, grammars, and algorithms to generate an N-best list of reference patterns. As used herein, the term reference pattern is interchangeable with models, waveforms, templates, throughput models, specimens, hypotheses, or other types of references. A reference pattern may include a series of feature vectors representative of one word (or multiple words) or subtotals and based on particular speakers, speech styles, and ambient acoustic conditions. Those skilled in the art will recognize that reference patterns can be created by appropriate reference pattern training of the ASR system and stored in memory. Those skilled in the art will further appreciate that stored patterns may also be altered in which the parameter values of the reference patterns are adjusted due to differences in the signals of speech input in reference pattern training and the current use of the ASR system. For example, a set of reference patterns learned for one vehicle occupant, or certain acoustics, may be adapted for another vehicle occupant, or other acoustic conditions, and stored as a different set of reference patterns based on limited other person's learning data or acoustic conditions , In other words, the patterns are not necessarily rigid, they can be adjusted during speech recognition.

Using vocabulary grammar and all appropriate decoder algorithms and acoustic models, the processor accesses numerous reference patterns in memory that interpret the test pattern. For example, the processor may create a list of N-best vocabulary results or reference patterns along with corresponding parameter values, and save. Illustrative parameter values may include confidence and probability scores, noise ripple spacing values, and / or the like for each reference pattern in the N-best list of the vocabulary and associated segment durations. The N-best list of the vocabulary can be sorted in decreasing size of the parameter values. For example, the vocabulary reference pattern is the best reference pattern with the highest confidence, and so on. Once a set of recognized subwords has been created, they can be used to form words with word model inputs 222 whereas sentences are used with input from the language models 224 ,

Finally, the post processor software module (s) receive 276 the output data from the decoder module (s) 214 for all suitable purposes. In one example, the postprocessor software module (s) may / may 276 Identify one or more reference patterns from the N-best list for reference patterns of single or multiple words as recognized speech. In another example, the post-processor software module (s) may / may 276 used to convert audio data into text or numbers for use with other functions of the ASR system or other vehicle systems. In another example, the postprocessor software module (s) may / may 276 used to provide training feedback for the decoder 214 or preprocessor 212 provide. More specifically, the postprocessor 276 used for acoustic models for the decoder model (s) 214 or adjustment parameters for the preprocessor module (s) 212 to train.

In at least some embodiments, the ASR system may also be integrated with a natural language system (NLG) that may perform the machine learning process NLG described herein. In other embodiments, the machine learning process NLG may be part of a separate module and / or performed with different components or systems. In addition, the NLG process may be performed at different locations and / or using different components. In one embodiment, the vehicle or other client device may be used to perform the entire NLG process, or in another embodiment, a remote server or device may be used to perform the entire NLG process. In other embodiments, a combination of the client device and the remote server or device may be used to perform the NLG process. For example, selecting an NLG template at a remote facility 80 while generating the NLG template and / or generating the NLG output using the NLG template on the vehicle or other client device. There are various other embodiments that will be understood by those skilled in the art in light of the following discussion.

And, as will be apparent from the discussion below, the ASR system and / or the machine learning NLG application may be implemented in a client device, such as a vehicle 12 or mobile device 90 , and / or into a server device, such as one in the remote device 80 located server. In at least some embodiments, the ASR system located on the remote server may include more computing power as well as more speech recognition information that may be used to provide a more powerful ASR system than that residing on the client device; however, as will be appreciated by those skilled in the art, other embodiments exist.

The machine learning NLG process or portions thereof may be implemented in a computer program product (or "application") embodied in a computer readable medium and containing instructions used by one or more processors of one or more computers of one or more systems can. A computer software product may include one or more software programs consisting of program instructions in source code, object code, executable code, or other format; one or more firmware programs; or files of a hardware description language (HDL); and other program related data. The data may include data structures, value tables, or data in another suitable format. The program instructions may include program modules, routines, programs, objects, components, and / or the like. The computer program may be executed by one or more computers in conjunction with each other.

The program (s) may be embodied on computer readable media that may be non-transitory and include one or more storage devices, articles of manufacture, or the like. Examples of computer-readable media include computer system memory such as RAM (Random Access Memory), ROM (Read Only Memory); Semiconductor memory, eg EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory; magnetic or optical disks or tapes; and/ or similar. A computer-readable medium may also include computer-to-computer connections when, for example, transmitting or providing data over a network or other communication link (wired, wireless, or a combination of both). All combinations of the above examples also fall within the scope of computer readable media. It is therefore to be understood that the method may be practiced, at least in part, by electronic articles and / or devices that may execute instructions in accordance with one or more steps of the disclosed method. Additionally, the ASR system or portions thereof may be embodied in a computer program product, similar to the computer program product that may embody the machine learning NLG process.

To get up now 3 come back, will be an embodiment of a method 300 for generating a Natural Language Generation (NLG) output. The procedure 300 can generally be used to receive voice signals, to determine whether to use a static NLG template or a NLG dynamic template using a machine learning NLG template selection process, and NLG output using a selected one To create NLG template. The machine learning selection process for NLG templates may result in the use of a dynamic template being specified, and in this case, an NLG template generation process may be performed to create the NLG dynamic template. After an NLG template has been selected and / or possibly generated, the NLG output can be generated using the selected NLG template, which is the case, at least in some embodiments. With respect to the particular embodiment of the method 300 discusses a NLG process of neural networks, which is a machine-learning NLG process using artificial neural networks to perform a selection process for NLG templates, a NLG template generation process and / or a NLG generation process To carry out expenses. Those skilled in the art will recognize that other machine learning methods can be used to perform the machine learning NLG process, including decision tree learning, learning association rules, deep learning, inductive logic programming, clustering, Bayesian networks, and many others.

The machine-learning NLG process may be on the processor 36 the wireless communication device 30 in vehicle 12 and / or on another VSM of the vehicle 12 be implemented. The machine learning NLG process may be on the wireless communication device 30 (or another VSM) can be implemented by the device 30 is configured with a machine-learning NLG application, which may be in the form of software and / or firmware instructions or part of a computer program product. The machine learning NLG application may initially be in a remote device, such as the remote device 80 , configured and / or compiled and may be on the wireless device 30 (or other client device) may be installed by using a provisioning process or an initial manufacturing process, before, during, or after the sale of the vehicle 12 (or another client device) can be performed. In some embodiments, the NLG template learning learning process may be implemented at a remote device, and a NLG output machine learning generation process may be implemented on a client device; however, this is only an example as those skilled in the art will recognize various other implementations in light of the discussion below.

In at least one embodiment, the NLG machine learning application may first include a set of computer instructions and a set of first NLG information, such as NLG template information, including a plurality of predefined or static NLG templates. The machine-learning NLG template selection process can be implemented on a client device in the form of a computer application and can therefore be referred to as a machine learning selection application for NLG templates. In addition, a dynamic NLG template may be generated for a particular user, and based on received speech signals from that user and / or based on a user profile, the NLG dynamic template may be updated to better reflect an NLG template tailored to the particular user. When a new user is first identified by the client device or the communication system, a predefined template based on the user settings can be selected. In subsequent machine learning NLG process iterations, the NLG template selection process may select a template for the user based on the profile of the user and / or the history of the user, which may be stored during the ASR processes, including the NLG processes , A static NLG template or a dynamic NLG can be selected using the user's history, and when a dynamic NLG template is selected, a dynamic NLG template can be used based on the history of usage, the profile and settings of the user, the received speech signals, and the communication destination. The history of the user may include information about previous machine learning NLG process iterations (including input, output, and performance metrics associated with NLG machine learning iterations), previously received speech signals, and analysis of previously used speech signals, including style, emotion, Dialect and / or character of the language. The profile of the user may include: one or more styles of speech signals previously communicated by the user, one or more styles selected by a user, or other information about the user's language. In one embodiment, the user's profile may include a typical speech style that tends to speak more than others to the user. For example, a first user may speak at length, and so the user profile may include an indication of it.

In addition, various NLG templates can be designed and / or predefined for different types of users, including: a verbose NLG template that uses whole sentences and / or wordier phrases; a direct NLG template that uses phrases and / or a reduced number of words; a formal NLG template that includes predefined statements or phrases using a formal language; an informal NLG template containing predefined statements or phrases using an informal or colloquial language; a clear-cut NLG template that produces NLG audio output that is designed to be very clear so that, for example, a person with damaged or below-average hearing or understanding can interpret the NLG output more likely; and / or a modern NLG template using popular expressions. One of these predefined templates can be selected to use as a starting point for a new user. The user can configure his user profile and / or settings to select a startup template, or the system can automatically select a template that determines that it best suits the user.

The machine learning NLG application can be generated and / or trained with various training information, including information obtained from previous iterations of the machine learning NLG processes. In some embodiments, an NLG application of neural networks may be trained with various statistical algorithms. The machine-learning NLG application may be periodically updated and / or trained to improve the generation of NLG outputs. This may include using inputs and outputs from previous NLG process iterations (as well as performance counters) as training information for the machine learning NLG process. In addition, further training information (eg, input sentences and their associated or desired outputs) as well as updated software / firmware instructions may be provided by a remote server, such as the remote device 80 , sent to the client device. In one embodiment, the machine learning selection process for NLG templates may select static NLG templates until the NLG machine learning process is sufficiently trained so that dynamic templates can be generated that accurately reflect a style or type of response that reflects the NLG templates received speech signals or the profile / history of the user.

The training of the NLG machine learning or NLG learning application may be performed before the machine learning NLG application is installed on the client device and / or before the machine learning NLG application is configured for use on the client device. In addition, the initial training can be monitored using training information (inputs, outputs, and accuracy of expenses) obtained from previous NLG models. In other embodiments, the initial training may be unattended training. The initial training may be at a remote facility, such as the remote facility 80 , and once the initial training is completed and the client device is ready to be configured to use, the machine learning NLG application may be installed. The machine learning NLG application may be included as part or module of a speech recognition application and may be performed and / or integrated similarly to the ASR system discussed above.

In embodiments employing neural networks for the NLG machine learning process, a flat neural network or a deep neural network may be used for the NLG process of neural networks. As used herein, a flat neural network includes a single hidden layer while a deep neural network includes a plurality of hidden layers. Each layer of the neural network may include one or more nodes, each of which may point to one or more other nodes within the same hidden layer, to one or more other nodes within another hidden layer or to one or more output nodes.

For example, shows 4 an exemplary neural network model that can be used in the selection process or in the application for NLG templates of neural networks. The neural network model 100 includes a set of input nodes 102 - 108 , Output node 122 - 124 for neural networks (or machine learning) and a hidden layer covering the nodes 110 - 118 includes. The set of input nodes 102 -108 may each correspond to a different input, including a communication destination ( 102 ), a communication value ( 104 ), a user profile ( 106 ) and a user history ( 108 ). The hidden layer, including the knots 110 - 118 , can be used to associate the inputs with the appropriate output. The selection nodes 122 - 124 The NLG templates of neural networks may correspond to an NLG template, such as a static or predefined NLG template, or a NLG dynamic template. In one embodiment, the expenses 122 and 124 each associated with a probability or a statistical measure that has a posterior probability of using the static NLG template (output 122 ) and a posterior probability of using the NLG dynamic template (Issue 124 ). The template generation selection process can select the template with the highest probability. Various statistical models can be used as part of the machine learning selection process for NLG templates. Although the neural network model 100 With reference to the selection process for NLG templates of neural networks, a neural network model could also be used for the other processes, including the generation process for NLG templates of neural networks and the generation process for NLG outputs.

The procedure 300 starts with step 310 wherein voice signals are received at the client device. As mentioned above, the client device may be one of a variety of devices, including the vehicle 12 , the mobile device 90 and / or the computer 78 ; the procedure 300 However, this is done using vehicle 12 as the client device discussed. The vehicle 12 Can the voice signals through the microphone 56 at the wireless communication device 30 receive. The speech signals may be sampled to obtain a digital representation of the speech signals and from the processor 36 can be used. The scanning can be done on the microphone 56 , at the wireless communication device 30 or on another VSM of the vehicle 12 be performed. Once the voice signals have been received and sampled, the procedure goes 300 to step 320 over.

In step 320 a user or a probable user who has transmitted the voice signals is identified. In one embodiment, the received speech signals are processed with respect to the history of various users, and due to the processing, a particular user can be identified as a transmitter of the speech. Or the system may determine a likelihood that a particular user has transmitted the voice signals, and if the likelihood is above a certain threshold or higher than any other probabilities with respect to other users, then the system may determine that a user is likely to be the voice signals has transferred. These probabilities may be based on a user's history and / or profile, which may also include information about the voice signals previously used by a user. In a particular embodiment, a fingerprint may be generated for the voice of a particular user and used to identify a particular user.

Alternatively or additionally, a state of the client device, such as the vehicle, may be used to determine a user or a likely user who has transmitted the voice signals. In one scenario, the client device may be a smartphone belonging to a single user, and thus the system may be configured to determine that the user or likely users is the single user of the smartphone, and thus transmit the voice signals. In another embodiment, a vehicle may determine which users are on the vehicle by, for example, determining with which key the vehicle was started, which seats of the vehicle are occupied, and / or which personal mobile devices are in or near the vehicle are located. Other different vehicle conditions may be used. Once a particular user has been identified, the procedure goes 300 to step 330 over.

In step 330 a communication destination is determined based on the received speech signals. As used herein, a communication destination is an intent that attempts to be communicated to the ASR system using the received speech signals. The intent may, as discussed above, be through the use of the ASR system 210 be determined. This can be done using various speech recognition techniques and then using post-processing methods for determining an intention to be communicated by the user. These post-processing methods may include using natural language understanding (NLU) processing techniques or a dialog manager that can interpret the received speech signals. Once the communication destination has been determined by the NLU system or the dialogue manager, the communication destination can be stored and assigned to the identified user.

In some embodiments, communication values may be obtained from the received voice. These communication values obtained from the received voice may be referred to as input communication values. As used herein, communication values are values that are specific to received speech or NLG output or response, such as values that are the subject of NLG output (eg, the current temperature, address, and name of a user or memory). Proper names, properties that are unique to a situation or user, and other dynamic information may be considered a communication value. For example, a user may ask, "What is the weather in Royal Oak, Michigan?" The city name "Royal Oak" and the state name "Michigan" may be considered communication values, the communication goal being to provide weather information about the entered communication values (or Royal Oak, Michigan). The communication values may be recognized as unique or dynamic information contained in received speech signals using a natural language understanding system (NLU). In at least some embodiments, these communication values may be used as input to the neural network NLG template selection process, as described below in step 350 explained. As explained in more detail below, communication values can also be obtained for entering in input fields of an NLG template. These communication values, which are obtained for input in input fields of an NLG template, can be referred to as reactive communication values. The procedure 300 then go to step 340 over.

In step 340 It is determined whether an NLG output or a response should be generated. In some scenarios, an ASR system may receive voice signals and perform an operation, but may not need to generate and send NLG output because, for example, the execution of the operation speaks for itself-that is, performing the operation provides clues sufficient to the user convey that the language was interpreted accurately and / or led to the desired operation of the user. Whether or not a response is to be generated can be analyzed by analyzing the step in 330 determined communication goal. However, some communication destinations may require an NLG output or response, or may be supplemented by an NLG output or response, such as those in which the user polls the system via the voice signals, and based on the fact that the user is asking a question with his vote, it may be determined that the results of the query should be communicated or transmitted to the user preferably using speech. Thus, when it is determined that an NLG output or response is to be generated, the procedure proceeds 300 to step 350 otherwise, the procedure ends 300 ,

In step 350 select a Natural Language Generation (NLG) template to use when generating an NLG output. In some embodiments, the NLG template is selected using a neural network NLG template selection process, wherein the neural network NLG template selection process uses an artificial neural network to solve a set of inputs to use an NLG template for use when generating a NLG output. Before the selection process for NLG templates of neural networks is performed, a set of inputs to be used in the selection process for NLG templates of neural networks can be obtained. The set of inputs may include the communication destination, a user history associated with the user, or a user profile associated with the user.

According to one embodiment, the set of inputs includes a communication destination (eg, that in step 330 certain communication destination), communication values, information from the profile of a user and / or information from the history of a user. One or more of these metrics may be available at the time of reaching step 350 determined or earlier determined and stored in memory on the client device, such as the memory 38 the wireless communication device 30 , According to the latter scenario, upon reaching step 350 One or more metrics used in the set of inputs for the machine learning process NLG are retrieved from the storage device. In one embodiment, the user's history may include information identifying prior styles, dialects, or character information related to the user's previous use of language. In addition, based on the received language of a user or other user-configured settings, a style profile associated with the user may be used as input to the neural network template selection process.

Once all the inputs to the NLG Nuclear Template Selection process have been received, the selection process for NLG Nuclear Networks templates can be performed. In some embodiments, the neural network NLG template selection process may be used: (i) determining whether to use a static NLG template or a NLG dynamic template; (ii) if it is determined that a static NLG template is used, then selecting an NLG static template from a plurality of static NLG templates; and (iii) if it is determined that a NLG static template is used, then generating a NLG dynamic template created using the NLG template generation process (step 360 ) can be carried out. Some or all of these steps may be performed on the client device, e.g. B. vehicle 12 , Or, one or all of these steps may be performed on a remote server, such as the remote device 80 contained. In one embodiment, the client device may be located in the remote device 80 NLG template database to query for predefined or static NLG templates that match the style of interaction of a particular user (eg, formal, informal, verbose, direct, and concise). The NLG template database may be queried using the communication destination, the style of the user, the communication values from the received voice, and / or the communication values to be used in the NLG output.

In many embodiments, a template is selected that corresponds to the vocabulary, style, style, and / or character of the user. If a predefined or static NLG template does not sufficiently match these attributes of the user, the template selection process can specify that a dynamic NLG template be created and used. The procedure 300 go to step 360 over, when a dynamic NLG template is selected to be used; otherwise, if a predefined or static NLG template is to be used, the procedure goes 300 to step 370 over.

When in step 360 it is determined that a NLG dynamic template is used to generate the NLG output, then the NLG dynamic template is generated. This may include generating an NLG template, in particular for the communication destination, and using a particular style to be used in the NLG output. In some embodiments, the set of inputs to the neural network NLG template selection process may be used for a neural network NLG template generation process used to generate the NLG dynamic template. The dynamic template can be based on a predefined or static NLG template, which can be changed according to the style or character of the user. For example, if the user is considered chatty and uses a particular type of dialect or idiom, the process may begin with a verbose static NLG template. Then, the detailed static NLG template can be changed with respect to the user's style to create a NLG dynamic template that better matches the user's style. In other embodiments, a rule-based approach rather than a neural network approach may be used to generate the dynamic templates. A rule-based approach may use a predefined set of rules for certain predefined templates, styles or emotions of language, user history, user profiles, communication goals, and / or communication values. These rules can be followed to develop a dynamic NLG template. Once a dynamic NLG template has been created, the procedure goes 300 to step 370 over.

In step 370 An NLG output is generated based on the selected NLG template. The NLG output can be generated by using predefined phrases or phrases from the selected NLG template and then inserting the communication values to be transferred into designated input fields within the predefined phrases and sentences. The communication values may be obtained and may be based on a user's style and / or emotion. The communication values may be obtained based on sending a communication value request to a remote server that can process the request and return communication values. For example, the communication value request may include an indicator of a communication destination and communication values included in the received speech signals. A remote server may then be queried to provide an appropriate response (ie, a communication value response) that satisfies the user's request or intent as communicated in the received voice signals. The communication value response can be received and used with the selected NLG template to generate the NLG output. Once the NLG output is generated, the NLG output can be communicated to the user, e.g. Through speakers or other audio playback device, such as the audio system 54 of the vehicle 12 , In other embodiments, the NLG output may include text displayed on a screen, such as the touch screen 58 of the vehicle 12 ,

As mentioned above, the NLG process of neural networks and / or the application (and sub-processes) may adapt, at least in some embodiments, based on previous iterations. In one embodiment, the neural network NLG template selection process may adapt by maintaining the set of inputs of a particular iteration and the NLG output for the given iteration. Additionally, the client device may determine a measured success of the given iteration based on one or more performance indicators. The performance indicators may include the time to perform speech recognition or the NLG machine-learning process, the accuracy of matching the style or emotion of the NLG output to the style or emotion of the received speech signals, and / or other indications of accuracy and / or Efficiency of the selection and / or production processes of NLG templates. In some embodiments, each one or more of the inputs may be used as a performance indicator. The procedure 300 ends then.

It should be understood that the foregoing is a description of one or more embodiments of the invention. The invention is not limited to the particular embodiment (s) disclosed herein, but is defined solely by the following claims. In addition, the statements made in the foregoing description refer to particular embodiments and are not to be construed as limitations on the scope of the invention or the definition of the terms used in the claims except where a term or phrase has been expressly defined above. Various other embodiments and various changes and modifications to the designated embodiment (s) will be apparent to those skilled in the art. All of these other embodiments, changes, and modifications should be understood to be within the scope of the appended claims.

As used in this specification and claims, the terms "for example", "for example", "e.g. "," "As" and "equals" and the verbs "include," "comprise," "include," and their other verb forms, when used in conjunction with a listing of one or more components or other items, are each open which means that the listing will not be considered as excluding other, additional components or elements. Other terms are to be interpreted in their broadest reasonable sense, unless they are used in a context that requires a different interpretation. In addition, the term "and / or" is understood to include an inclusive OR. As an example, the term "A, B, and / or C" includes the following: "A"; "B"; "C"; "A and B"; "A and C"; "B and C" and "A, B and C".

Claims (10)

A method for generating natural language generation (NLG) output, the method comprising: Receiving speech signals from a user at a microphone of a client device; Determining a requested communication destination and at least one input communication value based on the received speech signals; Determine whether to use a static natural language generation (NLG) template or a dynamic NLG template to produce NLG output, determining whether to use a static NLG template or a dynamic NLG template. using a selection process for NLG templates of neural networks; Selecting an NLG template after determining whether to use a static NLG template or a dynamic NLG template; and Generate an NLG output based on the selected NLG template. Method according to Claim 1 further comprising the step of identifying a user or likely user who has transmitted the voice signals, wherein the selection process for NLG templates of neural networks is based on a history of the identified user or likely user and / or profile of the identified user or likely user is carried out. Method according to Claim 2 wherein the selected NLG template is selected based on the history of the identified user or the probable user and / or the profile of the identified user or the probable user, and wherein the method further comprises the step of identifying a style based on the received speech signals comprising history of the identified user or the probable user and / or the profile of the identified user or probable user, and wherein the identified style in the selection process is used for NLG templates of neural networks. Method according to Claim 1 further comprising the step of determining that an NLG dynamic template is to be used to generate the NLG output and then generating the NLG dynamic template using a NLG template generation process; Method according to Claim 4 wherein the NLG dynamic template is generated based on the communication destination and one or more received communication values, and wherein the method further comprises the step of identifying a user or a likely user who has transmitted the voice signals, wherein the dynamic NLG template is generated based on user information associated with the user or likely user, and wherein the user information includes either or both user history information or user profile information. Method according to Claim 5 where the NLG template generation process is a rule-based generation process for NLG templates or a NLG template generation process for neural networks. Method according to Claim 1 , further comprising the steps of: sending a communication value request to a remote server, the communication value request being a request to obtain reactive communication values to be used in the NLG output based on the communication destination and the entered communication values; and receiving a communication value response from the remote server, the communication value response including reactive communication values to be used in generating the NLG output. Method according to Claim 1 and further comprising the step of adjusting the selection process for NLG templates of neural networks based on previous iterations of the neural network NLG template selection process. Method according to Claim 8 wherein the adjusting step further comprises adjusting the neural network NLG template selection process based on a set of inputs, the NLG output, and a measured success of the neural network NLG template selection process, wherein the set of inputs is the communication destination, the input Communication values and information regarding a user or a probable user who has transmitted the voice signals includes. A method for generating natural language generation (NLG) output, the method comprising: Receiving speech signals from a user at a microphone of a client device; Identifying a user or a likely user who transmitted the voice signals; Determining a communication destination and at least one input communication value based on the received speech signals; Determine whether to use a static NLG template or a NLG dynamic template for use in generating NLG output, determining whether to use a static NLG template or a NLG dynamic template using a selection process for NLG templates of neural networks, the neural network NLG template selection process uses an artificial neural network to solve a set of inputs to select whether to use an NLG static template or an NLG dynamic template for use in generating an NLG output, and wherein the set of inputs includes the communication destination, the input communication values, and either a user history associated with the user or a user profile associated with the user; wherein it is determined whether a static NLG template is used, and then a static NLG template is selected; if it is determined to use a NLG dynamic template, then generating the NLG dynamic template; and Generate the NLG output using the selected NLG template.

Images