KR102342623B1 - Voice and connection platform - Google Patents

Abstract

A system and method for providing a voice assistant comprises, at a first device, receiving a first audio input from a user requesting a first action; performing automatic speech recognition on the first audio input; obtaining the user's context; performing natural language understanding based on speech recognition of the first audio input; and taking a first action based on the user's context and natural language understanding.

Description

VOICE AND CONNECTION PLATFORM

Current voice assistants include Apple's Siri, Google's Google Now and Microsoft's Cortana. A first problem with these current systems is that they do not allow the user to interact with a personal assistant in a conversational manner as the user does with a human. A second problem with these current systems is that the user too often is not understood or misunderstood, or that current systems default to a quick web search. A third problem with these current systems is that they are not proactive in helping their users. A fourth problem is that these current systems have limited applications with which they interact, eg, these voice assistants can only interact with a limited number of applications. A fifth problem is that these current systems do not use the user's context. A sixth problem is that these current systems do not integrate with other voice assistants.

In one embodiment, the voice and connectivity engine provides a voice assistant that addresses one or more of the aforementioned deficiencies of existing voice assistants. In one embodiment, the speech and connectivity engine uses an agnostic and modular approach to one or more of automatic speech recognition, natural language understanding, and text-to-speech components, thereby Frequent updates make it possible, of course, to simplify the adaptation of the system to different languages. In one embodiment, the voice and connectivity engine provides a more natural and human-like conversation with the user and increases the accuracy of understanding the user's requests and reduces the amount of time between receiving and executing the request. Manage context to reduce. In one embodiment, the voice and connectivity engine provides a work around to obtain the user's intended request rather than immediately defaulting to a web search. In one embodiment, the voice and connectivity engine uses modules to interact with various applications of the user device (eg, phone, unified messenger, news, media, weather, browser for web search, etc.) and , modules may be individually added or modified as applications are added and updated over time. In one embodiment, modules for interacting with applications provide some level of standardization in user commands. For example, a user may use the verbal request “send a message” to send a message via Facebook, email or Twitter.

In one embodiment, the method includes: receiving, at a first device, a first audio input from a user requesting a first action; performing automatic speech recognition on the first audio input; obtaining the user's context; performing natural language understanding based on speech recognition of the first audio input; and taking a first action based on the user's context and natural language understanding.

Other aspects include corresponding methods, systems, apparatuses, and computer program products for these and other innovative features. Each of these and other implementations may optionally include one or more of the following features. For example, the operations further include receiving the first audio input in response to the internal event. For example, the operations further include initiating the voice assistant without user input and receiving a first audio input from the user after initiation of the voice assistant. For example, the actions further include wherein the context includes one or more of a context history, a conversation history, a user profile, a user history, a location, and a current context domain. For example, the actions may include, after taking the action, receiving a second audio input from a user requesting a second action not related to the first action; taking a second action; receiving a third audio input from a user requesting a third action related to the first action, the third audio input omitting information used to take the third action; using context to obtain missing information; and taking a third action. For example, the actions further include that the missing information is one or more of an action, an actor, and an entity. For example, the actions may include receiving, at a second device, a second audio input from a user requesting a second action related to the first action, the second audio input information being used to take the second action is missing -; using context to obtain missing information; and taking a second action based on the context. For example, the actions may include determining that the context and the first audio input are missing information used to take the first action; determining which information is missing; and prompting the user to provide a second audio input providing the missing information. For example, the actions may include determining that information used to take the first action cannot be obtained from the first audio input; determining which information is missing; and prompting the user to provide a second audio input that provides information that cannot be obtained from the first audio input. For example, the actions may include determining that information used to take the first action cannot be obtained from the first audio input; determining which information is missing from the information used to take the first action; providing a plurality of options for selection by the user, the options providing potential information for completing the first action; and receiving a second audio input selecting the first option from the plurality of options.

The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art upon consideration of the drawings and description. Moreover, it should be noted that the language used in the specification has been principally chosen for readability and educational purposes and not to limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure is illustrated by way of example and not limitation in the drawings in the accompanying drawings in which like reference numbers are used to indicate like elements.
1 is a block diagram illustrating an exemplary system for a voice and connectivity platform, according to one embodiment.
2 is a block diagram illustrating an exemplary computing device, in accordance with one embodiment.
3 is a block diagram illustrating an example of a client-side voice and connectivity engine, according to one embodiment.
4 is a block diagram illustrating an example of a server-side voice and connectivity engine, according to an embodiment.
5 is a flowchart of an exemplary method of receiving and processing a request using a voice and connectivity platform, in accordance with some embodiments.
6 is a flowchart of an exemplary method of obtaining additional information to determine a user's intended request, in accordance with some embodiments.
7 illustrates an exemplary method of receiving and processing a request using a voice and connectivity platform, according to another embodiment.
8 is a block diagram of an example of managing context in a voice and connectivity platform, according to an embodiment.

1 is a block diagram illustrating an exemplary system 100 for a voice and connectivity platform, according to one embodiment. The illustrated system 100 includes automatic speech recognition (ASR), client devices 106a ... 106n, communicatively coupled via a network 102 for interaction with each other. a server 110 , a voice and connection server 122 , and a text to speech (TTS) server 116 . For example, client devices 106a ... 106n may be coupled to network 102 via signal lines 104a ... 104n, respectively, and to lines 110a ... 110n, respectively. may be accessed by users 112a ... 112n (individually and collectively referred to as user 112 ) as illustrated by The automatic speech recognition server 110 may be coupled to the network 102 via a signal line 108 . Voice and connectivity server 122 may be coupled to network 102 via signal line 120 . Text-to-speech server 116 may be coupled to network 102 via signal line 114 . The use of the nomenclature “a” and “n” in the reference numbers indicates that any number of the elements having the nomenclature may be included in the system 100 .

Network 102 may include any number of networks and/or network types. For example, network 102 may include one or more local area networks (LANs), wide area networks (WANs) (eg, the Internet), virtual private networks (VPNs), mobile networks (eg, cellular networks), Wireless wide area networks (WWANs), Wi-Fi networks, WiMAX® networks, Bluetooth® communication networks, peer-to-peer networks, other interconnected data paths over which multiple devices can communicate in -, various combinations thereof, and the like. Data transmitted by network 102 may include packetized data (eg, Internet Protocol (IP) data packets) that are routed to designated computing devices coupled to network 102 . In some implementations, network 102 may include a combination of wired and wireless (eg, terrestrial or satellite-based transceivers) networking software and/or hardware that interconnects the computing devices of system 100 . For example, network 102 may include packet-switching devices that route data packets to various computing devices based on information included in headers of the data packets.

Data exchanged over the network 102 may include hypertext markup language (HTML), extensible markup language (XML), JavaScript Object Notation (JSON), Comma Separated Values (CSV), Java DataBase Connectivity (JDBC), Open DataBase Connectivity (ODBC). ) and/or the like. In addition, all or some of the links may use conventional encryption techniques, such as secure sockets layer (SSL), Secure HTTP (HTTPS) and/or virtual private networks (VPNs) or Internet Protocol security (IPsec). can be encrypted using In another embodiment, entities may use custom and/or proprietary data communication technologies in place of or in addition to those described above. Depending on the embodiment, network 102 may also include links to other networks. Additionally, data exchanged over the network 102 may be compressed.

Client devices 106a ... 106n (referred to individually and collectively as client device 106) are computing devices having data processing and communication capabilities. Although FIG. 1 illustrates two client devices 106 , this disclosure applies to any system architecture having one or more client devices 106 . In some embodiments, the client device 106 includes a processor (eg, virtual, physical, etc.), memory, power source, network interface, and/or display, graphics processor, wireless transceivers, keyboard, speakers, camera, sensor files, firmware, operating systems, drivers, and other software and/or hardware components, such as various physical connection interfaces (eg, USB, HDMI, etc.). Client devices 106a ... 106n may couple to and communicate with each other and other entities of system 100 via network 102 using wireless and/or wired connections.

Examples of client devices 106 include automobiles, robots, cell phones (eg, feature phones, smart phones, etc.), tablets, laptops, desktops, netbooks, server appliances, servers, virtual machines, TVs, set top boxes, may include, but are not limited to, a media streaming device, a portable media player, a navigation device, a personal digital assistant (PDA), and the like. Although two or more client devices 106 are shown in FIG. 1 , system 100 may include any number of client devices 106 . Additionally, the client devices 106a ... 106n may be the same or different types of computing devices. For example, in one embodiment, the client device 106a is a car and the client device 106n is a mobile phone.

In the illustrated implementation, the client device 106a includes instances of a client-side voice and connection engine 109a, an automatic speech recognition engine 111a, and a text-to-speech engine 119a. Although not shown, the client device 106n may include its own instances of a client-side voice and connection engine 109n, an automatic speech recognition engine 111n, and a text-to-speech engine 119n. In one embodiment, instances of the client-side voice and connection engine 109 , the automatic speech recognition engine 111 , and the text-to-speech engine 119 are storable in the memory of the client device 106 and the client device 106 . executable by the processor of

Text-to-speech (TTS) server 116 , automatic speech recognition (ASR) server 110 , and voice and connectivity server 122 may include one or more computing devices having data processing, storage, and communication capabilities. . For example, these entities 110 , 116 , 122 may include one or more hardware servers, server arrays, storage devices, systems, etc., and/or may be centralized or distributed/cloud based. have. In some implementations, these entities 110 , 116 , 122 operate in a host server environment and implement, for example, processor, memory, storage, network interfaces, etc., through an abstraction layer (eg, virtual machine manager). It may include one or more virtual servers, including access to the physical hardware of the host server.

The automatic speech recognition (ASR) engine 111 performs automatic speech recognition. For example, in one embodiment, the ASR engine 111 receives audio (eg, voice) input and converts the audio to a text string. Examples of ASR engines 111 include, but are not limited to, Nuance, Google Voice, Telisma/OnMobile, and the like.

According to an embodiment, the ASR engine 111 may be on-board, off-board, or a combination thereof. For example, in one embodiment, ASR engine 111 is onboard and ASR is performed by ASR engine 111a and ASR engine 111x on client device 106, and ASR server 110 may be omitted. have. In another example, in some embodiments, the ASR engine 111 is offboard (eg, streaming or relay) and ASR is performed by the ASR engine 111x on the ASR server 110 , and the ASR engine 111a is omitted. can be In another example, ASR is performed both by the ASR engine 111a at the client device 106 and by the ASR engine 111x at the ASR server 110 .

A text-to-speech (TTS) engine 119 performs text-to-speech conversion. For example, in one embodiment, the TTS engine 119 may provide text or other non-voice input (eg, additional information as discussed below with reference to the work around engine 328 of FIG. 3 ). request) and output a human recognizable voice presented to the user 112 via the audio output of the client device 106 . Examples of ASR engines 111 include, but are not limited to, Nuance, Google Voice, Telisma/OnMobile, Creawave, Acapella, and the like.

Depending on the embodiment, the TTS engine 119 may be on-board, off-board, or a combination thereof. For example, in one embodiment, TTS engine 119 is onboard and TTS is performed by TTS engine 119a and TTS engine 119x on client device 106, and TTS server 116 may be omitted. have. In another example, in some embodiments, TTS engine 119 is offboard (eg, streaming or relay) and TTS is performed by TTS engine 119x on TTS server 116 , and TTS engine 119a is omitted. can be In another example, TTS is performed by the TTS engine 116a at the client device 106 and also by the TTS engine 116x at the TTS server 116 .

In the illustrated embodiment, the voice and connectivity engine is split into two components 109 , 124 ; One is on the client side and one is on the server side. Depending on the embodiment, the voice and connectivity engine may be onboard, offboard, or a hybrid of the two. In another example, in one embodiment, the voice and connectivity engine is onboard and the features and functionality discussed below with respect to FIGS. 3 and 4 are performed on the client device 106 . In another example, in one embodiment, the voice and connectivity engine is offboard, and the features and functionality discussed below with respect to FIGS. 3 and 4 are performed on the voice and connectivity server 122 . In another example, in one embodiment, the voice and connectivity engine is a hybrid, and the features and functionality discussed below with respect to FIGS. 3 and 4 include the client-side voice and connectivity engine 109 and the server-side voice and connectivity. It is divided between engines 124 . It will be appreciated, however, that features and functionality may be divided in different ways than the illustrated embodiments of FIGS. 3 and 4 . In one embodiment, the voice and connection engine provides a voice assistant that uses context and artificial intelligence and provides a natural conversation with the user 112 , and disadvantages in user requests (eg, failure to recognize voice). can be avoided.

In one embodiment, a client-side (onboard) voice and connectivity engine 109 manages the conversation and connects to a server-side (offboard) voice and connectivity platform 124 for extended semantic processing. Such an embodiment may advantageously provide synchronization to enable loss and recovery of the connection between the two. For example, assume a user is passing through a tunnel and does not have a network 102 connection. In one embodiment, system 100 detects a lack of network 102 connectivity and uses a "lite" local version of automatic speech recognition engine 111 and natural language understanding engine 326 to execute, ASR and Natural Language Understanding (NLU) have greater semantics when parsing input (ie, queries/requests) locally on the client device 106 , but when network 102 connectivity is available. , vocabularies, and processing power in server-side versions of those engines. In one embodiment, if the user's request requires a network 102 connection, the system may verbally notify the user that the system does not have a network 102 connection, and when the network 102 connection is re-established. Your request will be processed.

It will be appreciated that the system 100 illustrated in FIG. 1 represents an example system for voice and connectivity in accordance with one embodiment, and that a variety of different system environments and configurations are contemplated and are within the scope of the present disclosure. For example, various functionality may be moved from server to client and vice versa, and some implementations may include additional or fewer computing devices, servers, and/or networks, and various functionality may be moved client-side or It can be implemented on the server side. Moreover, the various entities of system 100 may be integrated into a single computing device or system, partitioned among additional computing devices or systems, or the like.

2 is a block diagram of an exemplary computing device 200 , according to one embodiment. Computing device 200 includes a processor 202 , memory 204 , communication unit 208 , and storage device 241 , which may be communicatively coupled by a communication bus 206 , as illustrated. may include The computing device 200 shown in FIG. 2 is provided by way of example, and it should be understood that the computing device 200 may take other forms and include additional or fewer components without departing from the scope of the present disclosure. you will know For example, although not shown, computing device 200 may include input and output devices (eg, display, keyboard, mouse, touch screen, speaker, etc.), various operating systems, sensors, additional processors, and It may include other physical configurations. In addition, the computer architecture shown in FIG. 2 and described herein includes a number of entities within the system 100 having various modifications - eg, a TTS server 116 (eg, a TTS engine 119 ); By omitting the other illustrated engines), the ASR server 110 (eg, by including the ASR engine 111 and omitting the other illustrated engines), the client device 106 (eg, server-side voice and by omitting the connection engine 124) and the voice and connection server 122 (eg, by including the server-side voice and connection engine 124 and omitting the other illustrated engines). you will know that you can

The processor 202 may be an arithmetic logic unit, microprocessor, general purpose controller, FPGA (field) that executes software instructions by performing various input, logical, and/or mathematical operations to provide the features and functionality described herein. programmable gate array), application specific integrated circuit (ASIC), or any other processor array, or any combination thereof. Processor 202 may execute code, routines, and software instructions by performing various input/output, logical, and/or mathematical operations. The processor 202 may be configured for a variety of computer architectures that process data signals, including, for example, a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, and/or an architecture implementing a combination of instruction sets. have them The processor 202 may be physical and/or virtual and may include a single core or multiple processing units and/or cores. In some implementations, the processor 202 generates and provides electronic display signals to a display device (not shown), assists in displaying images, captures and transmits images, extracts various types of features and performing complex tasks including sampling. In some implementations, the processor 202 may be coupled to the memory 204 via a bus 206 to access data and instructions from and store data in the memory 204 . Bus 206 may couple processor 202 to other components of application server 122 , including, for example, memory 204 , communication unit 208 , and storage device 241 .

Memory 204 may store data and provide access to the data to other components of computing device 200 . In some implementations, memory 204 may store instructions and/or data that may be executed by processor 202 . For example, as shown, memory 204 may store one or more engines 109 , 111 , 119 , 124 . Memory 204 may also store other instructions and data, including, for example, an operating system, hardware drivers, software applications, databases, and the like. Memory 204 may be coupled to bus 206 to communicate with processor 202 and other components of computing device 200 .

Memory 204 may contain, store, transfer, propagate, or transmit instructions, data, computer programs, software, code, routines, etc. for processing by or in connection with processor 202 . includes non-transitory computer-usable (eg, readable, writable, other) media, which can be any apparatus or device that exists. In some implementations, memory 204 may include one or more of volatile memory and non-volatile memory. For example, memory 204 may include a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a discrete memory device (eg, PROM, FPROM, ROM), a hard disk drive, an optical disk drive (CD, DVD, Blu-ray ^TM, etc.), but is not limited thereto. It will be appreciated that memory 204 may be a single device or may include multiple types of devices and configurations.

The bus 206 is a communication bus for transferring data between components of the computing device or between the computing devices 106/110/116/122, a network bus system including the network 102 or portions thereof, a processor a processor mesh, combinations thereof, and the like. In some implementations, engines 109 , 111 , 119 , 124 , their subcomponents, and various software (eg, operating system, device drivers, etc.) running on computing device 200 cooperate, and bus 206 ) through a software communication mechanism implemented in connection with A software communication mechanism may be, for example, interprocess communication, local function or procedure calls, remote procedure calls, an object broker (eg CORBA), direct socket communication between software modules (eg, TCP/IP socket), UDP broadcasts and receptions, HTTP connections, and/or the like. Moreover, any or all of the communication may be secure (eg, SSL, HTTPS, etc.).

The communication unit 208 may include one or more interface devices (I/F) for wired and/or wireless connection with the network 102 . For example, the communication unit 208 may include CAT-type interfaces; radio transceivers (4G, 3G, 2G, etc.) for communication with the mobile network 103, and for Wi-Fi ^TM and close-proximity (eg, Bluetooth®, NFC, etc.) connections, etc. wireless transceivers for transmitting and receiving signals using the wireless transceivers; USB interfaces; various combinations thereof; Others may include, but are not limited to. In some implementations, the communication unit 208 may link the processor 202 to a network 102 , which in turn may be coupled to other processing systems. The communication unit 208 provides other connections to the network 102 and to other entities of the system 100 using, for example, a variety of standard network communication protocols, including those discussed elsewhere herein. can do.

Storage device 241 is an information source that stores data and provides access to it. In some implementations, storage device 241 may be coupled to components 202 , 204 , and 208 of a computing device via bus 206 to receive data and provide access to it. The data stored by the storage device 241 may vary based on the computing device 200 and the embodiment. For example, in one embodiment, the storage device 241 of the client device 106 may store information regarding the user's current context and session, and the storage device 241 of the voice and connectivity server 122 is and long-term contexts, aggregated user data used for machine learning, and the like.

The storage device 241 may be included in the computing device 200 and/or a storage system separate from but coupled to or accessible by the computing device 200 . Storage device 241 may include one or more non-transitory computer-readable media for storing data. In some implementations, storage device 241 may be included in or separate from memory 204 . In some implementations, storage device 241 may include a database management system (DBMS) operable on application server 122 . For example, the DBMS may include a structured query language (SQL) DBMS, NoSQL DMBS, various combinations thereof, and the like. In some cases, the DBMS stores data in multidimensional tables of rows and columns, and can use programmatic operations to manipulate rows of data: insert, query, update, and/or delete.

As noted above, computing device 200 may include other and/or fewer components. Examples of other components may include a display, input device, sensor, etc. (not shown). In one embodiment, the computing device includes a display. The display may include, for example, any conventional display device, monitor, or screen, including organic light-emitting diode (OLED) displays, liquid crystal displays (LCDs), and the like. In some implementations, the display may be a touch screen display capable of receiving input from a stylus, one or more fingers of user 112 , or the like. For example, the display may be a capacitive touch screen display capable of detecting and interpreting multiple points of contact with the display surface.

An input device (not shown) may include any device for inputting information into the application server 122 . In some implementations, the input device can include one or more peripheral devices. For example, an input device may include a keyboard (eg, a QWERTY keyboard or a keyboard in any other language), a pointing device (eg, a mouse or touchpad), a microphone, an image/video capture device (eg, a camera), etc. have. In one embodiment, computing device 200 can be representative of client device 106 , which includes a microphone for receiving voice input and speakers for facilitating text-to-speech (TTS). include In some implementations, the input device can include a touch screen display that can receive input from one or more fingers of the user 112 . For example, user 112 may interact with an emulated (ie, virtual or soft) keyboard displayed on a touch screen display by using fingers to contact the display in keyboard areas.

Exemplary client-side voice and connection engine 109

Referring now to FIG. 3 , illustrated is a block diagram of an exemplary client-side voice and connectivity engine 109 , in accordance with one embodiment. In the illustrated embodiment, the client-side speech and connection engine 109 includes an automatic speech recognition (ASR) engine 322 , a client-side context holder 324 , and a natural language understanding (NLU) engine 326 . ), a suboptimal engine 328 and a connection engine 330 .

Automatic speech recognition (ASR) interaction engine 322 includes code and routines for interacting with automatic speech recognition (ASR) engine 111 . In one embodiment, the ASR interaction engine 322 is a set of instructions executable by the processor 202 . In another embodiment, the ASR interaction engine 322 is stored in the memory 204 , and is accessible and executable by the processor 202 . In either embodiment, the ASR interaction engine 322 is adapted for cooperation and communication with the processor 202 , the ASR engine 111 , and other components of the system 100 .

ASR interaction engine 322 interacts with ASR engine 111 . In one embodiment, the ASR engine 111 is local to the client device 106 . For example, ASR interaction engine 322 interacts with ASR engine 111 which is an onboard ASR application, such as ASR engine 111a. In one embodiment, the ASR engine 111 is remote from the client device 106 . For example, ASR interaction engine 322 interacts with ASR engine 111 , which is an offboard ASR application accessible and used over network 102 , such as ASR engine 111x . In one embodiment, the ASR engine 111 is a hybrid comprising both components local to the client device 106 and components remote therefrom. For example, the ASR interaction engine 322, when the client device 106 has a network 102 connection, reduces the processing burden on the client device 106 and improves its battery life, the offboard ASR It interacts with engine 111x and when network 102 connectivity is unavailable or insufficient, with onboard ASR engine 111a.

In one embodiment, the ASR interaction engine 322 interacts with the ASR engine 111 by initiating speech input of the ASR engine 111 . In one embodiment, the ASR interaction engine 322 may initiate speech input of the ASR engine 111 in response to detecting one or more events. In some embodiments, the ASR interaction engine 322 proactively initiates the ASR without waiting for the user 112 to initiate a conversation. Examples of events include, but are not limited to, a wake-up word or phrase, expiration of a timer, user input, internal event, external event, and the like.

In one embodiment, the ASR interaction engine 322 may initiate speech input of the ASR engine 111 in response to detecting the wakeup word or phrase. For example, suppose a voice and connectivity platform is associated with a persona for interacting with users, and the persona is named "Sam". In one embodiment, the ASR interaction engine 322 detects when the word “Sam” is received via the microphone of the client device and initiates voice input to the ASR engine 111 . In another example, assume that the phrase “Hey!” is assigned as the wake-up phrase; In one embodiment, the ASR interaction engine 322 detects when the phrase “Hey!” is received via the microphone of the client device and initiates voice input to the ASR engine 111 .

In one embodiment, the ASR interaction engine 322 initiates voice input of the ASR engine 111 in response to detecting expiration of the timer. For example, system 100 may indicate that a user wakes up at 7 am and leaves work at 6 pm; In one embodiment, it may set a timer for 7 am and a timer for 6 pm, and determine that the ASR interaction engine 322 initiates voice input to the ASR engine 111 at those times. For example, a user can thus request news or weather when waking up at 7 am, and request a traffic report or ask a spouse to initiate a call when leaving work at 6 pm.

In one embodiment, the ASR interaction engine 322 initiates speech input of the ASR engine 111 in response to detecting the user input. For example, the ASR interaction engine 322 may include a gesture (eg, a specific swipe or movement on a touch screen) or a button (physical or soft/virtual) selection (eg, selecting a dedicated button or long-pressing a multipurpose button). In response to detecting the pressing), the voice input of the ASR engine 111 is started. It will be appreciated that the mentioned button may be on the client device 106 or a component associated with the client device 106 (eg, a dock, cradle, Bluetooth headset, smart watch, etc.).

In one embodiment, the ASR interaction engine 322 initiates voice input of the ASR engine 111 in response to detecting an internal event. In one embodiment, the internal event is based on a sensor (eg, GPS, accelerometer, power sensor, docking sensor, Bluetooth antenna, etc.) of the client device 106 . For example, the ASR interaction engine 322 may detect that the user device 106 is located in the user's vehicle (eg, on-board diagnostics of the vehicle, power and connection to a cradle/dock in the vehicle, etc.) in response to the detection) initiates voice input of the ASR and initiates voice input of the ASR engine 111 (eg, to receive navigation directions or a user's request for music to be played). In one embodiment, the internal event is based on an application (not shown) of the client device 106 . For example, the client device 106 is a smartphone with a calendar application and the calendar application includes appointments for a user at a remote location; In one embodiment, assume that the ASR initiates voice input of the ASR engine in response to detecting an appointment (eg, to receive a user's request for directions to an appointment). In one embodiment, the internal event is based on the operation of the local text-to-speech engine 119a. For example, text-to-speech engine 119 is operative to present a context prompt (eg, "I think I'm leaving work. Shall I call your wife and direct you home?"), or other prompt to the user; In one embodiment, assume that the ASR interaction engine 322 detects a text-to-speech prompt and initiates speech input of the ASR engine 111 to receive the user's response to the prompt.

In one embodiment, the ASR interaction engine 322 initiates voice input of the ASR engine 111 in response to detecting an external event (eg, from a third-party API or database). In one embodiment, the internal event is based on the operation of the remote text-to-speech engine 119x. For example, text-to-speech engine 119 may trigger a contextual prompt (eg, "I think I'm leaving work, would you like to call my wife and direct me home?" or "You are approaching your destination. Get directions to an available parking space?" Do you want a guide?"), or other prompts; In one embodiment, assume that the ASR interaction engine 322 detects a text-to-speech prompt and initiates speech input of the ASR engine 111 to receive the user's response to the prompt.

In one embodiment, the ASR interaction engine 322 is agnostic. For example, in one embodiment, the ASR interaction engine 322 may use one or more different ASR engines 111 . Examples of ASR engines 111 include, but are not limited to, Nuance, Google Voice, Telisma/OnMobile, Creawave, Acapella, and the like. The agnostic ASR interaction engine 322 may advantageously enable flexibility in the language of the ASR engine 111 and ASR engine 111 used, as new ASR engines 111 become available and When existing ASR engines are shut down, it may allow the ASR engine(s) 111 used to change over the lifecycle of the voice and connectivity system 100 . In some embodiments, system 100 includes multiple ASR engines, and the ASR engine 111 used is context-dependent. For example, Google Voice provides better recognition of proper nouns than Nuance; In one embodiment, assume that the ASR interaction engine 322 is able to interact with Google Voice ASR when it is determined that the user has accessed the contact list of the phone application. In some embodiments, system 100 may switch between ASR engines at any time (eg, processing a first portion of speech input with a first ASR engine 111 and a second portion of speech input with a second Treated with ASR(111)). Similar to the ASR engine 111 , in one embodiment, the system 100 is agnostic with respect to the TTS engine 119 being used. Also similar to the ASR engine 111 , in some embodiments, the system 100 may include multiple TTS engines 119 and may select different TTS engines for different contexts and/or when You can even switch between different TTS engines. For example, in one embodiment, the system 100 may start reading headlines in English, the user may request French, and the system will switch to an English-French TTS engine.

The ASR engine 111 receives the voice input after the ASR interaction engine 322 initiates the voice input. In one embodiment, in response to initiation, the ASR engine 111 receives the voice input, without further intervention of the ASR interaction engine 322 . In one embodiment, after initiating the voice input, the ASR interaction engine 322 passes the voice input to the ASR engine 111 . For example, the ASR interaction engine 322 is communicatively coupled to the ASR engine 111 to transmit voice input to the ASR engine 111 . In another embodiment, after initiating the voice input, the ASR interaction engine 322 stores the voice input on a storage device (or any other non-transitory storage medium accessible by communication), and the voice input is stored on the storage device ( or other non-transitory storage media) and retrieved by the ASR engine 111 .

In some embodiments, system 100 proactively provides an electronic voice assistant without receiving user input, such as voice input. For example, in one embodiment, system 100 may determine that a car (ie, client device 106 ) is in a traffic jam and automatically initiates a TTS to initiate a conversation with the user (eg, “alternative would you like to provide a route to?") action (eg, determine an alternative route, such as parking space and taking a train, and update the navigation route accordingly).

The client-side context holder 324 contains code and routines for context synchronization. In one embodiment, context synchronization includes managing the definition, use, and storage of context workflows from the client side and sharing the context workflows with the server side. In one embodiment, the client-side context holder 324 is a set of instructions executable by the processor 202 . In another embodiment, the client-side context holder 324 is stored in the memory 204 , and is accessible and executable by the processor 202 . In either embodiment, the client-side context holder 324 is adapted for collaboration and communication with the processor 202 , other components of the client device 106 , and other components of the system 100 .

The client-side context holder 324 manages the definition, use, and storage of context workflows from the client side and shares the context workflows with the server side. In one embodiment, the client-side context holder 324 is a context agent using a context synchronization protocol to synchronize the context within the system 100, notwithstanding the low capacity and itinerancy on the network 102. 420 (server-side context holder), which may be particularly beneficial for some networks, eg, mobile data networks.

The client-side context holder 324 manages the definition, use, and storage of contexts. Context is the current state of the personal assistant provided by the voice and connectivity engine. In one embodiment, the context includes one or more parameters. Examples of parameters include context history, conversation history (eg, the user's previous requests and previous responses and actions of the system), user profile (eg, the user's identity and preferences), user history (eg, the user's habits), location (physical location of client device 106 ), current context domain (eg, client device 106 , application(s) being used, interface currently presented to the user). not limited In some embodiments, a parameter may be a variable or a serialized object.

In one embodiment, the context is a multidimensional context and may describe any dimensional variable or characteristic. In some embodiments, the context uses a multidimensional matrix. As described herein, in some embodiments, context is synchronized in real time between a client side (eg, client device 106a ) and a server side (eg, voice and connectivity server 122 ). Because of the tight integration of both parts of the platform (client and server) and the combination of which contexts can describe variables or characteristics with arbitrary dimensions, a context may sometimes be referred to as a "Deep Context".

Depending on the embodiment, the context may affect the ability of the system 100 to accurately recognize words from speech, determine the user's intended request, and facilitate a more natural conversation between the user 112 and the system 100 . used by system 100 to provide one or more advantages, including, but not limited to, increasing.

In one embodiment, context is used to more accurately recognize words from speech. For example, the user has the phone application open; In one embodiment, the context is used to limit the dictionary used by the natural language understanding engine 326 (eg, to names of contacts and words associated with manipulating a phone call or making a call) (eg, during preprocessing). by the NLU engine 326). In one embodiment, this pre-restriction advantageously removes the car company "Renault" but the name "Renaud" so that the NLU engine 326 can accurately determine that the user wants to call Renaud and not Renault. can be left The NLU engine 326 may even determine which Renaud the user intends to call (assuming multiple contacts named Renaud) based on previous phone calls made by the user. Thus, the previous example also describes one embodiment in which context is used to more accurately determine the intended request of the user. As such, the context may also minimize the amount of time from receiving a user's request to executing correctly upon request.

In one embodiment, context is used to facilitate a more natural conversation (two-way communication) between the user and system 100 . For example, a user requests news about Yahoo!; Context can be used to facilitate a conversation when the system starts reading headlines of articles about Yahoo! The user asks "Who is the CEO?"; The system 100 understands that the user's intended request is for the CEO of Yahoo! and searches for that name and serves it. The user then asks about today's weather; The system 100 understands that the request is associated with a weather application and that the user's intended request is for weather for the user's physical location, determines that the weather application should be used, and determines that the weather application should be used to obtain the weather. Make an API call to The user then says "and tomorrow"; The system 100 understands that the user's intended request is for tomorrow's weather at the user's current location. The user continues, "How about a stock trade?" ask questions; The system 100 responds to the user's intended request from Yahoo! Understand that it is about the current trading price of a stock and perform a web search to obtain that information. To summarize and simplify, in some embodiments, to facilitate a more “natural” conversation between user 112 and system 100 by supporting such context jumping, context tracks a topic and , switch between applications, and track status in the workflows of various applications.

In some embodiments, for example, to learn a probability of a next step or command based on aggregated data from numerous users and how users generally interact with system 100 , or data of those users. and for a particular user based on how that user interacts with the system 100 , machine learning is applied to contexts.

In one embodiment, the client-side context holder 324 synchronizes the user's current context with the context agent 420 of FIG. Synchronizing the context with the server-side voice and connection engine 124 allows the client-side voice and connection engine 109 to optionally cause the server-side engine 124 to manage the conversation and perform various operations, or, for example, a server to perform functions at the client device 106 based on the connection to 122 .

In one embodiment, the client-side holder 324 and the context agent 420 (ie, the server-side holder) communicate using a context synchronization protocol that provides the communication protocol, as well as ensure that the context information being synchronized is communicated. verify In one embodiment, the context synchronization protocol standardizes key access (eg, context ID) for each attribute (eg, variable or parameter) of a state or substate of the current context.

Referring now to FIG. 8 , a schematic diagram 800 is shown that provides further details regarding the synchronization of context between the client side and the server side, according to one embodiment. In the illustrated embodiment, the client-side context holder 324 of the client device maintains one or more contexts 810a/812a/814a of the client device 106 . In one embodiment, each context 810a/812a/814a is associated with a module. In one embodiment, the client-side context holder 324 maintains a context comprising screens (Screen 1 through Screen N) that include the user's flow through the functions of the application and the functions available on each screen. For example, in the illustrated embodiment, the user is presented with screen 1 820a, which provides a set of functions, and the user has selected a function (from F1 through Fn of screen 1). The user was then presented with screen 2, where the user selected a function (from F1 to Fn of screen 2). The user was then presented with screen 3, where the user selected a function (from F1 to Fn of screen 3), and so on. For example, in one embodiment, module 1 810a is a module for a phone application and module 2 812a is a module for a media application; In one embodiment, the screens 820a, 822a, 824a, and 826a of module 1 810a facilitate a conversation between the user and the system to select a contact and search for a workaround (discussed below) to make a call. , and assume that the screens of module 2 812a may represent a flow in which the user searches for genres, artists, albums and tracks to be played.

The home screen 830a resets the contexts of the various modules 810a, 812a, and 814a. For example, module 1 810 is associated with a news application; In one embodiment, assume that the user is sent to the home screen 830a (eg, automatically by a mechanism such as a timeout period or based on the user's request). In one embodiment, when the user is sent to home screen 830a, a reset of context information in one or more of modules 810a, 812a, 814a is triggered.

In one embodiment, the context synchronization protocol 804, also described below with reference to FIG. 4, transfers contexts from a client-side context holder 324 to a context agent 422, also referred to as a server-side context holder or the like. Provides a protocol for delivery. In some embodiments, the context synchronization protocol provides a high degree of compression. In some embodiments, the context synchronization protocol allows contexts to successfully synchronize between the client side and the server side, such that the information 806 of the context agent 422 is the same as the information 802 of the client side context holder 324 . It provides a mechanism to verify that

In one embodiment, the context engine 424 collects contexts from the context agent 422 . In one embodiment, the context engine 424 manages context information 808 for the user. For example, the context agent 424 maintains context information for an application over time (eg, long-term and medium-term contexts) and various context information for each user session in the application. Such information may be useful for machine learning (eg, predicting a user's intent based on the current context, such as a request to call Victoria, and past contexts, such as the last request to Victoria was to Victoria P).

In one embodiment, the client-side context holder 324 passes the context to one or more components of the system 100 , including, for example, a natural language understanding (NLU) engine 326 and/or a context agent 422 . . In one embodiment, the client-side context holder 324 stores the context in the storage device 241 (or any other non-transitory storage medium accessible by communication). Other components of system 100 , including, for example, natural language understanding engine 326 and/or context agent 422 , may retrieve context by accessing storage device 241 (or non-transitory storage medium).

A natural language understanding (NLU) engine 326 includes code and routines for receiving the output of the ASR engine 111 and determining, based on the output of the ASR engine 111 , an intended request of the user. In one embodiment, the NLU engine 326 is a set of instructions executable by the processor 202 . In another embodiment, the NLU engine 326 is stored in the memory 204 , and is accessible and executable by the processor 202 . In either embodiment, the NLU engine 326 is adapted for cooperation and communication with the processor 202 , the ASR engine 111 , and other components of the system 100 .

In one embodiment, the NLU engine 326 preprocesses the ASR engine 111 output to correct errors in speech recognition. For clarity and convenience, the output of the ASR engine 111 is sometimes referred to as "recognized speech". In one embodiment, the NLU engine 326 preprocesses the recognized speech to correct any errors in the recognized speech. In one embodiment, the NLU engine 326 receives the recognized voice from the ASR engine 111 and, optionally, associated credits, and the context from the client-side context holder 324 and recognizes any misrepresentation in the recognized voice. Correct the terms used. For example, if the user speaks French and the voice input is "donne-moi l'information technologique" (ie, "Give me information technology"); Assume that the ASR engine 111 outputs "Benoit la formation technologique" (ie, "Benoit skill training") as a recognized voice. In one embodiment, the NLU engine 326 adds "Benoit" to "donne-moi" and "formation" to "information" in the context to increase the accuracy of the subsequently determined user intent of the NLU engine 326. based on pre-processing.

The NLU engine 326 determines the user's intent based on speech recognized from the ASR engine 111 , which may optionally be preprocessed in some embodiments. In one embodiment, the NLU engine 326 determines the user's intent as a tuple. In one embodiment, a tuple contains an action (eg, a function to be performed) and an actor (eg, a module that performs the function). However, in some embodiments, a tuple may contain additional or different information. For example, the NLU engine 326 receives the recognized voice “Call Greg”; In one embodiment, the NLU engine 326 includes an action (ie, placing a call), an actor (ie, a phone module), and an entity (ie, a recipient of a call), sometimes also referred to as an “item”. Assume that we determine a tuple containing /Greg as object).

In one embodiment, the NLU engine 326 detects one or more of a keyword or a short cut. A keyword is a word that provides direct access to a module. For example, when the user says "phone", the phone module is accessed and the phone application is started (or brought to the foreground). Shortcuts are phrases (eg, send a message). Examples of keywords and shortcuts can be found in table 710 of FIG. 7 . In some embodiments, system 100 creates one or more shortcuts based on machine learning, which may be referred to as intent learning. For example, in one embodiment, system 100 "sends a message to Louis" causes user 112 to dictate an email (eg, rather than an SMS text message) by NLU engine 326 to a contact We learn that it should be interpreted as requesting to send to Louis Monier and to receive voice input to dictate an email and proceed directly to the interface setting "Send a message to Louis" as a shortcut.

In one embodiment, the natural language understanding function of the NLU engine 326 is modular, and the system 100 is agnostic with respect to the modules that perform natural language understanding. In some embodiments, modularity allows the NLU module of the NLU engine 326 to be updated frequently to continuously improve accurate understanding or to replace the natural language understanding module as new more accurate natural language understanding systems become available. make it possible

When the NLU engine 326 is unable to determine the user's intended request (eg, the request is ambiguous, the request does not make sense, or the requested action and/or action is not available or does not match, or the value is a tuple , or otherwise), the NLU engine 326 initiates a suboptimal solution. For example, when the user's request is incomplete (eg, the tuple is not complete), the NLU engine 326 requests the suboptimal engine 328 (discussed below) to prompt the user for additional information. . For example, when the user asks "What's on TV?", in one embodiment, the NLU engine 326 determines that the channel and time are missing, and initiates a suboptimal solution.

In one embodiment, the NLU engine 326 passes the tuple to the connection engine 330 . For example, the NLU engine 326 is communicatively coupled to the connection engine 330 to send the tuple to the connection engine 330 . In another embodiment, the NLU engine 326 stores the tuple in the storage device 241 (or any other non-transitory storage medium accessible by communication), and the connection engine 330 stores the tuple in the storage device 241 (or other non-transitory storage media).

In one embodiment, the NLU engine 326 passes the request for additional information to the suboptimal engine 328 . For example, NLU engine 326 is communicatively coupled to suboptimal engine 328 to transmit a request for additional information to suboptimal engine 328 . In another embodiment, the NLU engine 326 stores the request for additional information in the storage device 241 (or any other non-transitory storage medium accessible by communication), and the suboptimal engine 328 stores the request for additional information in the storage device ( 241) (or other non-transitory storage medium) to retrieve the request for additional information.

The suboptimal engine 328 includes code and routines for generating a request for additional information to the user so that the NLU engine 326 can determine the user's intended request. In one embodiment, the suboptimal engine 328 is a set of instructions executable by the processor 202 . In another embodiment, the suboptimal engine 328 is stored in the memory 204 and is accessible and executable by the processor 202 . In either embodiment, the suboptimal engine 328 is adapted for cooperation and communication with the processor 202 , other components of the server-side connection engine 124 , and other components of the system 100 .

The suboptimal engine 328 generates the request for additional information so that the user's intended request can be understood and executed. In one embodiment, the suboptimal engine 328 generates a conversation with the user to obtain the additional information by generating one or more requests for the additional information. For example, the suboptimal engine 328 generates a request for additional information and transmits the request for presentation to the user 112 via the client device (eg, sends the request to the text-to-speech engine 111 ). and the text-to-speech engine 111 presents the request to the user as audio output and for display on the display of the client device). A user's response is received (eg, as audio input received by the ASR engine 111 or via another user input device such as a keyboard or touch screen). The NLU engine 326 determines the user's intended request. When the NLU engine 326 is still unable to determine the user's intended request, the suboptimal engine 328 generates another request and the process repeats.

Examples of types of requests for additional information include a request that the proposed information is correct, a request to the user to repeat the original request in its entirety, a request to the user to clarify part of the original request, and a request to select from a list of options. may include, but is not limited to, one or more of a request to a user; For clarity and convenience, it may be beneficial to discuss the operation of the suboptimal engine 328 in the context of the following scenario. Assume that the user requests "Navigate to 1234 virtual street in any town in California". However, for whatever reason (eg, due to background noise, the user's intonation, errors in speech recognition), the NLU engine 326 cannot "guide" and I understood "California".

In some embodiments, the suboptimal engine 328 generates a request as to whether the suggested information is correct. In some embodiments, system 100 suggests side information based on machine learning. For example, suppose the system knows that the user drives to 1234 virtual street in any town in California every Wednesday. In one embodiment, the suboptimal engine 328 suggests the side information "California. Did you want to go to 1234 virtual street in any town?" In one embodiment, if the user says "yes", the tuple is complete and directions to the full address are performed, and if the user answers "no", then the suboptimal engine 328 sends another request (eg, a list of options). a request to the user to select from or to spell the destination).

In some embodiments, the suboptimal engine 328 issues a request to the user to fully repeat the original request. For example, the suboptimal engine 328 generates a request "Sorry. I don't understand. Would you like to repeat that?", which request is sent to the user via the user device 106 (visually, audibly, or both), and the user repeats "Navigate to 1234 Virtual Street in any town in California." In one embodiment, the suboptimal engine 328 does not issue a request to the user to repeat the original request, but one of the other types of requests is used. In one embodiment, the suboptimal engine 328 limits the number of times it will issue a request to the user to fully repeat the original request, based on a predetermined threshold (eg, 0 or 1). In one such embodiment, in response to meeting the threshold, the suboptimal engine 328 uses a different type of request for additional information (eg, prompting the user to select from a list of options).

In some embodiments, the suboptimal engine 328 issues a request to the user to partially repeat the original request or provide missing information from the original request. For example, the suboptimal engine 328 determines that "Navigation" and "California" are understood, determines that the street address and city are missing, and that the user provides the missing information (which was part of the original request). So, let's say we're generating the request "Sorry, what's the city and street address in California?" The request is presented to the user (visually, aurally, or both) via the user device 106 , and the user can say “1234 virtual address of any town”. In one embodiment, the suboptimal engine 328 limits the number of times it will issue a request to the user to repeat the same portion of the original request, based on a predetermined threshold (eg, 0, 1, or 2). In one such embodiment, in response to meeting the threshold, the suboptimal engine 328 uses a different type of request for additional information (eg, prompting the user to select from a list of options).

In some embodiments, the suboptimal engine 328 generates a request for the user to select from a list of options, sometimes referred to as a “default list.” For example, the suboptimal engine 328 determines that "Navigation" and "California" are understood, determines that the street address and city are missing, and generates the request "What letter does the city of the destination start with?" , and suppose that it generates a list of options such as "A through E equal 1, F through J equal 2, ... etc." The request is presented to the user (visually, audibly, or both) via the user device 106 , and the user can say or select “1” or select by saying the content of options “A through E”. can Since the NLU engine 326 is still unable to determine the user's intended request from the city of California that begins with "navigation" and the letters between 'a' and 'e' (inclusive), the next best workaround engine 328 gives rise to another list of options such as "A is 1, B is 2, ...etc". The request is presented to the user (visually, audibly, or both) via the user device 106 , and the user can say or select “1” or select by the content of option “A”. The suboptimal engine 328 filters the options and lists the filtered options until "any town" is identified as a city, "virtual street" is identified as a street, and "1234" is identified as a street number. may continue to issue requests with

Depending on the embodiment, the options may be listed visually on the display of the client device, read to the user 112 via the client device 106 using text-to-speech, or both. In one embodiment, the list options are presented as groups at a time (eg, as groups of three to five). For example, a list of 8 options may consist of 2 sets and presented as a first set of 4 options, the user may request the next set by saying "next", Two sets are presented. Limiting the number of options presented at one time may reduce the likelihood that the user will be overwhelmed and may improve usability. To navigate the lists of options divided into multiple sets, in one embodiment, the user "begins" to go to the first set of the list, "end" to go to the end of the list, and to the next set in the list. to use commands, such as "next" to go, and "previous" to go to the previous set in a list, or "go to ___" to navigate or filter by letter (eg, "go to letter V"). can

In some embodiments, the conversation resulting from the requests of the suboptimal engine 328 may switch between request types in any order. For example, in one embodiment, the suboptimal engine 328 may prompt the user for additional information without a list of options when the user selects an option. For example, upon receiving/determining that “any town” is a city using the list of options as described above, the next best option engine 328 may ask, “What is the name of a street in any town in California?” may generate a request, and the user may verbally respond with "virtual street". If the response “virtual street” is not understandable, in one embodiment, the suboptimal engine 328 may ask the user to repeat or ask the user to select from a list of options generated by the suboptimal engine 328 .

In some embodiments, requests generated by suboptimal engine 328 are generated to minimize or eliminate the need for the user to respond negatively (eg, say "no"). For example, the suboptimal engine 328 generates a list of options for the first letter of a city, and "Does a California city start with the letter A?" - would be 'yes' in the case of the above example, but this request would In some cases, it is likely to result in a 'no' - rather than sending similar requests, it asks the user to select the appropriate option.

It will be appreciated that the above ".... Navigate to 1234 Virtual Street" example is one use case, and that many other use cases exist. For example, a user requests "Call Greg" and the user has multiple contacts named Greg in the address book (eg, Greg R., Greg S. Greg T.); In one embodiment, the suboptimal engine 328 sends a request with a list of options: "Which Greg do you want to call? Greg R. is 1, Greg S. is 2, Greg T. is 3." , suppose the user can say the number associated with the desired Greg.

Moreover, in the examples above, the actors (ie, navigation application and phone application, respectively) and portions of entities (ie, California and Greg, respectively) that were part of the original request were understandable by NLU engine 326 , but suboptimal Engine 328 may operate when the entire original request was not understandable by NLU engine 326 or when other parts of the tuple are missing. For example, the suboptimal engine 328 may include a desired actor (eg, an application that the user intends to use), a desired action (eg, a function or characteristic of an application), a desired entity (eg, an object of the action, a recipient of the action). , input for action, etc.) may make one or more requests. In one embodiment, suboptimal engine 328 issues requests upon request of NLU engine 326 or until NLU engine 326 has a complete tuple representing the user's intended request. In another example, the NLU engine 326 understands the message, but does not understand the actors (eg, which services in the Unified Messaging client to use - email, SMS, Facebook, etc.) and entities (eg, recipients) and ; In one embodiment, assume that the suboptimal engine 328 requests this side information.

Features and functionality discussed above with reference to suboptimal engine 328 advantageously require the user to type portions of the request, which may be dangerous or illegal in certain constrained operating environments (eg, while driving). provide automatic troubleshooting mechanisms by which the user's intended request can be determined and ultimately executed without notice (eg, the user may speak and/or may make a simple selection via a touch screen or other input); It will be appreciated that this improves the safety of user 112 and those around user 112 . Moreover, because of the features and functionality discussed above with reference to suboptimal engine 328, advantageously, more It will be appreciated that user satisfaction can be obtained.

In one embodiment, the suboptimal engine 328 passes the request for additional information to one or more of a text-to-speech engine 119 and a graphics engine (not shown) for displaying the content on the display of the client device. . In another embodiment, the suboptimal engine 328 stores the request for additional information in the storage device 241 (or any other non-transitory storage medium accessible by communication). Other components of system 100, including, for example, text-to-speech engine 119 and/or graphics engine (not shown) may access storage device 241 (or other non-transitory storage medium) for additional information. It can retrieve the request and send it for presentation to the user 112 via the client device 106 .

Connection engine 330 includes code and routines for handling the user's intended request. In one embodiment, the connection engine 330 is a set of instructions executable by the processor 202 . In another embodiment, the connection engine 330 is stored in the memory 204 and is accessible and executable by the processor 202 . In either embodiment, the connection engine 330 is adapted for cooperation and communication with the processor 202 , other components of the client device 106 and other components of the system 100 .

In one embodiment, the connection engine 330 includes a library of modules (not shown). A module may contain a set of code and routines that expose the functionality of an application. For example, the phone module exposes the functionality of a phone application (eg, making a call, answering a call, searching for voice mail, accessing a contact list, etc.). In one embodiment, the module exposes the functionality of an application (eg, a phone application), such that a user can access such functionality on a client device (eg, a phone) via another client device 106 (eg, a car). make it In some embodiments, certain features and functions may require the presence of a particular device or type of device. For example, in some embodiments, phone or SMS texting functionality may not be available via the vehicle unless the vehicle is communicatively coupled with the phone. The library of modules and the modular nature of modules may allow for easy updating as applications are updated or when it becomes desirable for the voice and connectivity engine to interface with new applications.

In some embodiments, when a function takes a long time to complete (eg, when creating a long report), the agent/assistant notifies the user when the function is complete (eg, TTS, email, SMS text, etc.) something to do. In one such embodiment, the system 100 determines the fastest way to contact you - for example, the system determines that the user is logged in to Facebook and sends a Facebook message to the user indicating that the function has been completed. Send -.

In one embodiment, the voice assistant of system 100 includes one or more modules for interacting with one or more other voice assistants (eg, Siri from Apple, Cortana from Microsoft, Google Now from Google, etc.). For example, in one embodiment, in response to the user providing voice input including a shortcut or keyword such as "Search Google Now for X" or "Ask Siri for Y", the connectivity module 330 selects a module 330 for connecting to or interacting with Google Now or Siri, respectively, and forwards the query to its voice assistant. In one embodiment, the voice and connectivity engine 109/124 triggers the personal assistant of the system 100 to resume control of the flow of the user experience (eg, to resume a conversation or provide functionality and assistance). Voice inputs can be monitored for a wake-up word. This embodiment advantageously enables an entity operating in system 100 to provide its customers with access to other voice assistants and their features. For example, car manufacturers may advantageously allow customers to access the voice assistant of their customer's cell phone (eg, Siri when the customer is using an iPhone) or supplement a customer's voice assistant options with another voice assistant (eg, customer (providing access to Google Now and/or Cortana) when using this iPhone.

The connection engine 330 processes the user's intended request. In one embodiment, the connection engine 330 receives the tuple from the NLU engine 326, determines a module (eg, a phone module) based on the actor (call) in the tuple, and determines the behavior of the tuple (eg, a call). ) and an entity/item (eg, Greg) to the determined module, which causes the actor application to perform an action using the entity/item (eg, causes the phone application to call Greg).

Exemplary Server-Side Voice and Connectivity Engine (124)

Referring now to FIG. 4 , a server-side voice and connectivity engine 124 is illustrated in greater detail in accordance with one embodiment. In the illustrated embodiment, the server-side voice and connection engine 124 includes a context agent 422 , a context engine 424 , and a federation engine 426 . It will be appreciated that not all components 422 , 424 , 426 included in the server-side voice and connectivity engine 124 are necessarily on the same voice and connectivity server 122 . In one embodiment, modules 422 , 424 , 426 and/or their functionality are distributed across multiple voice and connectivity servers 122 .

The context agent 422 includes code and routines for synchronizing and maintaining the context between the client device 106 and the voice and connectivity server 122 . In one embodiment, the context agent 422 is a set of instructions executable by the processor 202 . In another embodiment, the context agent 422 is stored in the memory 204 and is accessible and executable by the processor 202 . In either embodiment, the context agent 422 is the processor 202 , (eg, via bus 206 ) voice and other components of the connectivity server 122 , and other components of the system 100 (eg, communication). The unit 208 is adapted for cooperation and communication with client devices 106 ) and other components of the server-side voice and connectivity engine 124 .

As discussed above with reference to client-side context holder 324 , context agent 422 acts as a server-side context holder and is synchronized with client-side context holder 324 . In one embodiment, if the client-side and server-side contexts are not the same, the client-side substitutes. Because the client side interacts more directly with the user 112 and thus may be more likely to have more accurate real-time data (e.g., location, luminosity, local time, temperature, speed, etc.) to define the context, It may be beneficial for the client-side to replace the server-side because, for example, the associated sensors are located on the client device 106 and the network 102 reliability is accurate and maintains an up-to-date context. Because it can affect your ability.

In one embodiment, the context agent 422 passes the current context to the context engine 424 . For example, the context agent is communicatively coupled to the context engine 424 for sending the current context. In one embodiment, the context agent 422 stores the current context on the storage device 241 (or any other non-transitory storage medium accessible by communication), and the context engine 424 stores the current context on the storage device 241 ( or other non-transitory storage media) to retrieve the current context.

Context engine 424 includes code and routines for generating and maintaining one or more contexts. In one embodiment, context engine 424 is a set of instructions executable by processor 202 . In another embodiment, the context engine 424 is stored in the memory 204 , and is accessible and executable by the processor 202 . In either embodiment, context engine 424 is adapted for collaboration and communication with processor 202 , other components of server-side voice and connectivity platform 124 , and other components of the system.

In one embodiment, the context engine 424 archives the current context to create a history of contexts. Such an embodiment may be useful in recognizing patterns or habits, predicting next steps in a workflow, etc., in order to inform understanding of the NLU engine 326 or to proactively initiate a conversation, etc. It can be used in connection with learning. For example, user x is a closed profile from a group of user type X; In one embodiment, assume that the context engine 424 detects the difference between x and all other people in the group to capture a particular behavior, habit, query, etc. and create a proactivity for the user. . For example, a user is asking a question about a theater and the context engine 424 detects that other users in the same group like a particular Japanese restaurant; In one embodiment, because the system 100 has detected in the user's schedule that the user will not have time before the movie, the system 100 proactively suggests to the user to make a reservation at the Japanese restaurant after the movie. Let's assume In some embodiments, system 100 may access an API from a restaurant menu (some websites provide this kind of API). The system 100 may understand that the menu or specials of the day are well suited to the user's preferences, and in the agent's answer, read the menu or specials of the day directly to grab the user's attention.

Federation engine 426 includes code and routines for managing a user's accounts and one or more of client devices 106 . In one embodiment, the association engine 426 is a set of instructions executable by the processor 202 . In another embodiment, the federation engine 426 is stored in the memory 204 and is accessible and executable by the processor 202 . In either embodiment, the federation engine 426 is adapted for collaboration and communication with the processor 202 , other components of the application server 122 and other components of the development application 124 .

In one embodiment, federation engine 426 manages a unified identity. The unified identity can be applied to the user's accounts (eg, Facebook, Google+, Twitter, etc.), the user's client devices 106 ( eg, tablet, cell phone, TV, car, etc.), previous voice inputs and conversations, and the like. The unified ID provides aggregated information about the user that can enhance the features and functionality of the system 100 . For example, suppose user 112 provides the input “I need gasoline”. In one embodiment, access to aggregated data of the aggregated ID allows the system 100 to determine that the user's intended request is for directions to a gas station and that the gas station's favorite bar (eg, the user is a regular). The nearest gas station is behind or closer to the user, but the system 100 does not allow the user to Even if you're away from where you decide you're heading, it'll make sure you understand that you're on your way to the brand's gas station in the direction you're heading to along the way to the bar. In another example, system 100 provides system 100 with user 112 regarding food (eg, previous reservations made using a service such as open table, the user's restaurant comments on Yelp, and food). ) may use the aggregated data to select a particular restaurant and direct the user to it) based on aggregated data, such as previous voice queries and conversations between

Federation engine 426 manages a user's devices to coordinate the user's transition from one client device 106 to another. For example, suppose user 112 requested today's headlines via the user's tablet (eg, client device 106 ) and system 100 begins reading the headlines to user 112 . . Also assume that the user 112 then realizes that he will be late for work and requests to stop reading the headlines. In one embodiment, the federation engine 426 manages the user's transition from the tablet to the user's car (ie, another client device 106 ), so that the user 112 , once in the car, the system 100 . ) to continue, and the system 100 will continue reading the headlines from where it left off with the tablet. The federation engine 426 may also suggest and manage the transition to the user's mobile phone (ie, another client device 106 ) when the user arrives at work. Such embodiments advantageously provide for continuity or “continuous service” of service from one client device 106 to another. In another example, a user may plan a road trip through a tablet on the sofa and have the route mapped to the car's navigation system. In one embodiment, system 100 may recognize that the user has a habit of reviewing headlines before going to work and continuing in the car on the way to work, and when to go to work (perhaps based on real-time traffic data). may prompt the user on the tablet to confirm and ask if the user would like to resume headlines in the car.

In one embodiment, the federation engine 426 passes the context from one client device 106 to another to manage the transition to the recipient device. For example, the federation engine 426 is communicatively coupled to the client-side context holder 324 of the recipient device. In another embodiment, the federation engine 426 stores the current context in the storage device 241 of the server 122 (or any other non-transitory storage medium accessible by communication), and the client of the recipient device 106 . The side context holder 324 can retrieve the current context by accessing the storage device 241 (or other non-transitory storage medium).

Exemplary methods

5, 6, and 7 illustrate various methods 500, 508, and 700 performed by the system described above with reference to FIGS.

5 , illustrated is an exemplary method 500 of receiving and processing a request using a voice and connectivity platform in accordance with one embodiment. At block 502 , the NLU engine 326 receives the recognized speech. At block 504 , the NLU engine 326 receives the context. At block 506 , the NLU engine 326 preprocesses the recognized speech, optionally based on the context received at block 504 . At block 508, the NLU engine 326 determines the user's intended request. At block 510 , the connection engine processes the intended request and method 500 ends.

6 , shown is an exemplary method 508 of determining a user's intended request, according to one embodiment. At block 602, the NLU engine 326 generates the tuple based on the user's request and context. At block 604, the NLU engine 326 determines if additional information is needed to complete the tuple. When the NLU engine 326 determines that no side information is needed to complete the tuple (604-No), the method 508 ends. When the NLU engine 326 determines that additional information is needed to complete the tuple (604-Yes), the method 508 continues at block 606 .

At block 606, the suboptimal engine 328 determines what additional information is needed to complete the tuple, and at block 608 generates a prompt for the user to provide the necessary additional information. At block 610, the NLU engine 326 modifies the tuple based on the user's response to the prompt generated at block 610, and the method continues at block 604, where the NLU engine 326 Blocks 604, 606, 608, and 610 are repeated until it is determined that no additional information is needed to complete (604-No) and the method 508 ends.

Referring to FIG. 7 , illustrated is an exemplary method 700 of receiving and processing a request using a voice and connectivity platform, according to another embodiment.

In the foregoing description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. However, it will be understood that the technique described herein may be practiced without these specific details. In addition, various systems, devices, and structures are shown in block diagram form in order to avoid obscuring the description. For example, various implementations are described as having specific hardware, software, and user interfaces. However, the present disclosure applies to any type of computing device capable of receiving data and instructions and any peripherals that provide services.

Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The phrases "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

In some cases, various implementations may be presented herein as algorithms and symbolic representations of operations on bits of data within a computer memory. An algorithm is here generally conceived of as being a self-consistent set of operations that produces a desired result. Operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transmitted, combined, compared, and otherwise manipulated. Because of their common usage in principle, it has been found convenient at times to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient designations applied to these quantities. Unless specifically stated otherwise, throughout this disclosure, as will be apparent from the discussion below, discussions using terms including "processing" or "compute" or "calculate" or "determining" or "display" and the like , data represented in physical (electronic) quantities in registers and memories of a computer system, to other data similarly represented in physical quantities in computer system memories or registers or other such information storage, transmission or display devices. It will be understood that reference to the operations and processes of a computer system or similar electronic computing device that manipulates and transforms.

Various implementations described herein relate to apparatus for performing the operations herein. This apparatus may be configured in a particular manner for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored thereon. This computer program is a read-only (ROM) disk, suitable for storing electronic instructions, each of which is coupled to a computer system bus, including floppy disks, optical disks, CD-ROMs, and magnetic disks. memory), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, flash memory including a USB key with non-volatile memory, or any tangible medium, including but not limited to computer-readable storage media. can be stored in

The technology described herein may take the form of an entirely hardware implementation, an entirely software implementation, or implementations comprising both hardware and software elements. For example, the subject technology may be implemented in software including, but not limited to, firmware, resident software, microcode, and the like.

Furthermore, the subject technology may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium is any non-transitable medium that can contain, store, transfer, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. It may be a temporary storage device.

A data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. Memory elements include local memory used during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code to reduce the number of times the code must be retrieved from the mass storage during execution. may include Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intermediary I/O controllers.

Network adapters may also be coupled to the system to enable the data processing system to be coupled to other data processing systems, storage devices, remote printers, etc. via intermediate private and/or public networks. Wireless (eg Wi-Fi ^™ ) Transceivers, Ethernet adapters, and modems are just a few examples of network adapters. Private and public networks may have any number of configurations and/or topologies. Data may be transferred between these devices over networks using a variety of different communication protocols, including, for example, various Internet layer, transport layer, or application layer protocols. For example, if the data is TCP/IP (transmission control protocol / Internet protocol), UDP (user datagram protocol), TCP (transmission control protocol), HTTP (hypertext transfer protocol), HTTPS (secure hypertext transfer protocol), DASH (dynamic adaptive streaming over HTTP), real-time streaming protocol (RTSP), real-time transport protocol (RTP) and real-time transport control protocol (RTCP), voice over Internet protocol (VOIP), file transfer protocol (FTP), WS (WebSocket), wireless access protocol (WAP), various messaging protocols (SMS, MMS, XMS, IMAP, SMTP, POP, WebDAV, etc.), or other known protocols.

Finally, the structure, algorithms, and/or interfaces presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the required method blocks. The required structure for a variety of these systems will emerge from the description above. In addition, the specification is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the specification as described herein.

The foregoing description has been presented for purposes of illustration and description. This description is not intended to be exhaustive or to limit the specification to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims of the present application. As will be understood, the specification may be embodied in other specific forms without departing from its spirit or essential characteristics. Likewise, specific naming and divisions of modules, routines, features, properties, methods, and other aspects are neither essential nor critical, and that the specification or mechanisms implementing its features may differ by different names, divisions, and/or divisions. or formats.

Moreover, engines, modules, routines, features, properties, methods, and other aspects of the present disclosure may be implemented as software, hardware, firmware, or any combination thereof. Moreover, in any case in which a component of the specification - an example of which is a module - is implemented as software, the component is a standalone program, as part of a larger program, as a plurality of separate programs, statically or dynamically linked. It may be implemented as a library, as a kernel loadable module, as a device driver, and/or in any and all other ways now known or known in the future. In addition, the disclosure is in no way limited to implementations in any particular programming language or for any particular operating system or environment. Accordingly, the present disclosure is intended to be illustrative and not limiting of the scope of the subject matter set forth in the following claims.

Appendix A: Car Personal Assistant and GoPad

GoPad Project Summary

GoPad is an accessory product that generates Android device-user and vehicle-user behavior data by providing a safer and more convenient in-car Android device experience. GoPad will more tightly integrate selected Android devices into vehicles. However, GoPad is not limited to integration with Android devices and may integrate with other devices (eg, iOS, Windows, Fire, etc.).

The GoPad device is a hardware cradle that will be secured to the dashboard of the user's vehicle near the windshield via a clip mechanism. This will provide the following features:

차량 정보를 포착하고 분석 및 사용자에의 제시를 위한 시스템들로 전송하는 OBD2 리더(OBD2 Reader) 하드웨어 디바이스

OBD2 Reader hardware device that captures vehicle information and transmits it to systems for analysis and presentation to the user

임베디드 블루투스 연결이 없는 차량들에서 핸즈프리 능력을 제공하기 위해 크레이들에 있는 블루투스 무선 및 듀얼 마이크로폰들

Bluetooth radio and dual microphones in cradle to provide hands-free capability in vehicles without an embedded Bluetooth connection

차량 스테레오 시스템에의 보조 입력(Aux-in) 연결을 통한 오디오를 사용한, 음성 다이얼링 및 제어를 비롯한, 핸즈프리 휴대폰 사용

Hands-free mobile phone use, including voice dialing and control, with audio via an Aux-in connection to the vehicle stereo system

차량 스테레오 시스템에의 보조 입력 연결을 통한 오디오를 사용한, 음성 개시 및 음성 제어를 비롯한, 핸즈프리 내비게이션

Hands-free navigation, including voice initiation and voice control, using audio via an auxiliary input connection to a vehicle stereo system

보조 입력 스테레오 연결을 통한 자동차 스테레오에의 오디오 출력을 사용한 미디어 재생

Media playback with audio output to car stereo via auxiliary input stereo connection

충전 및 사용을 위해 USB(차량 보조 전원 포트)를 통한 안드로이드 디바이스에의 전력 공급

Powering Android devices via USB (Vehicle Auxiliary Power Port) for charging and use

음성 및 연결 플랫폼을 통한 모든 음성 제어 기능들에 대한 지능적 에이전트 도움

Intelligent agent assistance for all voice control functions via voice and connectivity platform

음성 및 연결 플랫폼을 통한 지능적 에이전트, 사용자 데이터 포착, 및 콘텐츠의 전달을 위한 클라우드-연결 웹 서비스

Cloud-connected web services for intelligent agents, user data capture, and content delivery via voice and connectivity platforms

사용자의 운전 경험을 향상시키기 위한 안드로이드 디바이스 상의 운전 효율 및 피드백 특징들

Driving Efficiency and Feedback Features on Android Devices to Improve User's Driving Experience

안드로이드 디바이스의 아이즈프리(eyes-free) 사용을 추가로 가능하게 하기 위한 크레이들 상의 최적화된 한 세트의 물리적 컨트롤들

An optimized set of physical controls on the cradle to further enable eyes-free use of Android devices.

운전자들이 사용하고자 하는 앱들을 쉽고 안전하게 시작할 수 있게 하는 간단한 앱 시작 관리자(launcher) 메커니즘

A simple app launcher mechanism that allows drivers to easily and safely launch the apps they want to use.

API 써드파티 소프트웨어가 크레이들의 물리적 버튼들을 이용할 수 있게 하는 간단한 물리적/에이전트 컨트롤들

Simple physical/agent controls that allow API 3rd party software to use the cradle's physical buttons

핸즈프리 착신 문자 메시지 읽기

Read incoming text messages hands-free

핸즈프리 페이스북 활동 읽기

Read hands-free Facebook activity

Cradle Hardware

Cradle design

mechanical design

The cradle will be designed in two parts: 1) a base cradle unit, and 2) a device specific adapter. All major functions will go into the base cradle unit, and the adapter only provides Android device-specific physical and electrical fit capabilities.

The cradle's physical form factor should accommodate the device + adapter (rigidly), specified physical controls, and cradle motherboard while minimizing size and volume. The device shall not be insertable in the opposite direction or upside down.

The cooling of the cradle electronics would be passive by vents included in the design or hidden from view as much as possible.

industrial design

The overall design of the cradle should help the user complete actions with as little direct observation/interaction as possible. Buttons should have tactile differentiation, audible/tactile cues should be used where appropriate, etc.

The cradle industrial design is undecided, but a very high fit-and-finish level is required for public demonstration. The cradle has the feel of a real commercial product made to a luxury level. The cradle does not feel out of place in the interior of a high-end Audi or Mercedes car, and matches these interiors in terms of material quality and appearance.

Finishing material surveys should include paints, machined metals, machined plastics, rubber paints, etc.

physical control

button

The cradle will include handpicked physical controls (buttons) that aid in ease of use of Eyesfree.

The following buttons are required:

에이전트 버튼: 음성 제어를 활성화, 앱 시작 관리자를 활성화, 기타

Agent button: enable voice control, enable app launcher, etc.

앞으로 버튼: 다음 미디어 트랙, 전화 통화 종료/거부

Forward button: next media track, end/reject phone call

뒤로 버튼: 이전 미디어 트랙, 전화 통화 응답

Back button: previous media track, answer phone calls

재생/일시중지 버튼: 재생 또는 미디어 재생을 일시중지, 전화 통화 음소거

Play/Pause button: Pause play or media playback, mute phone calls

Buttons enable multiple overloaded actions based on how they are used (single press, double press, long press, etc.).

light

Needs backlighting/highlighting of physical controls for use in low light environments. The light/legend should behave like this:

앞으로/통화 종료 버튼: 전화 통화가 활성일 때를 제외하고는 디폴트 조명을 사용해야만 한다. 통화가 활성일 때, 통화가 종료될 때까지 통화 종료 범례가 발광되어야 한다.

Forward/End Call Button: The default lighting should be used except when a phone call is active. When a call is active, the call end legend should flash until the call is ended.

뒤로/통화 응답: 전화 통화가 들어오고 있을 때를 제외하고는 디폴트 조명을 사용해야만 한다.

Back/Call Answer: You should use the default lighting except when an incoming phone call is in progress.

재생/일시중지, 음소거: 통화가 활성일 때, 통화 음소거 범례가 발광되어야 한다. 버튼이 눌러지면, 통화가 음소거 상태에 들어가야만 하고, 음소거 상태를 표시하기 위해 음소거 범례 백라이트가 적색으로 변해야 한다. 버튼을 다시 누르는 것은 음소거 상태 및 범례 백라이트 색상을 토글시킬 것이다.

Play/Pause, Mute: When a call is active, the call mute legend should flash. When the button is pressed, the call should enter the muted state, and the mute legend backlight should turn red to indicate the muted state. Pressing the button again will toggle the muted state and the legend backlight color.

There is a need for an unobtrusive and/or attractive pilot light to indicate cradle power on.

Upgradable Firmware

Cradle firmware is designed so that field upgrades can be performed under the control of the GoPad Android application running on the device.

Mechanisms exist to recover from a corrupted firmware update, such as can be from a device being removed from the cradle during an update operation.

USB Audio

The cradle design can accommodate receiving USB audio from a device (if the device has the capability) and relaying it to the cradle line-out for playback via the car stereo auxiliary input.

everyone

maximum power supply

The cradle will always be able to supply 2A to the device at 5.1V, in addition to its own power requirements.

device charging

The cradle can supply sufficient power to each device so that the following functions can increase its state of charge while being used simultaneously.

핸즈프리 전화 통화 진행 중

Hands-free phone call in progress

핸즈프리 내비게이션 진행 중

Hands-free navigation in progress

미디어 재생 진행 중(어쩌면 일시중지됨)

Media playback in progress (maybe paused)

Unique device and version ID

The cradle can support both hardware and firmware version numbers as well as unique device IDs. The Android application will be able to read/query for these unique IDs.

Cradle logging

Cradle can support activity logging for software development and debugging. These logs may be accessible by the Android application.

Examples of items to log include, but are not limited to: USB connection status, button press, Bluetooth connection status, and the like.

cable

The cables you will need are:

USB 케이블(전원용)

USB cable (for power)

스테레오 보조 케이블(오디오 출력용)

Stereo auxiliary cable (for audio output)

OBD2 Reader

A hardware OBD2 reader device is required. The device will collect vehicle information and upload this vehicle information to OPI systems for analysis and subsequent presentation to the user.

The OBD2 reader module will include a Bluetooth radio, collecting information whenever the GoPad is in use. The OBD2 reader module sends the information to the device, which later uploads the information to the OPI systems for analysis.

An alternative OBD2 reader module that includes a cellular radio that collects vehicle information whenever the vehicle is running, regardless of whether a GoPad is being used, is highly desirable for future GoPad versions. This solution will be studied in parallel with the development of GoPad2. A third party partner (OEM manufacturer) is desired.

GoPad-based hands-free capabilities

For vehicles that do not have the built-in Bluetooth hands-free capability, the GoPad will provide these features. The following hardware components are required.

dual microphone

Along with echo cancellation and noise suppression techniques, dual microphones are required. A very high level of audio quality is required. It is desirable that the person at the remote end of the phone call cannot determine that the user is speaking through a hands-free device in the vehicle.

The audio quality benchmark device is a Plantronics Voyager Legend BT headset.

bluetooth wireless

The GoPad cradle will include a Bluetooth radio that supports the hands-free profile. The device will automatically connect to the cradle BT radio when inserted into the cradle and disconnect when removed. When the BT connection is broken for any reason, the connection will be reestablished immediately.

android app software - one embodiment of release

Lightweight Launcher

The lightweight launcher will be activated automatically when the device is placed in the cradle. The lightweight launcher, if active, should be deactivated when the phone is removed from the cradle. The initial setup experience should be as smooth as possible and should require minimal manual configuration by the user.

In the first release, the launcher provides access to the following features:

디폴트 바로 가기 바:

Default shortcut bar:

o phone calls

o Messages: text, mail and Facebook messages

o Navigation

o Newscaster: general and topical news + Facebook user timeline

o Media playback: local and online streaming media

자동차 개인 어시스턴트

car personal assistant

애플리케이션 리스트

application list

차량 모듈

vehicle module

GoPad 설정

GoPad Settings

Upon insertion into the cradle, the launcher will display a splash screen for a short duration. The launcher will then display the lightweight launcher home screen and wait for user input.

Subsequent double press of the agent button, whatever application is currently in the foreground, will bring up a lightweight launcher and allow the user to select a new function. If the GoPad app is already in the foreground, a double press of the agent button will take the user back to the home screen.

system volume

The launcher will set the audio output volume to a fixed level (to be determined), and the user will adjust the volume using the vehicle stereo volume control.

screen brightness

When in the cradle, the device should have automatic screen brightness control. When the device is removed from the cradle, it must revert to the user's settings.

physical control

Physical controls on the cradle will have the following functions depending on how they are used:

Control one click double click click and hold Previous

이전 트랙(미디어)

통화 응답(전화)

Previous track (media)

Answering Calls (Phone) next

Next Track (Media)

End/Reject Call (Phone) Play/Pause

Play/Pause Toggle (Music)

Mute a call (phone)

media player
(GoPad) agent

Initiation/Cancellation Agent

Home screen (GoPad)

GoPad Launcher (Third Party App)

car personal assistant

The car personal assistant (agent) is activated by pressing the agent button once. The agent will respond with a voice, announcing his readiness status.

The sequence behavior of the agent button consists of three steps:

1. Standby mode: the user needs to press a button to activate voice recognition.

2. Talk mode: The agent is speaking a prompt to the user.

3. Listening mode: The agent is listening to the user's sentences.

In this release, the functions the agent will handle are limited to the following:

특징 카테고리들(전화, 메시지, 내비게이션, 미디어, 뉴스/페이스북, 차량, 설정) 간의 앱내 탐색

In-app navigation between feature categories (Phone, Messages, Navigation, Media, News/Facebook, Vehicle, Settings)

통화 응답/통화 거부/연락처로부터 다이얼링/통화 이력으로부터 다이얼링/임의의 번호의 다이얼링. 통화를 거부하는 것이 API에 의해 지원되지 않는 것처럼 보이기 때문에, 사용자가 거부하기로 선택하는 경우, 벨 울림을 중단시키고 착신 통화 디스플레이를 지워야 하며, 이어서 사용자가 통화에 응답하지 않은 것처럼 통화가 음성 메일로 자연스럽게 넘어갈 수 있게 한다(이것이 본질적으로 일어난 일임).

Call answer/Call reject/Dial from contact/Dial from call history/Dial any number. Since rejecting a call does not appear to be supported by the API, if the user chooses to reject, it should stop ringing and clear the display of the incoming call, then the call will go to voicemail as if the user did not answer the call. Let it pass naturally (this is essentially what happened).

내비게이션을 개시/취소하는 것. 주소를 직접 말하는 것 또는 주소를 간접적으로 말하는 것(주소의 일부: 국가, 타운, 스트리트, ...에 의해), 연락처로부터 주소를 가져오는 것, 위치 즐겨찾기로부터 주소를 가져오는 것.

To start/cancel navigation. Saying an address directly or indirectly saying an address (part of an address: by country, town, street, ...), getting an address from a contact, getting an address from a location favorite.

지역 업체를 탐색하고("가장 가까운 스타벅스를 찾아줘") 그곳으로 내비게이션을 개시하는 것.

Searching for local businesses ("find the nearest Starbucks") and starting navigation there.

o Local vendors are found in Google Maps API or Yelp, and a generic connector is needed to enable integration of any local vendor location source API in the future.

로컬 미디어를 재생하는 것. 재생 리스트/앨범/아티스트/노래/셔플.

Playing local media. Playlist/Album/Artist/Song/Shuffle.

o Online media needs to be integrated with the second version of CPA: Spotify, Pandora,

차량 상태 경고(통지 전용). 연료 부족. 엔진 점검 등, 기타.

Vehicle health alerts (notifications only). lack of fuel. engine check, etc.

써드파티 애플리케이션을 이름으로 시작하는 것.

Starting a third-party application by name.

뉴스 카테고리를 선택하고 읽는 것

Selecting and reading news categories

페이스북 업데이트를 읽는 것

Reading Facebook Updates

A disambiguation function that reduces multiple matches is needed (see screens below).

General User Experience: Speech and Normal Patterns

normal pattern

The approach to building the voice scenario of the application is based on the facts:

음성 인식이 동작할 확률이 아주 제한된다

Speech recognition has a very limited chance of working

에이전트가 부정적 상호작용을 제한할 필요가 있다

Agents need to limit negative interactions

사용자가 원하는 행동을 달성하기 위해 가능한 한 음성 명령을 보다 적게 제공할 필요가 있다.

There is a need to provide as few voice commands as possible to achieve the desired behavior of the user.

임의의 상호작용의 수행을 ASR 신뢰도에 의해서가 아니라 달성하는 시간으로 평가할 필요가 있다.

It is necessary to evaluate the performance of any interaction by the time to achieve it, not by the ASR reliability.

For this vision to be successful, the agent needs to use an intelligent combination of both types of scenarios: the Direct Voice pattern and the Work-a-round pattern.

direct speech pattern

Direct speech patterns are common in the realm of speech recognition. Its quality is validated by the reliability of ASR and the reliability of Natural Language Understanding (NLU).

In the case of the phone module and the action to make a call, you can ask "Call Bastien Vidal" (unique contact with one phone number), the agent will find the contact immediately and tell the user the action to call Bastien Vidal will suggest

The problem with direct speech patterns arises when there is no direct match to the user's voice query or when additional information is required from the user to achieve an explicit action.

Sample of cases:

많은 전화 번호를 갖는 사람에게 전화하고자 하는 것

Trying to call someone who has many phone numbers

많은 전화 번호 및 이메일 주소를 갖는 사람에게 메시지를 보내고자 하는 것

You want to send a message to someone who has many phone numbers and email addresses.

직접 음성 인식에 의한 주소가 틀리고 (운전 중이기 때문에) 아무 것도 타이핑할 수 없는 것.

The address by direct speech recognition is wrong (because you're driving) and you can't type anything.

Next best pattern (WAR)

The WAR pattern is that speech and connectivity platforms continue the conversation between humans and machines (after any one question/answer, the agent will automatically initiate activation of the speech recognition button) and Temporal Dialog Matrix Context (TDMC) It is based on the fact that it enables the creation of (see below for a description).

Conversation continuation enables creation of different types of WAR scenarios

리스트 항목 선택

Select list item

o For any list with a selection of navigation item steps and numbers

빈도수 이력 사전 대응성

Frequency History Proactive Responsiveness

o

단계별 선택

Step-by-step selection

Each item screen of the application is based on presenting a list item with attributes:

각각의 항목은 1부터 5까지의 숫자를 갖는다

Each item has a number from 1 to 5

각각의 항목은 라벨에 의해 읽혀진다

Each item is read by a label

Present a list of general items

일반 리스트

general list

o Entity Filter

o Alphabet filter

알파벳 숫자

alphanumeric

이력 숫자

history number

Present frequency list of history

splash screen

When the Android app is launched and whenever the device is placed in the cradle, a splash screen displaying the brand will be displayed briefly.

login screen

The launcher login screen is followed by a splash screen when the phone is placed in the cradle for the first time or when the user has explicitly logged out of the Android application. This will display the brand and suggest login by username/password. An account creation rank will also be presented, allowing the user to create a new account, if necessary, via email, username/password or Facebook account.

login Registration option registration screen

home screen

When the home button is pressed or after the phone is placed in the cradle, the home screen displays a view of your current location with some status information (temperature and compass direction) spanning the bottom as well as shortcut buttons for key functions spanning the top. It will display a map. The top bar will also reflect status and notification information where appropriate.

Home screen (cradle) Home screen (no cradle)

The home screen will display the following notification:

부재중 전화

missed call

착신 메시지

Incoming message

차량 고장

vehicle breakdown

telephone

GoPad will use the built-in Android phone API to support custom GoPad phone UX.

Incoming call notification

The agent must read the incoming call information (caller name if the caller is in Contacts, caller ID otherwise) aloud, mute the ringer if necessary, pause media playback, and then request user action. Users can respond in one of three ways:

음성으로 통화를 수락하거나 통화를 거부하고 그것을 음성 메일로 송신한다.

Accept or reject a call by voice and send it to voicemail.

온스크린 터치 버튼을 통해 통화를 수락/거부한다

Accept/reject calls via on-screen touch buttons

이전 트랙/통화 수락 또는 다음 트랙/통화 거부 버튼을 통해

via previous track/accept call or next track/reject call button

When the interaction is over, any paused media must be resumed.

From the touchscreen, the incoming call will be presented as follows:

Incoming call end call

outgoing call

An outgoing call may be initiated audibly by pressing the agent button to wake the agent, followed by saying a dial command along with a number or contact name.

If multiple numbers match the contact name, the agent creates a list of numbered options, sorted alphabetically followed by contact recency (ie, number last called last, last called last, etc). will say The user will then voice select an option number to call. The agent will place a call and update the recency value for that number.

A call may be initiated via the phone touchscreen via the following methods:

dial pad Favorites recent

contact

call status display

All call status information will be handled by the status bar at the top of the screen (see home screen above).

audio playback

media playback

A media player will be used to play Android-native media files through the following selection categories.

아티스트

artist

앨범

album

재생 리스트

playlist

artist album playlist

Quick selection of items in a long list will be facilitated by alphabetic skipping into subgroups of the list. For example:

The alphabetical list on the right border of the screen can be scrubbed with your fingertips for quick navigation.

The main control over the media player will be through the agent. When multiple matches are available in a given category, the agent will present an on-screen numbered list and allow the user to select a match by number. For example:

The media player will display artist/album/song information during playback, along with album art if available. The elapsed time and total time will be displayed. If space is available, the next track name may also be displayed.

Pressing the previous, next, and play/pause buttons on the cradle should affect playback, where appropriate.

The media player must play the media file in the default location on the device (ie, shared libraries from other media players must be accessible by the media player).

Because playlists are specific to the media player, the GoPad media player must import the playlists from the following media player application:

Google Play Music

Google Play Music

Android Music App

Android Music App

navigation

basic navigation

The navigation mechanism part will be handled via the built-in Android Google navigation application. The LW launcher and agent will provide Google Nav with a voice frontend that can be used to initiate navigation to a destination by selecting one of the following:

즐겨찾기

Favorites

최근 목적지

recent destination

주소록 연락처

address book contact

임의의 주소("333 West San Carlos, San Jose, California")

Any address ("333 West San Carlos, San Jose, California")

address book contact Favorites recent

The lightweight launcher will launch Google Nav and hand over the destination to it, which will then take over as the navigation provider.

The user will be able to return to the launcher (or other application) and put the navigation function in the background, without canceling the navigation function and returning to it later. Common ways to do this include pressing the agent button twice to return to the home screen or activating the agent and requesting a new function.

Reply to incoming text messages

The agent notifies the user by voice that a text message has been received, including the sender's name if the sender is in the address book, returns a call, or automated customization in the form of "I'm driving and I'll get back to you soon" You must give the user the option to send a boilerplate response.

Incoming text display

facebook activity leader

The GoPad Facebook Activity Reader will be part of the GoPad app. This feature will read Facebook wall posts to users and provide a large button for Likes.

Facebook activity

Incoming Facebook messages will also be read in much the same way as incoming text messages are read. Users can send boilerplate responses to the sender.

facebook message

news reader

The GoPad application will include integrated news reading in the manner of a newscaster. This will support the following features:

즐겨찾기

Favorites

최근

recent

뉴스 카테고리(즉, 기술, 스포츠, 기타)

News categories (i.e. technology, sports, other)

생일 미리 알림

birthday reminder

Favorites recent category

News stories will be presented in an easily parsed format with full screen text alternatives.

news story full screen text

Vehicle Condition/Efficiency

Starting the vehicle status feature will display the following information based on data from the BT OBD reader.

차량이 OBD를 통한 연료 레벨 측정을 지원하면, 연료 충전 이전의 현재 속도에서의 마일/킬로미터 및 시간의 범위가 필요하다(이 숫자는 보수적이어야만 한다). 이것은 최근 거동의 추후 결정되는 윈도우 상에서 계산되어야만 한다. OBD를 통해 연료 탱크 충전 상태를 제공하지 못하는 자동차에 대해 차선책이 매우 요망된다.

If the vehicle supports fuel level measurement via OBD, then a range of miles/km and hours at the current speed prior to refueling is required (this number should be conservative). It has to be calculated on a later determined window of recent behavior. A workaround is highly desirable for automobiles that do not provide fuel tank fill status via OBD.

이 여행의 MPG 및 모든 여행의 주행 평균

MPG for this trip and driving average for all trips

기본적으로 가속률/감속률을 측정하고 가속 페달 및 브레이크 페달을 부드럽게 다루라고 운전자를 그래픽으로 격려하는 순간 운전 효율 디스플레이 및 운전자가 시간에 따라 어떻게 했는지의 이력 디스플레이(아마도 자동차의 EPA 등급에 대해 플로팅됨?).

It basically measures the acceleration/deceleration rate and the moment it graphically encourages the driver to handle the accelerator and brake pedal gently, a driving efficiency display and a history display of what the driver has done over time (probably plotted against the car's EPA rating). ?).

경과 여행 시간, 여행 중의 효율, 사용된 연료 등을 비롯한 여행 통계.

Travel statistics including elapsed travel time, efficiency during travel, fuel used and more.

여행 통계를 0으로 설정하는 리셋 버튼

Reset button to set travel stats to zero

최적으로는 최근의 운전 이력에 기초하여 시간(일)으로 변환되는, (차량 데이터베이스로부터 유지관리 스케줄 정보에 기초한) 필요한 다가오는 유지관리

Upcoming maintenance required (based on maintenance schedule information from vehicle database), optimally converted to hours (days) based on recent driving history

고장 진단 오류 코드

Troubleshooting Error Codes

차량 보안(안전하지 않은 차량 거동, 임계 척도 등)

Vehicle security (unsafe vehicle behavior, critical measures, etc.)

In addition, voice alerts for the following high priority scenarios should interrupt any other currently displayed function and the device screen should switch to a vehicle status page with an error display.

연료 부족(문턱값 미정임, 최근 운전에 기초하여 변할 수 있다 - 상기 참조). 이것은 연료 레벨 읽기 능력에 의존한다(상기 참조).

Low fuel (threshold undetermined, may vary based on recent operation - see above). This depends on the ability to read the fuel level (see above).

즉각적인 운전자 행동을 요구하는 파국적 차량 오류(오류 코드 리스트는 미정임)(즉, 길 한쪽으로 차를 대고 엔진을 정지시키는 것이 안전하므로 곧바로 그렇게 함)

Catastrophic vehicle error that requires immediate driver action (error code list undecided) (i.e. it is safe to pull over to the side of the road and stop the engine, so do so immediately)

vehicle efficiency fault diagnosis vehicle security

vehicle travel information

third-party applications

The GoPad application will provide a quick and easy way to launch third-party Android applications that provide functionality that GoPad does not inherently provide. A third-party app launcher will provide a large touch target that makes it easy to launch applications while driving a vehicle.

The list of applications presented will be configured by the user, and the user will select from a list of all applications present on the device.

app launcher screen

setting

The settings area is where users configure the GoPad application according to their preferences. The final list of settings is undecided, but will include:

착신 문자 자동 응답 상용구

Auto Answer Incoming Text AutoText

착신 페이스북 메시지 자동 응답 상용구

Incoming Facebook Message Auto Answer AutoText

(페어링된 BT OBD2 어댑터들의 리스트로부터) BT OBD2 어댑터 선택

(from list of paired BT OBD2 adapters) BT OBD2 adapter selection

엔진 교체

engine replacement

엔진 유형(가솔린 또는 디젤)

Engine type (gasoline or diesel)

측정 단위(야드파운드법 또는 미터법)

Units of measurement (pound-pound or metric)

setting

vehicle identification

It is desirable to be able to identify multiple vehicles/cradles. Items tracked by vehicle/cradle include:

번호판

license plate

VIN(OBD가 VIN을 제공하지 않는 경우)

VIN (if OBD does not provide VIN)

크레이들 고유 ID

Cradle Unique ID

Bluetooth pairing

Cradle Pairing

It is desired that the launch manager be able to automatically pair the device to the cradle when the device is inserted into the cradle for the first time.

Pairing to an existing vehicle HFP or A2DP is not a feature for this release and no support is required.

data collection

The following data must be collected and stored in the system:

사용자 이름/이메일/ 전화 번호

Username/Email/Phone Number

자동차 정보

car information

o VIN number

o License plate number

운전 로그(모든 엔트리가 타임 스탬핑됨)

Driving log (all entries are timestamped)

o car

거리

Street

속도

speed

엔진 가동 시간

engine uptime

위치(들)

location(s)

내비게이션 목적지

navigation destination

o Applications

소프트웨어 미세 조정을 위해 모든 사용자 상호작용이 로깅되어야만 한다.

All user interactions must be logged for software fine-tuning.

오류 코드 로그

error code log

연비

Fuel efficiency

Data Collection Techniques

The easiest data collection method for each data or each type of data should be used. If data can be provided on behalf of the user, it should do so (eg, if fuel tank size can be determined based on the VIN number, do not ask the user for that information, but should do so).

The application must include camera capture of the license plate, and the application can parse the license plate number from the license plate and use it to determine the VIN number and any additional accessory data.

Data Anonymization

Certain types of collected data are of interest only as a whole - they have no value in a user-specific form -. Usability data of the application itself (ie patterns of button clicks, etc.), of the kind collected by services like Mixpanel, fall into this category. This data should be anonymized for data privacy reasons where practicable.

software update

The lightweight launcher requires an OTA update mechanism to allow new software versions to be pushed out to devices in the field.

Physical/Agent Control API

A simple software API is required that allows a third-party app developer to respond to agent commands as well as cradle physical controls while the device is in the cradle and his app is running in the foreground (or in the background, in some cases).

This API should be as simple as possible.

physical control

The physical control API should enable 3 command input (single press, double press, long press) for only the 3 buttons below.

이전 트랙

previous track

재생/일시중지

Play/Pause

다음 트랙

next track

Access to the agent button by third-party apps is not allowed.

agent

Third-party apps can register to receive specific voice commands through a simple API. Examples of instructions may include:

"다음 트랙"

"next track"

"이전 트랙"

"Previous track"

"일시중지"

"Pause"

Software UI flow

market opportunity

category ability market opportunity potential partner advantage Nav

경로 탐색 및 길 안내

POI 검색

Route navigation and directions

POI Search

Business Search Placement

Yelp

OpenTable

Michelin Guide







Oscaro
data
collection mobile service

message

newscaster

phone call

3rd party apps

Generates carrier traffic

ATT

T-Mobile

Boyges

Orange

sprint driving data

OBD2 Reader

GPS

travel data

Driving data for partners

insurance:
Ax
Allianz
AAA

rental car

car manufacturer music Third-party apps:

Pandora

Spotify

TuneIn

Account Registration Incentives

Sales of usage data

Pandora

Spotify

TuneIn

Appendix B: General presentation of the application

a. Application Description

Application Oscar is an application dedicated to the use of your favorite applications while driving.

Oscar allows users to use arbitrary functions or perform arbitrary actions in safe mode. The user experience that is created is key in its ability to achieve in any situation. You can use three media at any time:

터치 스크린(애플리케이션의 버튼 인터페이스)

Touch screen (button interface of the application)

물리적 버튼(OE의 경우에 자동차로부터 또는 애프터마켓에 대해 크레이들로부터)

Physical button (from car for OE or from cradle for aftermarket)

음성 명령

voice command

Key voice features include:

전화를 거는 것과 받는 것

making and receiving calls

메시지(문자, 메일 및 페이스북)를 보내는 것과 받는 것

Sending and receiving messages (text, mail and Facebook)

내비게이션을 정의하는 것: 원 샷.

What defines navigation: One shot.

뉴스를 읽는 것과 공유하는 것

reading and sharing the news

음악을 재생하는 것

playing music

The application is based on the following lemma:

음성 인식이 작동하지 않음 = 사용자의 문장을 제한함

Speech Recognition Not Working = Restricts User's Sentences

자연스러운 상호작용 = 사람의 대화와 가능한 한 가까운 것

Natural interaction = as close as possible to human conversation

에이전트의 피드백 길이를 제한함 = 짧은 문장

Limiting agent feedback length = short sentences

에이전트의 부정적 피드백을 제한함 = 아니다, 없다, 모른다,...

Limiting agent's negative feedback = no, no, don't know,...

사용자 반복을 제한함 = "다시 말해주세요"라고 요구하지 않음

Limit user repeats = don't ask "Tell me again"

These five lemmas are central to the creation of any user experience.

b. application architecture

go to do

c. User experience based on architecture

d. Key innovations detected

i. continuing the conversation

ii. Enable full screen agent

iii. automatic wake up

iv. list navigation

1. Voice Search Next, Previous, First, Last

2. Go Alphabetically

3. Play the list's voice

a. Voice Feedback Optimization

i. from the query

ii. from previous play

b. step-by-step play

4. Choice

a. by the number of the targeted item

b. By the partial content of the targeted item

5. Intelligent Choice

a. Learning from driver use

v. list filter

1. Alphabet filter

2. History filter

3. Frequency history

4. Continuous filter

vi. button pixel user experience

vii.

전화 모듈

phone module

e. Introduction

f. structure

g. Explanation

h. user experience

메시지 모듈

message module

내비게이션 모듈

navigation module

뉴스 모듈

News module

미디어 모듈

media module

Appendix C:

Voice and Connectivity Platform

Implementation overview

The automotive market is in one of the newest innovations that can be called the automotive market chaos, because it is in so many different kinds of fights. From electric engines to driverless vehicles, the digitization of automobiles continues, and automakers as a whole face one of the big challenges of the digital versus vehicle lifecycle.

But in the end, drivers are the voices who want to stop spending time alone on the road, and this time, if they can create new user experiences in a constrained environment, connect the car to the digital world and connect users to themselves in arbitrary contexts. If you can connect more with the applications of your favorites, it can turn into a useful and interesting time!

xBrainSoft is a scalable and flexible platform that works for all devices on the market in a continuous user experience in and out of the car , enabling hybrid mode and synchronized over-the- air update mode between the user's cloud and the car embedded platform. Manufactured in- car personal assistant products.

The best of each environment at the right time in any context! XXXXX is ready to meet the challenges of the short life cycle of the digital world without impacting the car life cycle!

Technical content

outline

The xBrainSoft voice and connectivity platform, in some embodiments, is an advanced platform adapted to establish a link between an onboard environment and an offboard environment.

Based on a hybrid, modular and agnostic architecture, the voice and connectivity platform provides its own "over the air" update mechanism between its embedded solution and offboard platform.

From embedded dialog management that has no connection to offboard extended semantic processing capabilities, the voice and connectivity platform enhances hybrid management with context synchronization enabling scenarios based on “lost and recover” vehicle connectivity.

Based on powerful, innovative and fully customizable natural language understanding technology, the speech and connectivity platform provides an immersive user experience without relying on any specific speech technology.

Its multi-channel capabilities enable interaction through multiple devices (vehicles, phones, tablets...) in an intrusive manner, sharing the same per-user context due to the fully synchronized mechanism.

The clustered server architecture of the voice and connectivity platform is scalable and thus responds to high loads and high consumption of services. It is based on industry standard technology and implements best practices based on communication security and end user privacy.

The Voice and Connectivity Platform also provides a full set of features and developer tools, integrated into the overall development environment, for designing complex voice user interaction flows.

added value

Below you will find some of the technological breakthroughs of xBrainSoft technology, a voice and connected environment consisting of a cloud platform and an embedded platform.

The following items are presented as a summary.

하이브리드 설계: "서버, 임베디드 자율적 동기화"

Hybrid Design: "Server, Embedded Autonomous Synchronization"

By design, the voice and connectivity platform provides assistants that operate both locally as well as remotely. This hybrid architecture of any assistant is based on a powerful mechanism for distributing processing, maintaining full context synchronization, and updating the user interface or even conversational understanding.

대화 흐름 생성을 위한 한 세트의 기능 도구들

A set of functional tools for creating conversation flows

From the very beginning, xBrainSoft has been dedicated to providing the best set of tools based on our technology to accelerate and improve the development of Assistant. It includes a conversational language manager, reusability of functional modules, automation of deployment or maintenance of any VPA, and a full developer environment that enhances portability on any client device.

ID 및 디바이스 연합 서비스(VCP-FS)

Identity and Device Federation Service (VCP-FS)

The voice and connectivity platform federation service is a service that associates a user ID with a device. VCP-FS handles social identities (Facebook, Twitter, Google+) and user-owned connected devices, enhancing the capabilities and capabilities provided by virtual personal assistants in an intrusive manner. The VCP federation service enhances the user experience by using the user's social network and even his habits.

자동차 애플리케이션 제품군 준비 완료(CPA)

Automotive Application Suite Ready (CPA)

On top of the voice and connectivity platform, xBrainSoft creates Car Personal Assistant (CPA) products , which are used by voice, touch screen or physical buttons, such as weather, stocks, news, TV shows, contacts, calendar, phone, etc. It provides a suite of automotive applications for

xBrainSoft also proposes an SDK for creating fully integrated applications that can access a car's CAN network, its GPS location, and various vehicle sensors, such as temperature, wiper status, engine status, and more.

오프보드 데이터 동기화기

Offboard Data Synchronizer

The voice and connectivity platform provides a global data synchronizer system. This mechanism solves the synchronization problems caused by the low capacity of traversal and mobile data connections. It provides a configurable abstraction of the synchronization system to allow the developer to focus on which data needs to be synchronized, not how it is done.

외부 API 자동 밸런서

External API Auto Balancer

Using an external API is a huge improvement for the scenario, but has side effects when the service becomes unavailable or when the client may want to use a particular service depending on a number of factors (price, user subscription...). have. To address this specific requirement, the voice and connectivity platform is highly configurable and designed to integrate third-party data providers as plug-ins (eg, managing API consumption by event handlers to connect on microbilling management systems).

Functions do not depend on a single external API, but rather on an internal provider that can manage many of the functions. In accordance with this architecture, VCP provides an autobalancing system that can be configured to meet XXXXX requirements.

사전 대응적 대화

Proactive Conversation

The voice and connectivity platform incorporates expert systems and mechanisms for initiating conversations with users without an initial request.

Together, they provide a set of tools to accomplish complex tasks, such as providing relevant information when user interests are available or proactively managing conversation frequency.

실제 컨텍스트 대화 이해

Understanding real-world context conversations

"Understanding Real Context Dialog" is a context-related multidimensional dialog flow with the following parameters: Context History, Conversation History, User History, User Profile, Localization, Current Context Domain, and the like.

This context-relevant approach to analyzing each conversation seeks to understand the best accuracy of any conversation flow, and to minimize the memory required to hold the assistant's knowledge, the continuity of the conversation after any kind of disconnection, and the transformation of any application. Many other positive effects are possible, such as the simplification of

공중을 통한 업데이트

update over the air

The VCP global data synchronization mechanism provides a way to update packages of any kind "over the air" between cloud platforms, embedded platforms and any connected devices throughout the life of the vehicle. When used internally to synchronize dialogs, UIs, logs, and snapshots between our online and embedded solutions, these "over the air" systems are designed to include third-party resources such as embedded TTS voices and embedded ASR dictionaries. can be expanded. Based on a version control system, dependency manager and highly compressed data transfer, it provides the best mechanism for a hybrid solution.

임의의 디바이스에 대한 서비스의 연속성

Continuity of service to any device

The voice and connectivity platform, through VCP federated services, can provide continuity of service without interruption through driver IDs and devices. Due to the increasing number of connected devices, driver attention accessible by the XXXXX virtual personal assistant exceeds the time spent in the car.

음성 및 음향 애그노스틱 통합

Voice and acoustic agnostic integration

The speech and connectivity platform does not rely on specific speech technology and can use a local speech engine or remote speech provider for both speech recognition and text-to-speech conversion. Local ones are encapsulated in VCP plugins and can be easily updated via the VCP data synchronization mechanism. Remote voice providers can manage VCPs directly from the cloud side.

Defining which speech technology the VPA uses for speech recognition and text-to-speech conversion is fully configurable for any conversation.

인공 지능 알고리즘

artificial intelligence algorithm

With a focus on getting results from constraint timing, voice and connectivity platforms take an agnostic approach to AI. This is why the best creative tools, when done with event-driven expert systems using the CLIPS engine, are built in an abstract way or integrated into the platform.

Our expertise lies in natural language, knowledge graphs, machine learning, social intelligence and general-purpose AI algorithms. Our set of tools is a link between currently available open source algorithms and the top frameworks to enable XXXXX to continue integrating recent evolutions in this scientific domain.

자연어 이해 애그노스틱 통합

Natural language understanding agnostic integration

In the same way as the strategy adopted for artificial intelligence algorithms, speech and connectivity platforms take an agnostic approach to integrate natural language processing modules. Based on our expertise in this field, this allows us to frequently update one of our core modules to optimize accurate understanding and ensure a unique user experience.

technical architecture

architecture

Architecture Description

The voice and connectivity platform is based on an asynchronous pipeline called "Smart Dispatcher". It is responsible for delivering message and user context across the platform and connected devices.

The VCP Federation service is responsible for managing user identities across platforms. It relies on third-party identity providers for numeric and social IDs such as My XXXXX, Facebook, Twitter, Google+ and Microsoft Live. It also has an internal mechanism that unites all of the user's connected devices, such as the user's car, phone, tablet, TV...

The Voice and Connected Cloud Platform provides an agnostic modular architecture with a "smart dispatcher" and full synchronization mechanism to work with VCP embedded solutions. Because ASR or TTS can be abstracted at the functional level using automatic ASR/TTS relay, VCP servers rely on third-party ASR/TTS providers such as Nuance, Google Voice, Telisma, CreaWave, etc.

The Voice and Connectivity Cloud Platform also includes all the technical blocks provided by the VCP Platform for conversation management, powered by semantic tools. When combined with event-driven expert systems, sensors, AI and proactive work, it provides a core stack used to develop applications.

Third-party data providers are included in an abstract way to support alternative scenarios or rule-based choices over user profile preferences or XXXXX business rules. This entry point allows the VCP to integrate all existing XXXXX connected services and make them available by the application development level.

The VCP embedded solution is the vehicle counterpart part of the VCP server. Being updatable "over the air", this embedded solution provides:

- UI delivery and management

- Onboard conversation management

- Context logging for "lost and recover" connection scenarios

- Snapshot manager for log or any other 3rd party sync

In vehicle architectures, embedded ASR and TTS providers may be included for onboard conversation management and are not provided as components of the voice and connectivity platform.

The VCP datastore is an Apache Hadoop-based infrastructure used to store and analyze all data inputs from voice and connectivity platforms. When used for machine learning or AI processing, the VCP data repository provides a mechanism to insert the analysis results into user profiles stored in the VCP federation service.

Technical details by field

voice and sound

설명

Explanation

The voice and sound lifecycle is one of the most important interactions that create the best user experience. This needs to be taken with a high level of care and high level of components to achieve the expected quality.

Achieving the expected quality can be achieved by combining a number of aspects:

o Top quality of microphone, filter, noise reduction, echo cancellation...

o Integration of multiple ASR / TTS providers (Nuance, Google, Telisma, Microsoft Speech Server...)

o What you can switch between those providers for your use case:

- ASR: onboard, offboard streaming or offboard relay

- TTS: onboard, offboard, emotional content, mixed continuity mode

o ASR correction management based on user conversation context

o Manage “real conversations”

At xBrainSoft, we grouped those aspects into two categories:

o From voice capture to ASR process termination

o From the ASR process to natural language processing through natural language understanding

As ASR providers or hardware microphone manufacturing are not our business scope, we have taken a technically agnostic approach to voice management, in order to be able to integrate and communicate with any kind of ASR/TTS engine. . Our experiences and projects have led us to a high level of expertise in those techniques in constrained environments, such as was done during VPA prototyping with Nuance data integration.

This type of architecture allows our partners to quickly create robust dialog scenarios in many languages for any type of ASR or TTS. This also makes it possible to easily upgrade any component to improve the user experience.

The second category is managed by different levels of software filters based on the user conversation context. Since the dialogue is not just a set of interactive sentences, we developed different filters based on “understanding real-world context dialogue” in speech and connectivity platforms. Real Context Dialog Understanding is a context-related multidimensional dialog flow with the following parameters: Context History, Conversation History, User History, Localization, Current Context Domain, etc.

As enhanced by our VCP semantics tools, we achieve a deep semantic understanding of user input.

This approach has enabled the "news and voice search" application (newscaster) to be reduced from 1.2 million language pattern entry points to less than 100 while maintaining the same exact semantics in terms of end-user conversation flow.

This new approach to the explanation of patterns has many positive aspects:

o Simplify disambiguation scenarios, erroneous keywords or incomplete entity extraction

o Simplifies debugging of patterns and enables creation of automated tools

o Simplifies pattern correction and maintenance “on the fly”

o Minimize the memory resources to load the pattern dictionary.

o Minimize the effort of translating random conversations for language adaptation

A fully hybrid and "over the air" updatable system, VCP component "Online or Embedded Conversation Manager" aims to provide the best solution for managing embedded conversations when the vehicle loses its connection to a fully online conversational experience. .

As such, when the best material is required, the voice and connectivity platform ensures that it is the most efficient at creating the best user experience ever intended.

On the one hand, xBrainSoft continues to push the limits of user experience by adding many research aspects such as sentiment analysis in conversation flow, social and educational behavior level in user context inferred from social and conversation flow, or prosody management based on VoiceXML standard. is pushing it

혁신적 특징

innovative features

o ASR/TTS providers' agnostic approach

o Offboard ASR/TTS relay capability

o Onboard conversation management

o Offboard conversation management

o Hybrid conversation management by update “over the air”

o VCP Semantic Tools

o Integrated development environment for conversation management

예시적인 요소

exemplary elements

o High quality microphone and sound capture

o Speech signal processing including noise reduction and echo cancellation

o Microphone audio API with automatic blank detection

o One or more speech recognition engines for onboard and offboard

o One or more text-to-speech engines for onboard and offboard

o VCP Embedded Solutions

o VCP Server

예시적인 제휴 파트너

Exemplary Affiliate Partners

Sound Capture: Parrott or Nuance

Speech signal processing: Parrott or Nuance

ASR: Google, Nuance or Telisma

TTS: Nuance, Telisma or CreaWave

Hybrid structure and behavior

설명

Explanation

A connected cloud-based personal assistant that can be autonomous when a data connection is not available. The goal is to always be able to provide users with quick and accurate answers.

A VCP embedded solution consists of a hybrid assistant that runs on an embedded device, such as an automobile, and connects to a server-side counterpart. Any user request is handled directly by the embedded assistant, which determines whether or not that user request should be forwarded to the server, based on criteria such as connectivity. In this way, all user requests can be handled either locally or remotely. Offboard level capabilities can be easily tuned to improve performance and user experience.

Like voice and connectivity platforms, VCP embedded solutions provide advanced natural language processing and understanding capabilities to handle user requests without requiring a data connection. This ensures that the VPA can quickly understand any user request locally and, if necessary, respond directly to the user, and asynchronously fetch a heavy computational response from the server. In the absence of a connection, if external data is needed to adequately answer the user (eg, a weather request), the response is adapted to notify the user that the user's request cannot be fulfilled. Depending on the scenario, the VPA may queue user requests so that they can be forwarded to the server as soon as the connection is restored.

The voice and connectivity platform also provides full context synchronization between embedded agents and servers, allowing data to be shared between them instead of being separated. A resynchronization is performed whenever a connection problem occurs to ensure that the data is always up-to-date.

VCP embedded solutions consist of plug-ins that can be easily updated or exchanged through an "over the air" process. Voice, IA, dialogue understanding, data processing and user interfaces are some of those upgradeable modules.

The VCP embedded solution also consists of a set of scripts, part of the AI, that process the response. To ensure consistency of response, whatever the level of connection, these scripts are synchronized between the server and the embedded agent.

혁신적 특징

innovative features

o User Interface Manager

o Local interface synchronized with the server

o Embedded dialog manager

- Pure Embedded Scenarios

- Onboard/Offboard Hybrid Scenarios

- Pure Offboard Scenarios

Always responding to user requests with or without an internet connection

Context Synchronization for Loss of Connection Use Cases

예시적인 요소

exemplary elements

Linux platform available on car computer system

성능

Performance

Efficient, performance-oriented programming language (C++)

High compression of exchanged data to optimize bandwidth and response time

The VCP embedded solution was compiled and tested on Raspberry PI Model A:

o CPU: 700 MHz low-power ARM1176JZ-F application processor

o RAM: 256MB SDRAM

A.I

설명

Explanation

Artificial intelligence is a large field that encompasses many areas of specialization, such as:

o Reasoning, analogy, and problem solving

o Discover the knowledge graph

o Planning and acting by an event-based expert system

o Natural Language Processing and Semantic Search

o Machine Learning, Map Reduce, Deep Learning

o Social intelligence, sentiment analysis, social behavior

o Other uses not yet discovered

xBrainSoft recognizes the enormous potential range and remains humble in the face of the current challenges of the state of science.

With a focus on getting results from constraint timing, voice and connectivity platforms take an agnostic approach to AI. This is why we create the best creative tools in an abstract way or integrate them into our platform, when done as an event-driven expert system using the unique CLIPS engine.

Our expertise lies in natural language, knowledge graphs, machine learning, social intelligence and general-purpose AI algorithms.

A key characteristic of our set of tools is that they bridge the gap between top-level frameworks and currently available open source algorithms.

Thereby, xBrainSoft can achieve 100% of the expected scenarios of the VPA project, because it is possible to replace our modules with any other more valuable ones available in the market.

This is why xBrainSoft is also collaborating with partners such as Kyron (Silicon Valley, AI Applied to Healthcare, Big Data and Machine Learning), Visteon or Spirops to expand the possibilities of AI available through our platform. because there is

혁신적 특징

innovative features

Ability to provide data to external AI modules in an anonymized manner. A user or session is represented as a random unique number so that it can operate at an appropriate level if the external system cannot correlate that information to the real user.

An agnostic approach to embedding AI into voice and connectivity platforms (VCPs) using xBrainSoft or external AI tools

Bridge to VCP federation service provided by VCP to get data back from AI tools and enhance user profile for better user context management

예시적인 요소

exemplary elements

o VCP data storage based on Apache Hadoop

o VCP event-based expert system

o VCP Federation Service

Offboard Platforms and Services

설명

Explanation

In order to enrich the services provided to users in their cars, the offboard platform brings a high level of connected functions due to its high availability and powerful components. Users are set at the center of a comprehensive and intelligent ecosystem focused on car services. Offboard platforms are also entry points that bring the ability to blend automotive and connected services.

The offboard platform has high availability to support both the brand's cars and the user's connected devices. This can evolve over time as it handles more and more users and handles load fluctuations.

To address all of those challenges, voice and connectivity platforms provide a clustered architecture that can be deployed “in the cloud” or on premises. All clustered nodes know each other, enabling a cross-nodes connected devices scenario to maintain service continuity through the clustered architecture.

The voice and connectivity platform offers the ability to consume third-party data services, from technical data services to user information, via their social accounts and devices. All of that information is useful for generating "appropriate" and intelligent scenarios.

The range of functions and services is wide and will evolve over time due to technological advances. The architecture of the platform must provide new services/functions without affecting existing functions based on its modular architecture.

혁신적 특징

innovative features

o In-cloud or on-premises hosting

o Ready to go clustered architecture deployment for high availability and load fluctuations

o Cross-device capabilities through clustered architecture

예시적인 요소

exemplary elements

o VCP Server

o Third-party data providers

성능

Performance

5k concurrent connection objects per server (car), the prototype will implement a set of 3 servers to ensure a high-level SLA, and we will propose 10k concurrent connection objects earlier.

Offboard frameworks and general security

설명

Explanation

As a third-party data service provider, the XXXXX SIG can be used by voice and connectivity platforms, in addition to our currently implemented providers. Due to the high-level abstraction, we can implement different third-party data service providers and integrate them throughout the project lifecycle without updating some of the functionality of the VPA.

Voice and connectivity platforms provide a means to implement alternative scenarios to ensure high availability of data through external providers. For example, multiple weather data providers to replace when a major provider is not available.

The voice and connectivity platform also provides implementation of its expert systems for provider qualifications. Based on business rules, the system helps manage billing optimization. The system can be used at different levels, such as User 1 based on subscription fees or Platform 1 based on supplier transaction agreements.

Because the voice and connectivity platform can be exposed by a full set of HTTP APIs, the system can be easily integrated into any kind of machine-to-machine network.

When it comes to communications and authentication, voice and connectivity platforms provide the latest practices used in the Internet industry. From securing all communications with SSL certificates to the Challenge Handshake Authentication Protocol (CHAP), the voice and connectivity platform ensures a high level of security when it comes to end-user privacy.

Security and user privacy are also considered during VCP federated service identity association, as end user logins and passwords never pass through the voice and connectivity platform. All of these systems are based, in the example, on token-based authentication provided by the identity provider: in the case of a Facebook account, the end user is authenticated on the Facebook server, which verifies the end user identity and provides an authentication token back to us. .

The way the VCP embedded solution is built avoids reliability or safety issues because it relies on basic existing functions provided by the integrator. In our technical proposal, the VPA cannot send commands directly to the car, but the VPA can send commands to the underlying system, which provides reliability and safety issues.

혁신적 특징

innovative features

o Modular architecture that enables full integration of XXXXX Connectivity Services APIs

o Our XXXXX can be implemented as the default identity provider for the VCP federation service, helping to give users peace of mind when linking their social IDs.

o High-level security to protect end-user privacy

예시적인 요소

exemplary elements

o Secure infrastructure for vehicle connectivity, such as M2M networks

o Token-based authentication API implementing the VCP federated service identity provider

Context and history awareness

설명

Explanation

Efficient context management is essential for personalizing conversations, assistant behaviors or functions. When implemented at the engine level, user context can be accessed by any component of the voice and connectivity platform to enable an enhanced personalized experience.

Because it is extensible by any data source - vehicle data (CAN, GPS...), social profiles, external systems (weather, traffic...), user interactions... etc. - the user context is also Overused by our event-based expert system to create reactive use cases.

Because it is shared across onboard and offboard, the voice and connectivity platform pays attention to context resynchronization between the two environments.

When it comes to historical recognition, voice and connectivity platforms provide a complete solution for aggregating, storing and analyzing data. The data may be from any source as described above.

When analyzed, the data results are used to enrich the user profile to help deliver a personalized experience.

혁신적 특징

innovative features

Because it is integrated as an engine feature, user context management is transversal in voice and connectivity platforms. It can be accessed from any module, dialog, task or rule within the system. It can also be shared across devices by implementation of the VCP federation service.

The voice and connectivity platform provides a full context resynchronization system between onboard and offboard to handle connectivity issues such as driving through tunnels.

Based on the Apache Hadoop stack and tools, the VCP datastore provides an infrastructure ready to perform machine learning goals such as user behaviorism, habit learning, and any other related machine learning classification or recommendation tasks.

예시적인 요소

exemplary elements

o VCP data store

o Define Hadoop infrastructure based on requirements

proactive

설명

Explanation

Proactiveness is one of the means of creating smarter applications for end users.

The VC platform provides two different levels of proactive responsiveness management:

o Background Worker : A complete background work system that can reconnect to the main pipeline and interact with user sessions or use alternative notification tools.

o Event-based expert system : a fully integrated business rules engine capable of reacting to external sensors and user context

When combined with the VCP federation service, it exploits the ability to be proactive beyond devices.

혁신적 특징

innovative features

o Event-based expert system that proactively responds to contextual items in real time

o Use of VCP federation services to enable a cross devices proactive experience

o Provide implementations of major notification providers for proactive alternative use cases (Google, Apple, Microsoft ...)

o Based on functional perspective, the level of proactive adjustment can be exposed as user settings

예시적인 요소

exemplary elements

VCP Federation Service for Device Information

Devices that support the notification process for alternate use cases

Overall upgradeability

설명

Explanation

Overall upgradeability is a critical process with respect to the automotive industry. Since cars don't go to car dealerships that often, the overall solution should provide a full mechanism of "over the air" updates.

The voice and connectivity platform is already implementing its "over the air" mechanism with their VCP embedded solutions to synchronize dialog boxes and user interfaces.

Based on the factory architecture, this “over the air” process can be extended to manage any kind of data between connected devices and voice and connected platforms.

혁신적 특징

innovative features

o Extensible “over the air” mechanisms that include versioning support, dependency resolution, and communication compression

o The VCP server is based on a modular architecture that makes it possible to add or remove (new) modules over the life of the vehicle.

o The VCP embedded solution is based on a plug-in architecture that makes it possible to add new vehicle functions or new interoperability functions to access messages.

예시적인 요소

exemplary elements

o Internet connection (depending on hardware and type of connection)

Internal & External Continuity

설명

Explanation

Device continuity means, via voice and connectivity platforms, drivers can connect to their virtual personal assistants in the car as well as on the street or outside such as at home. Drivers can use the service wherever they want.

This capability enables XXXXX to expand the scope of its relationships with its customers both within and outside the automobile. A brand extends its opportunities to serve and engage beyond its traditional sphere. As such, this opens the door for a greater number of potential business partnerships with third-party carriers that can deliver competitive APIs or services.

Based on the VCP federation service, VPA can be fully integrated into the end-user ecosystem. From the user's car, the user's multiple devices, to the user's number and social ID, all inputs to the ecosystem can enhance the user's intrusive experience.

혁신적 특징

innovative features

The voice and connectivity platform provides its services via a standard secure protocol (HTTPS) that can be accessed from all recognized devices. From an end-to-end perspective, the Voice and Connectivity Platform provides frameworks and tools for all major device platforms such as Android, iOS, Windows + Windows Phone and Embedded.

The VCP Federation service aggregates a user's numeric ID and device to provide the user with the best connected, intrusive experience. For example, the VCP may start a scenario on the user's phone, then in his car and end the scenario on another device.

VCP User Interface Administrators can download, store and run VCP web objects on any device that provides a web browser API. With this in mind, the user interface and logic of applications on connected devices can be cross platform and easily updateable “over the air”. The VCP user interface administrator can also apply different templates/logics for a particular platform, region or language.

예시적인 요소

exemplary elements

The VCP federation service is at the heart of service continuity.

Due to the heterogeneity of the connected devices (platform, size, hardware, usage ...), the scenario must be tailored to the best fit for the target device. For example, the device may not have a microphone, which will not be compatible with the voice user interface, and physical interaction must be used.

cultural and geographic context

설명

Explanation

Due to the high internationalization of XXXXX, VPAs may become suitable for users from a cultural or geographic point of view. This means the translation of all scripts and interfaces provided to the user, the configuration of ASR and TTS providers, and, if necessary, modification of the behavior of some scenarios.

혁신적 특징

innovative features

Based on a fully modular architecture, voice and connectivity platform modules can be plugged in according to internationalization settings. This makes it possible to manage different service delivery or characteristics according to region.

The Voice and Connectivity Platform provides a complete abstraction of ASR/TTS provider relays, which can be based on regional deployments or user settings. This enables a unified entry point for speech recognition and speech synthesis for automobiles or connected devices that is responsible for the separation of concerns between speech acquisition/playback providers and ASR/TTS providers.

The VCP dialog manager and VCP semantic tools provide a high-level abstraction that enables extensibility to new languages without affecting functional implementation.

예시적인 요소

exemplary elements

o External third-party data providers that support translations via their own APIs

o ASR/TTS provider(s) for selected language(s)

o In the example, we define the end-user social ID for the VCP federation service: Weibo instead of Twitter for China.

o Tailoring use cases and VPA behavior to end-user culture and geography

Appendix D: "directly and "suboptimal scenario process"

A general approach is described in which we will find our added value in relation to Siri, Google Now, Nuance, ... or other products such as any type of personal assistant according to various embodiments.

Legend:

VCP = 음성 및 연결 플랫폼

VCP = Voice and Connectivity Platform

ASR = 자동 음성 인식

ASR = automatic speech recognition

TTS = 텍스트-음성 변환

TTS = text-to-speech

TUI = 터치 사용자 상호작용

TUI = touch user interaction

VUI = 음성 사용자 상호작용

VUI = Voice user interaction

NLU =　자연어 이해

NLU = natural language understanding

VCPs are synchronous and asynchronous. This means that each action, event can be executed immediately or long after a request from the user. I can ask the agent to send me my sales report for a long job or long job on the 1st (asynchronously) of every month, i can ask for today's weather (using direct context) and immediately after his reply i can ask for tomorrow's weather you can ask about

The description of the life cycle (see Fig. 7) starts from the bottom left and goes to the top right.

life cycle

ASR Engine:

ASR(Automatic Speech Recognition) 이전에, 우리는 3가지 방법으로 ASR을 활성화시킬 수 있다:

Before Automatic Speech Recognition (ASR), we can activate ASR in 3 ways:

o ASR Auto Wakeup Words: Any keyword can be used to wake up the application and start ASR (Angie, Sam, ADA, ... etc.)

o ASR Proactive Activation: Rely on internal or external events

타이머: 매일 타이머에 기초한 자동 웨이크업

Timer: Automatic wake-up based on a timer every day

내부 이벤트: 디바이스 컴포넌트(GPS, 가속기, ...) 또는 애플리케이션의 임의의 기능 또는 모듈로부터 임의의 내부 이벤트

Internal Events: Any internal events from any function or module of the device component (GPS, accelerator, ...) or application.

당신이 집에 위치해 있다는 것을 검출하고 당신이 무언가를 할 때 ASR(컨텍스트 프롬프트를 갖는 TTS)을 시작할 수 있다

It detects that you are located at home and can initiate ASR (TTS with Context Prompt) when you do something.

내가 내 자동차에 위치해 있을 때(내가 전원 및 OBD를 검출하기 때문에), 나는 음악을 시작하고 내비게이션을 시작하라고 당신에게 제안할 수 있다.

When I am located in my car (since I detect power and OBD), I can suggest you to start music and start navigation.

당신이 캘린더에 새로운 약속을 가지고 있을 때, 에이전트가 자동으로 시작되고 (자동차가 필요로 하는 경우) 내비게이션이 다음 회의로 가기를 원하는지를 당신에게 질문할 수 있다.

When you have a new appointment in your calendar, the agent starts automatically (if your car needs it) and the navigation can ask you if you want to go to the next meeting.

외부 이벤트: 우리는 ASR/TTS를 활성화시키기 위해 데이터베이스 또는 제3 API로부터 임의의 외부 이벤트를 검출한다.

External Events: We detect any external events from a database or 3rd party API to activate ASR/TTS.

당신이 목적지 근방에 도착할 때, 시스템은 당신이 자동차를 주차할 수 있을 때를 당신에게 알려주기 위해 외부 주차 이용가능성 API를 살펴볼 수 있다.

When you arrive near your destination, the system may look at the External Parking Availability API to inform you when you can park your car.

당신이 교통 체증에 있을 때, 시스템은 자동차에 의한 방향 수정뿐만 아니라 목적지에 어떻게 갈 것인지를 변경하고 당신에게 당신의 자동차를 주차하고 기차를 타라고 제안하는 기회를 평가할 수 있다.

When you are in a traffic jam, the system can evaluate opportunities to change how you will get to your destination, as well as change directions by the car and offer you to park your car and take the train.

o ASR Push Button: Activation of an agent from a simple click (push) on a virtual button (screen) or a physical button (from a cradle or wheel button)

ASR의 활성화(음성 입력)

Activation of ASR (voice input)

ASR-NLU 전처리 = 애플리케이션의 컨텍스트에 기초하여, 우리는 (그의 신뢰도로) 문장을 취하고 자연어 이해 엔진에 보내기 이전에 그것을 재가공할 수 있다

ASR-NLU preprocessing = based on the context of the application, we can (with its confidence) take a sentence and reprocess it before sending it to the natural language understanding engine

o Knowing that we are in the context of the module to call, we can extract or change any word in the sentence, before sending it to the NLU engine.

o In French, when you speak:

"donne-moi l'information technologique" => ASR은 (사용자 의도를 완전히 벗어나) "Benoit la formation technologique"를 우리에게 보낼 수 있다.

"donne-moi l'information technologique"=> ASR can send us "Benoit la formation technologique" (completely out of user intent).

우리는 단어들을 고칠 수 있다: 'Benoit'를 'Donne-moi'로 그리고 'formation'를 'information'로

We can fix the words: 'Benoit' to 'Donne-moi' and 'formation' to 'information'

전처리 후에, 문장은 문장이 NLU에 의해 이해될 기회를 완전히 확장하고 사용자에 대한 행동을 생성할 것이다.

After preprocessing, the sentence will fully expand the chances that the sentence will be understood by the NLU and generate behavior for the user.

NLU Engine:

특정의 모듈을 시작하려는 사용자의 의도의 검출, 각각의 검출은 이하의 다음 장에서 설명하는 바와 같이 애플리케이션의 컨텍스트에서 기능한다.

Detection of the user's intent to start a particular module, each detection functioning in the context of an application, as described in the next chapter below.

o sample

Gregory에게 전화해 = 전화 모듈

Call Gregory = Phone Module

Bastien에게 문자를 보내 = 메시지 모듈

Text Bastien = Messages module

o keywords = keywords for direct access from modules

전화 = 전화에의 액세스를 제공함

Phone = Provides access to the phone

내비게이션 = 내비게이션에의 액세스를 제공함

Navigation = Provides access to navigation

o Shortcut = A statement that the user can speak from anywhere in the application, only for the main tasks, such as those listed in the schema.

모듈(의도)로부터 행동(기능)의 검출

Detection of actions (functions) from modules (intents)

o sample

전화를 거는 것 = Gregory Renard에게 전화를 거는 행동

Calling = Calling Gregory Renard

이 문장은 모듈, 행동 및 엔티티(사람 = Gregory Renard)를 검출하는 것을 가능하게 한다

This statement makes it possible to detect modules, actions and entities (person = Gregory Renard).

디폴트 모듈 리스트 = 우리는 애플리케이션이 할 수 있는 것과 할 수 없는 것을 정확히 알고 있기 때문에, 우리는 사용자가 애플리케이션이 할 수 없는 무언가를 하려고 시도하고 있다는 것을 검출할 수 있거나 우리가 ASR로부터 좋지 않은 응답을 받았을 수 있다. 이 경우에, 우리는 (전형적으로 Siri와 Google Now가 사용자를 웹 검색으로 밀어내는 경우에) 사용자의 의도의 의미를 검출하려고 시도하기 위해 디폴트 모듈을 활성화시킬 수 있다.

Default module list = Since we know exactly what the application can and cannot do, we can detect that the user is trying to do something that the application cannot do or we may have received a bad response from the ASR. can In this case, we can activate the default module to try to detect the meaning of the user's intent (typically when Siri and Google Now push the user to web searches).

o Proposing to the user a list of modules available in the application (not limited, we can expand the list of modules from any type of application if necessary)

o If the user repeats something wrong or if speech recognition doesn't work = the system suggests switching from sentence speech recognition to number recognition

사용자가 시스템이 인식하지 못하는 무언가를 말한 경우, 시스템은 = "어떤 애플리케이션을 시작하길 원하세요"라고 말할 것이고 + 애플리케이션의 리스트를 열 것이다.

If the user said something the system doesn't recognize, the system will say = "What application do you want to start" + open a list of applications.

사용자가 시스템이 인식하지 못하는 무언가를 다시 말한 경우, 시스템은 = "당신이 원하는 애플리케이션의 번호는 무엇입니까"라고 말할 것이다(우리는 연락처, 주소, 앨범, 아티스트, 뉴스 카테고리, 메시지와 같은 임의의 유형의 리스트에서 이 작업 흐름을 사용한다)

If the user retells something the system doesn't recognize, the system will say = "what's the number of the application you want" (we can use any type like contact, address, album, artist, news category, message) use this workflow from the list in

o The user makes a choice

시스템은 모듈에 대한 디폴트 항목 리스트를 보여주고 모듈에서 이용가능한 기능들을 (음성으로 그리고/또는 시각적으로) 제안한다. 사용자는, 이 경우에, 달성하기 위한 지침을 사용하여 선택을 할 수 있다.

The system shows a list of default items for the module and suggests (voice and/or visual) the functions available in the module. The user, in this case, can make a choice using the guidelines to accomplish.

o The list could be:

필터: Malvoisin에게 전화하는 것 => Celine을 바탕으로 필터링하는 것 = 연락처 리스트에 대한 Celine Malvoisin의 리스트를 보여주는 것

Filter: Calling Malvoisin => Filtering by Celine = Showing Celine Malvoisin's list to your contact list

글자에 의해 필터링하는 것: 임의의 리스트에 기초하여, 당신은 한 글자씩 필터를 생성할 수 있다.

Filtering by character: Based on an arbitrary list, you can create a filter by character.

사용자가: '글자 M, 글자 A, 글자 L을 바탕으로 필터링해'라고 말할 수 있다(이것은 발음할 수 없는 연락처에의 액세스를 가능하게 한다)

User can say: 'Filter based on letter M, letter A, letter L' (this allows access to non-pronounceable contacts)

항목 라벨에 있는 임의의 단어를 글자 필터링에 의해 필터링하는 것

Filtering any word in the item label by letter filtering

글자 탐색에 의해 필터링하는 것: 임의의 리스트에 기초하여, 사용자는 "글자 V로 가"라고 말할 수 있다.

Filtering by letter search: Based on an arbitrary list, the user can say "Go to letter V".

에이전트는 곧바로 글자 V로 시작하는 모든 연락처를 보여줄 것이다.

The agent will immediately show you all contacts starting with the letter V.

탐색: 사용자가 하기와 같이 리스트를 탐색할 수 있다

Navigation: The user can navigate the list as follows:

다음/이전 = 현재 리스트에서의 다음 또는 이전 항목 리스트를 보여준다

Next/Previous = Shows the list of next or previous items in the current list

처음 = 리스트에서의 첫 번째 항목을 보여준다

first = show the first item in the list

끝 = 리스트에서의 마지막 항목을 보여준다

end = show the last item in the list

o The list can be read at any time:

항목 리스트의 임의의 화면에서, 사용자는 리스트를 읽어주라고 요청할 수 있다

On any screen of the list of items, the user can request that the list be read.

리스트는 다음과 같이 읽혀질 것이다

The list will be read as

각각의 항목이 읽혀지고 이어서 번호에 의해 사용자가 항목 번호를 기억하는 데 도움을 준다.

Each item is read and subsequently numbered to help the user memorize the item number.

이전 항목 연락처가 우리가 이미 알고 있는 부분을 통합하지 않는 경우 각각의 항목의 내용이 읽혀질 것이다.

Each item's content will be read if the previous item contact doesn't incorporate what we already know.

우리가 전화 번호 리스트에 5개의 연락처 Malvoisin(Celine에 대해 3개의 상이한 유형의 전화, Luc에 대해 1개, 그리고 Gregoire에 대해 1개)를 갖는다고 상상해보자

Imagine we have 5 contacts Malvoisin (3 different types of phones for Celine, 1 for Luc, and 1 for Gregoire) in our list of phone numbers.

에이전트는 말할 것이다: 　(에이전트가 말하고 있을 때 우리는 어떤 내용도 반복하지 않는다)

The agent will say: (we do not repeat anything when the agent is speaking)

Celine 모바일 US는 번호 1이다(Malvoisin이 없는데 그 이유는 그것이 내 요청이었고 내가 읽고 있을 때 Malvoisin 연락처를 원한다는 것을 내가 알고 있기 때문이다.

Celine Mobile US is number 1 (I don't have Malvoisin because I know that was my request and I want Malvoisin contacts when I'm reading.

집은 번호 2이다

house is number two

사무실은 번호 3이다

office is number 3

Luc 모바일은 번호 4이다.

Luc Mobile is number 4.

Gregoire 집은 번호 5이다

Gregoire house is number 5

사용자에 의한 항목 선택　

Item selection by user

o Select item number = Allows the user to select an item from the number preceding the item (we only function for numbers 1 to 5)

o Select item content = Allows the user to select an item from the item's label (eg celine)

튜플 = 모듈, 기능 및 엔티티(항목 선택)의 검출 후에

tuple = after detection of modules, functions and entities (item selection)

o The system can execute processing for two types of functions

지식 유형 = 사용자에게 답변을 제공하는 데이터 지식(QA, 카탈로그, 위키 백과, ...)에의 액세스

Knowledge Type = Access to data knowledge (QA, Catalog, Wikipedia, ...) that provides answers to users

행동 유형 = 외부/내부 APi들에 액세스를 관리할 필요가 있음

Behavior Type = Need to manage access to external/internal APIs

이하에서 기술되는 NLU 처리의 결과에 기초하여, 시스템은 2개의 동기 요소를 발생시킨다:

Based on the results of the NLU processing described below, the system generates two synchronization factors:

o TUI = Touch User Interaction (designing the screen to the user as any type of application)

o VUI = Voice User Interaction (voice feedback with the ability to ask the user for additional information or details or ask other questions)

o VUI and TUI are fully synchronous, you can go to the next level of functional workflow by touch or voice, both are synchronous.

당신이 항목을 선택하기 위해 화면 상에서 클릭하면, 당신은 다음 단계로 갈 것이고, 에이전트는 애플리케이션에서의 당신의 컨텍스트 위치를 알고 있다.

When you click on the screen to select an item, you go to the next step, and the agent knows your context location in the application.

이 컨텍스트 위치는 음성이 시각과 동기될 수 있게 한다.

This contextual location allows speech to be synchronized with time.

현재 작업 흐름에 기초하여, 에이전트는 사용자의 현재 의도를 완성하는 데 추가 정보가 필요한지를 검출하고 (TTS에 문장 피드백을 보낸 후에) ASR의 새로운 시작과 함께 그것을 요청할 수 있다

Based on the current workflow, the agent can detect if additional information is needed to complete the user's current intent (after sending sentence feedback to the TTS) and request it with a fresh start of the ASR.

o User: What's on TV tonight?

o System: on which channel (because your intent was detected by TV = module and tonight = some channel prime time tonight in action)>

o The system understands that it is missing a variable that completes the action and asks about it.

o Users: on channel 1

o System: there is a prime time on channel 1.... blablabla

o User: and Channel 2 (in this case, we use the context to know what the current intent was and the last action from the user = TV/Providing the prime time program of the night)

o System: There is a prime time on channel 2.... bliblibli

o ...and the system can continue into this context without restrictions, we call this workflow "direct context ".

이전의 지점(의도/컨텍스트의 관리)에 기초하여, 우리는 상이한 유형들의 컨텍스트를 사용할 수 있다.

Based on the previous point (management of intent/context), we can use different types of context.

o See the explanations at the points below.

Time Context Matrix Dependencies.

Before going to the types of context, we need to define the context created in VCP from xBrainSoft.

The context is (defined as the current context)

3D 저장 매트릭스로 기능함:

Functions as a 3D storage matrix:

o Dimension 1: Current Module (Phone Module)

o Dimension 2: Current behavior (action to make a call from the phone module)

o Dimension 3: Current screen (step of action, e.g. selection of a contact for the action to place a call on a module phone)

여기서 당신은 임의의 저장 사례(컨텍스트 필드)에서 임의의 레벨에서 저장 항목을 확장하는 능력을 갖는 최소 3개의 항목을 갖는 튜플(객체 유형, ID '이름' 및 값)에 의해 임의의 유형의 정보를 저장할 수 있다.

Here you can store any type of information by a tuple (object type, id 'name' and value) with at least 3 items with the ability to expand the storage item at any level in any storage instance (context field). can be saved

o Variables of any type (int, string, Date, ...)

o Serializable objects of any type (Car Type, User Type, ...)

이력을 사용할 능력을 가짐 = 4D 저장 매트릭스(컨텍스트는 시간 변수에 의해 진행 중인 작업이다)

Has ability to use history = 4D storage matrix (context is work in progress by time variable)

o Each time state is stored for user sessions for short and medium durations

o Each time state can be stored in a file or database for a long period of time

Context relates to the user's functional current workflow to provide the possibility to generate intention learning for medium and long term periods.

We can have two context categories:

애플리케이션 컨텍스트 = 단기간, 중간 기간 또는 장기간 동안 많은 사용자들(애플리케이션의 모든 사용자들 또는 애플리케이션의 사용자들 중 일부)에 의한 전반적 컨텍스트 공유

Application context = overall context sharing by many users (all users of the application or some of the users of the application) for a short, medium or long term

세션 컨텍스트 = 고유의 사용자에 대한 컨텍스트.

Session context = Context for your own user.

Type of context:

직접 컨텍스트: 이상의 설명을 참조하십시오.

Direct context : see description above.

간접 컨텍스트(임시 컨텍스트) = (직접 컨텍스트를 사용하거나 사용하지 않는) 사용자와 에이전트 사이의 임의의 질문/대답 이후에, 사용자는 자신이 직접 컨텍스트를 다시 사용할 수 있는 다른 모듈/기능으로 갈 수 있다. 그러나 이 시점 이후에, 사용자는 이하에서 기술되는 바와 같이 사용자와 시스템 간의 대화를 계속하기 위해 이전의 직접 컨텍스트 모듈에 액세스할 수 있다:

Indirect Context (Temporary Context) = After any question/answer between the user and the agent (with or without direct context), the user can go to another module/function where he can directly reuse the context. However, after this point, the user can access the previous direct context module to continue the conversation between the user and the system as described below:

o user: how is the weather => agent gives me the weather in Palo Alto (agent detected my device GPS information to determine my location and give me the weather in Palo Alto)

o User: And when I am in San Francisco => Agent discovers my last direct context and gives me the weather in SF.

o user: what time is there => the agent understands that I want to change the intent module and determines again from the previous context the variables needed to complete the query for hours (hours) in SF.

o User: And how is tomorrow's weather => Agent detects that I want to return to the weather module (new intent), finds the place in my last weather query and provides me with the weather for tomorrow in SF.

o // An indirect context can persist for that time by storing it in a long-term storage such as a file or database. The same is true for a direct context that becomes an indirect context if there is any break in any action in the module.

의도 학습(IL) = 이상의 설명에 기초하여, 우리는 모듈/행동/화면/작업 흐름의 각각의 튜플에 대한 가중치를 사용해 의도 학습을 어떻게 생성할지를 쉽게 상상할 수있다.

Intention Learning (IL) = Based on the explanation above, we can easily imagine how we would generate intention learning using weights for each tuple of modules/actions/screens/workflows.

o What is IL = the system can understand the meaning of a user's query based on the user's historical queries, the workflow executed and the results achieved (similar to an Intelligent Macro for a personal assistant)

o Sample 1 (simple) =

시각 1

time 1

사용자: Louis에게 전화해주실 수 있으세요?

User: Could you please call Louis?

시스템: 연락처 리스트에 많은 Louis가 있습니다. 어느 Louis를 원하세요? 　(3명의 Louis가 있습니다)

System: I have a lot of Louis on my contact list. Which Louis do you want? (There are 3 Louis)

사용자: Monier(= 성)

User: Monier (= last name)

시스템: 알았습니다, Louis Monier에게 전화합니다

System: Okay, call Louis Monier.

시각 2

time 2

사용자: Louis에게 전화해!

User: Call Louis!

시스템: Louis Monier에게 전화합니다, 맞습니까?

System: I'm calling Louis Monier, right?

사용자: 예 ...시스템은 Louis Monier에게 전화합니다.

User: Yes ...The system calls Louis Monier.

사용자: 아니오 ... 맞아, 연락처 리스트에 여러 명의 Louis가 있어, ... (시스템은 시각 1의 두 번째 단계에서와 동일한 리스트를 나에게 보여준다)

User: No ... that's right, there are several Louis on the contact list, ... (the system shows me the same list as in the second step of time 1)

시각 3:

Time 3:

사용자: 지금 당장 Louis에게 전화해!

User: Call Louis right now!

시스템: Louis Monier에게 전화합니다. (시스템은 전화를 건다)

System: Call Louis Monier. (system dials)

o Sample 2 (more complex) =

시각 1

time 1

사용자: Louis에게 메시지를 보내

User: Send a message to Louis

시스템 = 메시지를 보내고자 하는 연락처가 누구입니까

System = Who is the contact you want to send the message to

사용자: Louis Monier(또는 Louis + Monier를 필터링, 또는 Monier + Louis를 필터링)

User: Louis Monier (or filter Louis + Monier, or filter Monier + Louis)

시스템 = 알았습니다, 문자 또는 메일을 보내기를 원하세요?

System = Okay, want to send a text or mail?

사용자 = 문자

user = character

시스템 = 알았습니다, 어떤 유형의 콘텐츠를 보내고 싶으세요(나는 자유 형식 텍스트, 음성 텍스트의 템플릿 텍스트를 선택할 수 있습니다)

System = OK, what type of content would you like to send (I can choose free-form text, template text for spoken text)

사용자 = 자유 형식 텍스트

user = free-form text

시스템 = 보낼 메시지를 말하세요

system = say message to send

사용자가 메시지를 말한다

user speaks a message

시스템은 메시지 내용의 확인, 내용 그리고 보내기 위한 유효성 확인을 요구한다.

The system requires confirmation of message content, content and validation for sending.

사용자는 예라고 말하거나 연락처, 메시지를 변경하거나, 더 많은 내용 메시지를 추가하거나 취소할 수 있다.

Users can say yes, change contacts, messages, add or cancel more content messages.

시각 2

time 2

사용자: Louis에게 메시지를 보내

User: Send a message to Louis

시스템 = 보낼 메시지를 말하세요

system = say message to send

사용자가 메시지를 말한다

user speaks a message

시스템은 메시지 내용의 확인, 내용 그리고 보내기 위한 유효성 확인을 요구한다.

The system requires confirmation of message content, content and validation for sending.

사용자는 예라고 말하거나 연락처, 메시지를 변경하거나, 더 많은 내용 메시지를 추가하거나 취소할 수 있다.

Users can say yes, change contacts, messages, add or cancel more content messages.

o Sample 3 (more complex)

시각 N

time N

사용자 = 매주 수요일, 사용자는 오후 6시에 사무실을 떠날 때 집으로 가기 전에 아내에게 전화를 한다 (아이들을 스포츠에 갈 준비를 하라고 요청함)

User = Every Wednesday, when he leaves the office at 6pm, he calls his wife before going home (requesting the kids to get them ready for sports)

그것을 위해, 사용자는 아내에게 전화하기 위해 상기 #1과 같은 샘플을 사용할 필요가 있다

For that, the user needs to use a sample like #1 above to call his wife.

시스템은 많은 정보 소스 = BT 데이터, OBD 데이터(또는 사용하지 않음), 시간(오후 6시), 지역화(집에 있지 않음), 이력 의도 (의제 미리 알림 + 지오펜싱 등)를 사용하고 있다.

The system is using many information sources = BT data, OBD data (or not used), time (6pm), localization (not at home), historical intent (agenda reminder + geofencing, etc).

시스템 = 사용자가 자신의 자동차에 도착할 때(자동차 BT 연결 또는 OBD 연결에 의해 검출됨) 그리고 x분(사용자가 자동차에 들어가는 데 걸리는 평균 시간) 후에　

System = when the user arrives in his car (detected by the car BT connection or OBD connection) and after x minutes (average time it takes the user to enter the car)

시스템이 자동으로 사용자에게 돌아와서 말한다:

The system automatically returns to the user and says:

시스템: "Greg, 집으로 내비게이션을 시작하고 아내에게 전화해주기를 원하세요."

System: "Greg, you want to start navigating home and call your wife."

사용자: 그래 => Celine Malvoisin에 대한 통화 행동이 시작된다.

User: Yes => Call action for Celine Malvoisin begins.

사용자: 아니오 => 에이전트는 아무것도 하지 않고 의도 학습 항목의 다운그레이드를 통지한다.

User: no => The agent does nothing and notifies the downgrade of the intent learning item.

In one embodiment, the IL was created to limit ASR interactions with the user and optimize the time to accomplish for any action the agent needs to execute. The IL stores a generic workflow execution based on the current context and requires parameters that cannot be discovered by themselves.

I have many different samples of the system's IL, like the one I'll be deploying next week.... I'm a French guy and the English ASR system doesn't recognize my voice (for my French accent) well, and I use the system. so if you want to send you text in english, i can use sample 2, just before i text you, i can ask you to translate the text in english have), the system will translate my French sentences into English and send them to you. At the same time, the system will understand that you are speaking English and will (by default) use TTS in English for any messages from you (before validation you send me texts in English). // It's funny how complex tasks can be hacked so easily; p = real-time text translation by voice.

Another interesting thing is that we can separate context or intent to give priority to any keyword or shortcut sentence from anywhere in the application of the workflow.

Appendix E: Context

Context: the current state of an existing personal assistant

Today, personal assistants have a first level of context, primarily to help the personal assistant understand the user's sentences and try to recognize words well. The sample below illustrates how Personal Assistant works

나는 Renaud에게 전화하고자 한다 => 성

I would like to call Renaud => Last name

나는 Renault로 드라이브 중이다 => 자동차 브랜드

I am driving with Renault => car brand

There are relationship and context definitions to define which [Renaud,Renault] the system needs to interpret and send back to the user. Context is also what the weather is like... and is used in certain cases like tomorrow (localization like context variable, but it could just be a process with a simple localization variable shared between the two steps).

assignment

The main task in personal assistants is to create real conversational exchanges between users and agents.

To understand this aspect, we need to understand the qualifications of "real conversation":

(질문 대답이 아니라) 임의의 사람 논의와 같은 대화 관리를 계속하는 것

Continuing conversation management, such as discussing random people (rather than answering questions)

o Ability to ask information about Yahoo... who the founders are, what's the stock and news (agents remember topics)

컨텍스트 대화 정보 메모리: 단기간, 중간 기간 또는 장기간 동안

Contextual conversation information memory : for short, medium or long periods of time

o Ability to remember information in the flow of discussion

프로세스 작업 흐름 메모리의 컨텍스트 상태: 단기간, 중간 기간 또는 장기간 동안

Context state of process work flow memory : for short, medium or long term

o Ability to remember where you are (steps) in a process or discussion workflow (to cause an action or not) to provide the ability to continue the process or workflow at any time in the future

Other than that, we need to bring about the evolution of the language used by agents to exchange with users. And in addition to that, we need to give the agent a sense of empathy.

General context management by xBrainSoft

The context, as described during the last call, consists of four components:

1. Context Client-Side Holder (CCSH)

This first component enables the client storage, use and definition (value) of the context workflow from the client side (robot, smartphone, vehicle, home, ...) for sharing with the server side. CCSH is an Fx with an API that creates, uses, and defines values of a context workflow from the client side and sends them through the CSP below.

2. Context Synchronization Protocol (CSP)

This second component defines the protocol (standardization) of key access (context ID) for each attribute (variable) of the state or substate of the current context, and verifies the format and existence of the key access. They can be simple text variables (name/value) or specific objects with instances of themselves. The goal of CSP is to be a communication protocol, build by a two-framework implementation on both sides of the agent (client/server), to validate correct protocol communication between client and server, and to ensure that contextual information is well communicated and synchronized. is in charge

3. Context Agent - Server-Side Holder (CA)

This third component enables server storage, use and definition (values) of context workflows from the server side (online server) to share with the client side via CSP. A CA is an Fx with an API that creates, uses, and defines the values of a context workflow from the server side and sends them through the above CSP.

4. Context Engine

This last component (based on any support) enables a level of variable sharing in the data store and medium to long term sessions.

Short-lived storage is managed by the current session shared between the client side and the server side.

It can define a classification or type of context type of a topic (a variable can be a simple variable or a serialized object + value(s)).

1. Current User Profile = Any information about the user profile (Facebook Profile, App Profile, ...)

2. Current module = any information about the module (phone, messages, navigation, news, ...)

3. Current function = any information about the function (to make a call, to receive a call, to send a text, to read the news, to share the news, ...)

1. Calling Louis for calling Louis Monier can be loaded from a medium/long term context engine that has learned that Louis = Louis Monier.

4. Current screen = Random information about the screen currently shown to the user

5. Custom Data = API that allows developers to use context in any aspect they want (new context geometry)

6. Workflow history = any information about the user's position in the workflow with information about the screens shown or to be shown, the values of variables at specific steps, the status of the workflow, ...

1. After I ask to share news on Facebook, and I say "continue", the agent will go to the next news in the list of news for the current category. The agent knows from the context the current category, the stage in reading the news he was in... and can send me the correct intent the user is asking for.

process

1. The voice and connectivity platform is operating in both synchronous and asynchronous modes, and we need to verify the perfect synchronization of the context between the client side and the server side at any time.

2. Each module, function, screen, application, session or any state, etc. needs to be identified with a unique ID (context ID) to be shared between the client and server.

3. The context ID (information storage memory) and its values are stored on both sides of the agent (client/server), and are synchronized between both sides in each interaction.

4. Context IDs enable:

1. Creating a filter and context-related action based on the value of a variable (simple variable or object) if ... then ... so ...

2. Finding information that needs to be loaded into short-term memory from medium- or long-term storage (or by machine learning from global user behavior/application level, probabilities for requested values)

3. Knowing which stage we are in the workflow (or by machine learning from global user behavior, probabilities for the next stage)

4. ...what else we discover from this innovation

How It Works (Life Cycle)

임의의 ASR 이후 그리고 NLU 프로세스 직전에, 디바이스는 문장 메시지와 함께 디바이스로부터 현재 컨텍스트 ID를 갖는 숨겨진 부분을 송신하고 있다.

After any ASR and just before the NLU process, the device is sending a hidden part with the current context ID from the device with a sentence message.

에이전트는 임의의 자연어 이해를 실행하기 전에 키 액세스(컨텍스트 ID)를 찾고 있다

The agent is looking for key access (context ID) before performing any natural language understanding.

o Agents view content and filter the global linguistic dictionary of actions and understandings for the current context.

에이전트는 컨텍스트 이해 시에 NLU 프로세스를 시작한다.

The agent starts the NLU process upon understanding the context.

o Action is what starts (API access or knowledge access)

o Agent interprets the meaning of the user's query... (see previous mail)

디바이스에 대답을 주는 것(또는 임의의 종류의 끝 지점) 이전에,　

Before giving the device an answer (or any kind of endpoint),

o Agent sends new context (module/function/screen) via reply message in hidden part (such as header for HTML page)

o New contexts can be defined from many variables:

종단점 유닛에서의 현재 화면

Current screen on endpoint unit

현재 모듈, 기능

Current module, function

문장, 대화 및 사용자의 선택 작업 흐름.

Sentences, conversations, and user selection workflows.

에이전트는 사용자에게 렌더링하기 위해 디바이스(종단점)로 송신할 답변(음성, 화면, 정보를 갖는 패키지)을 병합한다.

The agent merges the answers (packages with voice, screen, information) to send to the device (endpoint) for rendering to the user.

클라이언트측은 패키지를 실행하고 현재 컨텍스트를 저장한다.

The client side runs the package and saves the current context.

o Context can be enforced from any screen, function or module ... in the case of the home screen, we force reset the context and let the user start from a clean interaction with the agent.

In the case of a context conflict between a server and a client (endpoint), the client (endpoint: device, vehicle, home) is the master, since the client represents the behavior of the user (the actual master).

Samples used:

사용자가 말할 때 선택할 Louis를 컨텍스트화한다: 나는 (Louis의 이력 전화 거동에 기초하여) Louis에게 전화하고자 한다 => Louis Monier에게 전화한다

Contextualize which Louis to choose when the user speaks: I want to call Louis (based on Louis' history calling behavior) => Call Louis Monier

실행할 프로세스를 컨텍스트화한다: Louis에게 메시지를 보낸다　

Contextualize the process to run: send a message to Louis

o The system knows that message = email, Louis = Louis Monier.

o Enable voice shortcuts... and cut two steps in the workflow to email Louis Monier.

실행할 다음 단계를 컨텍스트화한다: 많은 세션에서, 나는 뉴스 순서 = 환경, 정치 및 스포츠를 요구한다. 내가 다음 번에 환경을 요청할 때, 에이전트는 정치 및 스포츠 뉴스를 읽을 것을 당신에게 제안할 것이다.

Contextualize the following steps to execute: In many sessions, I ask for news order = environment, politics and sports. The next time I request the environment, the agent will offer you to read political and sports news.

애플리케이션 전역 예측 작업 흐름에 따라 다음 단계를 컨텍스트화한다.

Contextualize the following steps according to the application-wide prediction workflow.

요청된 작업을 컨텍스트화하고 그것이 현재 컨텍스트를 목표로 하지 않는다는 것을 이해하며 이전 행동에 대해 그것을 사용할 수 있다.

You can contextualize the requested operation, understand that it does not target the current context, and use it for previous actions.

o I am reading a list of news, I am requesting the weather, I say “go on” and the agent goes to the next news.

뮤지컬 뉴스 또는 당신의 전화에서의 음악일 수 있는 뉴스의 컨텍스트에서 질문하는 "음악" ...과 같은 특정의 단어를 컨텍스트화한다.

Contextualize certain words like "music"... you are asking in the context of the news, which could be musical news or music on your phone.

o From the music context, it is clear to access the device's music tracks.

o In a news context, it may be about a music reproduction of the news, and the agent returns to the user to understand and ask more precisely.

o If the user tells you to play music in the news context, the agent understands that the user does not want to read the news.

우리는 현재 컨텍스트를 알고 있기 때문에, 우리는 임의의 입력 음성 인식을 컨텍스트화하고 문장의 의미를 이해하려고 하기 전에 문장에서의 단어를 변경할 수 있고 ... 또는 반대로 임의의 행동을 시작하기 위해 특정의 컨텍스트에서 이용가능한 어휘를 확장할 수 있다.

Because we know the current context, we can contextualize any input speech recognition and change words in a sentence before trying to understand the meaning of the sentence... or conversely, to initiate an arbitrary action. It is possible to expand the vocabulary available in the context.

o A second effect is that we do not need to generate many patterns to validate the behavior (e.g. music can be captured in arbitrary sentences, short or long, in the context of the root screen that initiates the behavior playing music). have)

o A third effect is for translation, because for each context module/function/screen you can limit keywords that capture the action intended by the user.

TV의 컨텍스트에서의 재생은 게임을 플레이하거나 TV 프로그램을 재생하는 것이다

Playback in the context of TV is playing a game or playing a TV show

스포츠의 컨텍스트에서의 플레이는 새로운 게임을 플레이하는 것이다

Playing in the context of sports is playing a new game

디스코테크의 컨텍스트에서의 재생은 음악을 재생하는 것이다

Playing in the context of a discotheque is playing music

... 1개의 단어, 컨텍스트에 의존하는 많은 의도는 ...임의의 언어로 번역하기 쉽다

...one word, many intentions dependent on context ...easy to translate into any language

o A fourth effect is the support of arbitrary agents, since dictionaries can be very limited.

뉴스캐스터의 경우에, 우리는 "뉴스"(+ 동의어) 및 뉴스 토픽 엔티티를 포착한다.

In the case of a newscaster, we capture "news" (+ synonyms) and news topic entities.

작업 우선순위의 파이프라인의 생성

Creating a pipeline of task priorities

o I am currently composing a message for a contact (usually, I want to end an action)

o I receive text from the contact during this time, the system sees the current context and knows when the user is composing a message, he does not need to interrupt the current action

o The agent creates a pipeline of messages, and at the end of the message creation context, the agent will offer me to read the message (when the context is changing).

컨텍스트에 따라 임의의 메시지의 번역

Translation of arbitrary messages according to context

o I write a message to Mark (he speaks English and I write the message in French), and the system, based on the context of the message, has to make sure he knows the recipient's language before sending it to translate it. know you need to

The context workflow is the state of the context matrix (module, function, screen) in the process workflow from the beginning to the end of a user session. We have built a system that allows computers to generate intuitions from collective intelligence (generating numerical intuitions) from intention learning.

A few things to note about the above:

설명된 바와 같이, 우리는 동기 및 비동기 모드에서 작업하고 있다.

As explained, we are working in synchronous and asynchronous modes.

o These two paths are used to enable more than proactive response to asynchronous mode.

o Allow both sides to know where both sides of the conversation are.

수명 주기에 대한 부가 기능:

Additional features for lifecycle:

o For the first point: 　 can also be sent not only from ASR, but also during application discovery (tactile interaction).

o About point 5: Packages can be sent with all or some content

우리는 모든 요소를 음성을 통합하지 않고 보낼 수 있고, 이 경우에, 에이전트는 전체 렌더링 및 컨텍스트의 생성/편집을 관리할 것이다.

We can send all elements without voice integration, in this case the agent will manage the entire rendering and creation/editing of the context.

Claims (20)

A computer-implemented method comprising:
detecting an event;
in response to detecting the event, proactively initiating a conversation of a voice assistant on a first user device with a user;
in response to initiating a conversation with the user, receiving, at the first user device, a first audio input associated with the conversation from the user requesting a first action;
performing automatic speech recognition on the first audio input;
determining, at the first user device, a first context of the user;
determining a first tuple describing a user intent, the first tuple comprising the first action and an actor associated with the first action, the first tuple being based on automatic speech recognition of the first audio input determined by performing natural language understanding -;
initiating the first action on the first user device based on the first tuple;
after initiating the first action, receiving a second audio input from the user requesting a second action unrelated to the first action;
initiating the second action;
After initiating the second action, at a second user device different from the first user device, receive a third audio input from the user continuing the conversation and requesting a third action related to the first action wherein the third audio input is missing information for completing a third tuple for initiating the third action;
obtaining missing information using the first context to complete the third tuple associated with the third action; and
initiating the third action at the second user device based on the third tuple.
A computer-implemented method comprising: The method of claim 1 , wherein the event is an internal event. The method of claim 1 , further comprising initiating the voice assistant without user input and receiving the first audio input from the user after initiation of the voice assistant. The method of claim 1 , wherein the first context comprises one or more of a context history, a conversation history, a user profile, a user history, a location, and a current context domain. The method of claim 1 , wherein the missing information is one or more of the third action, an actor associated with the third action, and an entity associated with the third action. According to claim 1,
determining that the first context and the first audio input are missing first information used to initiate the first action;
determining which information is the missing first information; and
prompting the user to provide an audio input providing the missing first information;
A computer-implemented method further comprising: According to claim 1,
determining that first information used to initiate the first action cannot be obtained from the first audio input;
determining which information is the missing first information; and
prompting the user to provide an audio input providing the missing first information that cannot be obtained from the first audio input;
A computer-implemented method further comprising: According to claim 1,
determining that first information used to initiate the first action cannot be obtained from the first audio input;
determining which information is the missing first information;
providing a plurality of options for selection by the user, the options providing potential information for completing the first action; and
receiving an audio input for selecting a first option from the plurality of options;
A computer-implemented method further comprising: The method of claim 1 , wherein the second action not associated with the first action is associated with a second context, and wherein the first action and the third action are associated with the first context. . As a system,
one or more processors; and
memory to store instructions
wherein the instructions, when executed by the one or more processors, cause the system to:
detecting an event;
in response to detecting the event, proactively initiating a conversation of the voice assistant on the first user device with the user;
in response to initiating a conversation with the user, receiving, at the first user device, a first audio input associated with the conversation from the user requesting a first action;
performing automatic speech recognition on the first audio input;
determining, at the first user device, a first context of a user;
determining a first tuple describing a user intent, the first tuple comprising the first action and an actor associated with the first action, the first tuple being based on automatic speech recognition of the first audio input determined by performing natural language understanding -;
initiating the first action on the first user device based on the first tuple;
after initiating the first action, receiving a second audio input from the user requesting a second action unrelated to the first action;
initiating the second action;
After initiating the second action, at a second user device different from the first user device, receive a third audio input from the user continuing the conversation and requesting a third action related to the first action wherein the third audio input is missing information for completing a third tuple for initiating the third action;
obtaining missing information using the first context to complete the third tuple associated with the third action; and
initiating the third action on the second user device based on the third tuple.
A system for performing steps comprising: 11. The system of claim 10, wherein the event is an internal event. 11. The method of claim 10, wherein the instructions, when executed by the one or more processors, cause the system to:
initiate the voice assistant without user input and receive the first audio input from the user after initiation of the voice assistant. The system of claim 10 , wherein the first context comprises one or more of a context history, a conversation history, a user profile, a user history, a location, and a current context domain. The system of claim 10 , wherein the missing information is one or more of the third action, an actor associated with the third action, and an entity associated with the third action. 11. The method of claim 10, wherein the instructions, when executed by the one or more processors, cause the system to:
determine that the first context and the first audio input are missing first information used to initiate the first action;
determine which information is missing first information;
prompt the user to provide an audio input providing the missing first information. 11. The method of claim 10, wherein the instructions, when executed by the one or more processors, cause the system to:
determine that first information used to initiate the first action cannot be obtained from the first audio input;
determine which information is missing first information;
prompt the user to provide an audio input providing the missing first information that cannot be obtained from the first audio input. 11. The method of claim 10, wherein the instructions, when executed by the one or more processors, cause the system to:
determine that first information used to initiate the first action cannot be obtained from the first audio input;
determine which information is missing first information;
providing a plurality of options for selection by the user, the options providing potential information for completing the first action;
receive an audio input selecting a first option from the plurality of options. The system of claim 10 , wherein the second action not associated with the first action is associated with a second context, and wherein the first action and the third action are associated with the first context. delete delete

Images