US20190391541A1 - Technologies for conversational interfaces for system control - Google Patents
Technologies for conversational interfaces for system control Download PDFInfo
- Publication number
- US20190391541A1 US20190391541A1 US16/352,018 US201916352018A US2019391541A1 US 20190391541 A1 US20190391541 A1 US 20190391541A1 US 201916352018 A US201916352018 A US 201916352018A US 2019391541 A1 US2019391541 A1 US 2019391541A1
- Authority
- US
- United States
- Prior art keywords
- home automation
- electronic device
- computing device
- consumer electronic
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B15/00—Systems controlled by a computer
- G05B15/02—Systems controlled by a computer electric
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/20—Pc systems
- G05B2219/26—Pc applications
- G05B2219/2642—Domotique, domestic, home control, automation, smart house
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/26—Speech to text systems
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/223—Execution procedure of a spoken command
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/226—Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics
- G10L2015/227—Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics of the speaker; Human-factor methodology
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/226—Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics
- G10L2015/228—Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics of application context
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
- Selective Calling Equipment (AREA)
- Machine Translation (AREA)
Abstract
Technologies for conversational control of a home automation system include a computing device in communication with one or more home automation devices. The computing device receives text input indicative of a spoken user request including one or more parameters associated with a home automation command. The computing device matches the text input against a home automation language model including several predefined dialog patterns. The computing device resolves parameters of the command using a context model. The computing device may resolve the location, device, or requested device state of the command. The computing device generates a text response and a home automation command using the language model, and then presents the text response to the user and performs the home automation command using the home automation devices. In some embodiments, the computing device may control an application automation system or a data exploration system. Other embodiments are described and claimed.
Description
- The present application is a continuation application of U.S. application Ser. No. 14/750,482, entitled “TECHNOLOGIES FOR CONVERSATIONAL INTERFACES FOR SYSTEM CONTROL,” which was filed on Jun. 25, 2015.
- Home automation systems include devices and software capable of controlling building functions such as lighting, climate control, security, and other functions. For example, home automation systems may provide for centralized control of light switches, dimmers, thermostats, heating ventilation and air conditioning (HVAC) systems, building sensors, and other household devices. Typical home automation systems may be operated using “control panel”-type hardware and/or software interfaces that allow a user to manually operate each home automation device. For example, the user may turn on lights by operating various wall switches and/or manually clicking various options in a home automation application executed by a standard computer.
- Certain computing devices may implement conversational user interfaces, which may allow for back-and-forth interaction between the computing device and user. For example, a conversational user interface may maintain a memory of the current conversation and generate clarifying questions as part of the conversation. Typical conversational user interfaces may require complex machine learning algorithms and/or training using large data sets. Alternatively, Chatbots (also known as “chatterbots”) allow a programmer to use a scripting language to develop interactions with the user that respond to natural language requests in an apparently conversational way. Chatbots such as ELIZA have existed at least since the 1960s. Current computing devices may implement chatbots using a scripting and pattern matching language such as ChatScript.
- The concepts described herein are illustrated by way of example and not by way of limitation in the accompanying figures. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.
-
FIG. 1 is a simplified block diagram of at least one embodiment of a system for a conversational interface for home automation control; -
FIG. 2 is a simplified block diagram of at least one embodiment of an environment that may be established by a computing device of the system ofFIG. 1 ; -
FIGS. 3A and 3B is a simplified flow diagram of at least one embodiment of a method for conversational control of a home automation system that may be executed by the computing device ofFIGS. 1 and 2 ; -
FIG. 4 is a simplified flow diagram of at least one embodiment of a method for dialog pattern learning that may be executed by the computing device ofFIGS. 1 and 2 ; -
FIG. 5 is a schematic diagram illustrating a conversational dialog that may be executed by the computing device ofFIGS. 1 and 2 ; -
FIG. 6 is a schematic diagram illustrating another conversational dialog that may be executed by the computing device ofFIGS. 1 and 2 ; and -
FIG. 7 is a simplified flow diagram of at least one embodiment of a method for conversational control of a system that may be executed by the computing device ofFIGS. 1 and 2 . - While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.
- References in the specification to “one embodiment,” “an embodiment,” “an illustrative embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may or may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. Additionally, it should be appreciated that items included in a list in the form of “at least one A, B, and C” can mean (A); (B); (C): (A and B); (B and C); or (A, B, and C). Similarly, items listed in the form of “at least one of A, B, or C” can mean (A); (B); (C): (A and B); (B and C); or (A, B, and C).
- The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).
- In the drawings, some structural or method features may be shown in specific arrangements and/or orderings. However, it should be appreciated that such specific arrangements and/or orderings may not be required. Rather, in some embodiments, such features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of a structural or method feature in a particular figure is not meant to imply that such feature is required in all embodiments and, in some embodiments, may not be included or may be combined with other features.
- Referring now to
FIG. 1 , anillustrative system 100 for a conversational interface for home automation control includes acomputing device 102, ahome automation controller 104, and one or morehome automation devices 106 in communication over anetwork 114. In use, a user provides thecomputing device 102 with input text that is indicative of a spoken home automation command such as “Turn on the lights.” Thecomputing device 102 matches the input text against a home-automation-specific language model that includes several dialog patterns representing potential interactions with a home automation system. The language model may be defined using a chatbot scripting language such as ChatScript. After matching a pattern and identifying a requested home automation command, thecomputing device 102 may resolve ambiguous parameters of the command using context data. The context data may include conversational context data as well as context data based on the current user and/or on the state of thehome automation devices 106. For example, when supplied with the input text “Turn on the lights,” thecomputing device 102 may identify the particular lights to be controlled based on conversational context data (e.g., which location within the home is the topic of the conversation), user context data (e.g., where is the user located), and/or device context data (e.g., what lights in the system are not currently on). After resolving the parameters of the command, thecomputing device 102 may generate a natural language response and a home automation command based on the language model, and then execute the home automation command using thehome automation controller 104 and/or thehome automation devices 106. If the command is not sufficiently resolved, thecomputing device 102 may generate a follow-up question using the language model for the user to resolve additional parameters. Thus, thecomputing device 102 may provide an intuitive and efficient conversational interface for controlling a home automation system. By using a predefined domain-specific language model, thecomputing device 102 may provide a conversational interface without performing complex machine learning algorithms and/or training using large data sets. Additionally, in some embodiments thecomputing device 102 may provide conversational control of systems other than home automation systems by using an appropriate domain-specific language model. - The
computing device 102 may be embodied as any type of device capable of performing the functions described herein. For example, thecomputing device 102 may be embodied as, without limitation, a desktop computer, a server, a consumer electronic device, a smart appliance, a notebook computer, a laptop computer, a smartphone, a cellular phone, a tablet computer, distributed computing system, a multiprocessor system, and/or any other computing device capable of performing the functions described herein. As shown inFIG. 1 , theillustrative computing device 102 includes aprocessor 120, an I/O subsystem 122,memory 124, adata storage device 126, andcommunication circuitry 128. Of course, thecomputing device 102 may include other or additional components, such as those commonly found in a portable computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, thememory 124, or portions thereof, may be incorporated in theprocessor 120 in some embodiments. - The
processor 120 may be embodied as any type of processor capable of performing the functions described herein. For example, the processor may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. Similarly, thememory 124 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, thememory 124 may store various data and software used during operation of thecomputing device 102 such as operating systems, applications, programs, libraries, and drivers. Thememory 124 is communicatively coupled to theprocessor 120 via the I/O subsystem 122, which may be embodied as circuitry and/or components to facilitate input/output operations with theprocessor 120, thememory 124, and other components of thecomputing device 102. For example, the I/O subsystem 122 may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem 122 may form a portion of a system-on-a-chip (SoC) and be incorporated, along with theprocessor 120, thememory 124, and other components of thecomputing device 102, on a single integrated circuit chip. - The
data storage device 126 may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. Thecommunication circuitry 128 of thecomputing device 102 may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between thecomputing device 102, thehome automation devices 106, and/or other remote devices. Thecommunication circuitry 128 may be configured to use any one or more communication technology (e.g., wireless or wired communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, Zigbee, Zwave, etc.) to effect such communication. In some embodiments, thecommunication circuitry 128 may use special-focus communication protocols to provide interconnectivity to member devices in a local Internet of Things (IoT). - The
computing device 102 may further include anaudio device 130 and anaudio sensor 132. Theaudio device 130 may be embodied as any device capable of generating audio signals for output, such as a paper cone speaker, an audio transducer, an audio output jack, a digital-to-analog converter (DAC), or other type of audio device. Theaudio device 130 may be used by thecomputing device 102 to output speech responses to the user, as described below. Theaudio sensor 132 may be embodied as any sensor capable of capturing audio signals such as a microphone, a line input jack and associated circuitry, an analog-to-digital converter (ADC), or other type of audio sensor. Theaudio sensor 132 may be used by thecomputing device 102 to detect speech commands uttered by the user, as described below. - The
home automation controller 104 is configured to execute rules, receive sensor and actuator changes, trigger actions, and otherwise control thehome automation devices 106. Thehome automation controller 104 may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a multiprocessor system, a server, a rack-mounted server, a blade server, a laptop computer, a notebook computer, a tablet computer, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and/or a consumer electronic device. Thus, thehome automation controller 104 includes components and devices commonly found in a server or similar computing device, such as a processor, an I/O subsystem, a memory, a data storage device, and/or communication circuitry. Those individual components of thehome automation controller 104 may be similar to the corresponding components of thecomputing device 102, the description of which is applicable to the corresponding components of thehome automation controller 104 and is not repeated herein so as not to obscure the present disclosure. Thehome automation controller 104 may be embodied as a computing device located at the home controlled by the home automation system or as one or more remote devices, such as a cloud-based service. Accordingly, in some embodiments, thehome automation controller 104 may be embodied as a “virtual server” formed from multiple computing devices distributed across thenetwork 114 and operating in a public or private cloud. Accordingly, although thehome automation controller 104 is illustrated inFIG. 1 as embodied as a single computing device, it should be appreciated that thehome automation controller 104 may be embodied as multiple devices cooperating together to facilitate the functionality described below. Additionally, although illustrated as including aseparate computing device 102 andhome automation controller 104, it should be understood that in some embodiments those functions may be combined in a single computing device. - Each of the
home automation devices 106 may be embodied as a smart appliance, connected device, smart thermostat, connected sensor, embedded computing device, or other computing device or collection of devices capable of performing the functions described herein. Accordingly, eachhome automation device 106 may include components and devices similar tocomputing device 102, such as a processor, an I/O subsystem, a memory, a data storage device, communication circuitry, and/or other peripheral devices. As illustrated inFIG. 1 , thehome automation devices 106 may include anHVAC system 108, alighting system 110, and asecurity system 112. TheHVAC system 108 may include a smart thermostat, heater, air conditioner, and/or other building comfort systems. Thelighting system 110 may include one or more light switches, dimmers, lights, and other devices. Thesecurity system 112 may include door locks, security cameras, sensors such as motion sensors, light sensors, and/or contact sensors, building safety systems (e.g., smoke detectors, carbon monoxide detectors, sprinkler systems, etc.), alarms, and/or other security systems. Of course, in other embodiments, thehome automation devices 106 may include additional or different devices and/or systems, such as smart appliances, entertainment systems, and other home automation devices. - As discussed in more detail below, the
computing device 102, thehome automation controller 104, and thehome automation devices 106 may be configured to transmit and receive data with each other and/or other devices of thesystem 100 over thenetwork 114. Thenetwork 114 may be embodied as any number of various wired and/or wireless networks. For example, thenetwork 114 may be embodied as, or otherwise include, a wired or wireless local area network (LAN), a wired or wireless wide area network (WAN), and/or a publicly-accessible, global network such as the Internet. As such, thenetwork 114 may include any number of additional devices, such as additional computers, routers, and switches, to facilitate communications among the devices of thesystem 100. In some embodiments, thenetwork 114 may include a pervasive network of smart embedded devices, such as a local or global Internet of Things (IoT). - Referring now to
FIG. 2 , in the illustrative embodiment, thecomputing device 102 establishes anenvironment 200 during operation. Theillustrative environment 200 includes an automatedspeech recognition module 202, a text-to-speech module 204, acontroller module 206, aconversational interface module 208, acontext management module 212, and alearning module 216. The various modules of theenvironment 200 may be embodied as hardware, firmware, software, or a combination thereof. For example the various modules, logic, and other components of theenvironment 200 may form a portion of, or otherwise be established by, theprocessor 120 or other hardware components of thecomputing device 102. As such, in some embodiments, any one or more of the modules of theenvironment 200 may be embodied as a circuit or collection of electrical devices (e.g., a controller circuit, a conversational interface circuit, etc.). - The automated
speech recognition module 202 is configured to perform automated speech recognition on audio input data to generate input text. Similarly, the text-to-speech module 204 is configured to generate audio speech output based on response text using a text-to-speech engine. - The
controller module 206 is configured to receive input text indicative of a spoken user request including one or more parameters associated with a system control command. After thecomputing device 102 processes the input text as described further below, thecontroller module 206 is further configured to present a text response to the user of thecomputing device 102 and perform the system control command. In some embodiments, the system control command may be embodied as a home automation command including home automation parameters such as a location (e.g., a room within the home), ahome automation device 106, and/or a requested device state (e.g., on or off). Thecontroller module 206 may be configured to determine whether the home automation command is valid based on the state of the associated home automation system. Additionally or alternatively, the system control command may be embodied as an application control command including application control parameters such as an application interface command or an application object. In some embodiments, the system control command may be embodied as a data exploration command including data exploration parameters such as a data query parameter. In some embodiments, thecontroller module 206 may be further configured to receive non-textual input data indicative of the user request, such as video data or sensor data. - The
conversational interface module 208 is configured to match the input text against alanguage model 210 to identify a matching pattern. Thelanguage model 210 includes several domain-specific dialog patterns, and each pattern may be associated with a corresponding system control command. Thelanguage model 210 may also include domain-specific dictionaries providing lists of synonyms or related words that may also be used for matching the input text. Theconversational interface module 208 is configured to generate a text response using thelanguage model 210 and to generate the system control command (e.g., a home automation command, an application control command, or a data exploration command) using thelanguage model 210. Theconversational interface module 208 is configured to generate the system control command after parameters of the system control command are determined by thecontext management module 212, as described below. Theconversational interface module 208 may be configured to determine whether the system control command is sufficiently resolved prior to generating the system control command and to generate a follow-up question for the text response using thelanguage model 210 if the system control command is not sufficiently resolved. Theconversational interface module 208 may be further configured to generate an error message for the text response using thelanguage model 210 if the system control command is not valid based on the state of thehome automation device 106 context data. - The
context management module 212 is configured to determine one or more of the parameters of the system control command using acontext model 214. Thecontext model 214 may include context data representing the state of the current conversation, the state of the user, the state of the system controlled by thecomputing device 102, or other appropriate contextual data. For example, in use with a home automation system, thecontext model 214 may include conversational context data, user context data, andhome automation device 106 context data. Thecontext management module 212 may be configured to update thecontext model 214 based on data received from the system being controlled, such as the home automation system (including one or more of the home automation devices 106). As another example, in use with an application control system, thecontext model 214 may include conversational context data and application context data, such as open files, the current view, recent commands, or other application context. As yet another example, in use with a data exploration system, thecontext model 214 may include conversational context data and query context data including parameters or other data relating to previous queries. - The
learning module 216 is configured to update thelanguage model 210 by performing learning by example. In particular, thelearning module 216 is configured to observe a change in device state of the home automation system in response to performance of the home automation command and generate a new pattern based on the input text. The new pattern is associated with the home automation command. For example, the new pattern may be based on individualized vocabulary or speech patterns used by the user to request a particular home automation command. Thelearning module 216 is further configured to update thelanguage model 210 to include the new pattern. - Although illustrated as being established by the
computing device 102, it should be understood that in some embodiments part or all of theenvironment 200 may be established by one or more computing devices distributed over a network such as the Internet. For example, in some embodiments part or all of the functions of the automatedspeech recognition module 202 and/or the text-to-speech module 204 may be performed by a remote and/or cloud-based speech recognition server. As another example, in some embodiments part or all of the functionality of the conversational interface module 208 (including the language model 210) and/or the context management module 212 (including the context model 214) may be provided by or maintained by one or more remote servers over the Internet. - Referring now to
FIGS. 3A and 3B , in use, thecomputing device 102 may execute amethod 300 for conversational control of a home automation system. Themethod 300 begins withblock 302 and block 312, executed in parallel. Inblock 302, thecomputing device 102 receives input text from a user of thecomputing device 102. The input text may represent a spoken user request relating to the home automation system, including one or more parameters associated with a home automation system command. For example, the user request may include a command to adjust the device state of one or morehome automation devices 106 that are identified by location. As another example, the user request may include a command to query the state of one or morehome automation devices 106. The input text is a natural language request and thus may not conform to any particular predefined format or syntax. For example, the input text may be embodied as the statement, “Turn on the lights,” “Turn the heat up,” or similar. In particular, the input text may not include all parameters required to execute the home automation command. - In some embodiments, in
block 304 thecomputing device 102 may receive the input text from an automated speech recognition system. Thecomputing device 102 may use any appropriate speech recognition system, including local speech recognition systems, cloud-based speech recognition systems, or any other speech recognition system. Inblock 306, thecomputing device 102 may gate the input text on a control word. In other words, thecomputing device 102 may disregard spoken commands that are not preceded by the control word. The control word may be embodied as a name or other identifier associated with thecomputing device 102. For example, thecomputing device 102 may respond to commands similar to “Homer, turn on the lights,” but disregard commands that do not start with “Homer.” Additionally or alternatively, in some embodiments thecomputing device 102 may use any other technique to activate input processing, including monitoring multiple modes of input. For example, in some embodiments thecomputing device 102 may activate input processing in response to a particular user gesture or in response to the user looking in the direction of a particular device. - In some embodiments, in
block 308 thecomputing device 102 may restrict the input text to a conversation with a single user selected from multiple users of thecomputing device 102. In those embodiments, thecomputing device 102 may have multipleaudio sensors 132 or otherwise be capable of interacting with multiple users simultaneously and thus may process each conversation separately. In some embodiments, inblock 310 thecomputing device 102 may receive non-textual input data along with the input text. For example, thecomputing device 102 may receive camera data or other data indicative of the user of thecomputing device 102. As described further below, the non-textual data may be used to match against thelanguage model 210 and/or the update thecontext model 214. - As described above, the
method 300 begins withblocks block 312, thecomputing device 102 updates thecurrent context model 214 based on the status of the home automation system. Thecomputing device 102 may update thecontext model 214 in response to data received from one or morehome automation devices 106. Thecomputing device 102 may receive asynchronous updates from thehome automation devices 106, poll thehome automation devices 106, or perform any other appropriate technique for determining the status of the home automation system. - In some embodiments, in
block 314 thecomputing device 102 updates device and/or device state data in thecontext model 214. For example, thecomputing device 102 may update thecontext model 214 to identify thehome automation devices 106 that are currently available in thesystem 100. As another example, thecomputing device 102 may update thecontext model 214 to identify device states associated with thehome automation devices 106. Continuing that example, thecomputing device 102 may update thecontext model 214 to indicate the current temperature set point of a particular thermostat, whether particular light switches are on or off, the setting of particular light dimmers, or other device state information. In some embodiments, inblock 316 thecomputing device 102 may update thecontext model 214 to identify a location of the user. Thecomputing device 102 may determine the location of the user, for example, based on sensor data such as camera data, motion sensor data, or other sensor data. - Although illustrated as executing
blocks computing device 102 may execute those functions in at different times or in different orders. For example, in some embodiments thecomputing device 102 may executeblocks computing device 102 may execute block 312 asynchronously, for example updating thecontext model 214 based on the state of the home automation system. After receiving input text inblock 302 and/or updating thecontext model 214 inblock 312, themethod 300 proceeds to block 318. - In
block 318, thecomputing device 102 matches the input text against dialog patterns defined in thelanguage model 210 to identify a matching pattern. As described above, thelanguage model 210 includes a collection of patterns arranged hierarchically to define one or more dialogs. The patterns may be embodied as, for example, one or more regular-expression-like script language statements in a language such as ChatScript. As an example, a pattern such as “(!not !˜state !˜location [lights it]˜dark)” may match any sentence that does not contain “not,” “don't,” “on,” “off,” or a specific room in the house and does contain either a word for “lights” or “it,” followed by a word that implies dimming the lights. That pattern would match inputs such as “Turn the lights down,” “Make it dimmer,” and “Set the lights to dark.” - Each pattern may be associated with a home automation command and one or more associated parameters of the home automation command. Each parameter may correspond to particular matching words, phrases, or other values of the input text. Parameters of the command may include any data necessary to perform the command, including parameters identifying the specified
home automation devices 106, requested device states, and/or locations. Each pattern may also be associated with one or more responses, including text responses. In some embodiments, inblock 320 thecomputing device 102 may match and identify values for location,home automation device 106, and/or device state in the input text. For example, given the input text “Please flip on the kitchen lights,” thecomputing device 102 may determine the location to be “kitchen,” thehome automation devices 106 to be “lights,” and the device state to be “on.” Thecomputing device 102 may partially match the input text to the parameters of a home automation command—for example, thecomputing device 102 may not identify matches for location, device, and/or device state in the input text. - In
block 322, thecomputing device 102 updates conversation context data of thecontext model 214 based on the input text and the matching pattern. Thecomputing device 102 may update thecontext model 214 to identify the object, location, and/or state change requested in the current conversation. For example, thecontext model 214 may be updated with values matched the input text for location,home automation device 106, and/or device state. In some embodiments, thecomputing device 102 may age thecontext model 214 to eliminate older data or other data not relevant to the current conversation. For example, thecomputing device 102 may clear data in thecontext model 214 after a predetermined time period has elapsed, after thecomputing device 102 determines that the current conversation has ended, or after any other appropriate event. - In
block 324, thecomputing device 102 resolves potential ambiguities or missing parameters in the requested home automation command using thecurrent context model 214. In other words, thecomputing device 102 may determine one or more parameters of the home automation command based on values selected from thecurrent context model 214. In some embodiments, inblock 326 thecomputing device 102 may supply the location,home automation device 106, and/or device state from conversation context data of thecontext model 214. For example, if the input text is “turn the lights on,” the location of the home automation command (e.g., the particular room having the lights) may be ambiguous. In that example, thecomputing device 102 may determine whether thecontext model 214 identifies a location associated with the current conversation. In some embodiments, inblock 328, thecomputing device 102 may supply the location from user context data of thecontext model 214. For example, if the input text is “turn the lights on,” thecomputing device 102 may identify the location as the room of the user's current location based on thecontext model 214. In some embodiments, inblock 330 thecomputing device 102 may supply thehome automation device 106 and/or device state from device context data of thecontext model 214. For example, if the input text is “turn the kitchen lights on,” thecomputing device 102 may identify the particularhome automation devices 106 used to control the kitchen lights based on the availablehome automation devices 106 described in thecontext model 214. As another example, if the input text is “toggle the lights,” thecomputing device 102 may determine the device state (e.g., on or off) based on the current device state described in thecontext model 214. - In
block 332, shown inFIG. 3B , thecomputing device 102 determines whether the home automation command is sufficiently resolved. Thecomputing device 102 may determine, for example, whether the parameters of the home automation command have been disambiguated or otherwise resolved sufficiently to allow the home automation command to be executed. In some embodiments, inblock 334 thecomputing device 102 may determine whether a location (e.g., a room within the home), ahome automation device 106, and a requested device state have been identified for the home automation command. Inblock 336, thecomputing device 102 checks whether the home automation command has been resolved. If the command has been resolved, themethod 300 branches ahead to block 340, described below. If the command has not been resolved, themethod 300 advances to block 338. - In
block 338, thecomputing device 102 generates a follow-up question for the response. The follow-up question may be embodied as any natural language response that prompts the user to supply additional information that may be used to resolve the home automation command. Thecomputing device 102 may determine the follow-up question using thelanguage model 210. For example, thelanguage model 210 may include one or more predefined follow-up questions associated with the current pattern matching the input text. For example, if the input text is “turn on the lights,” and thecomputing device 102 cannot resolve the location of the lights based on thecontext model 214, the follow-up question may be embodied as, “Which lights do you mean?” After generating the follow-up question for the response, themethod 300 branches ahead to block 350 to present the response to the user, as described further below. - Referring back to block 336, if the home automation command is sufficiently resolved, the
method 300 branches ahead to block 340, in which thecomputing device 102 determines whether the home automation command is consistent with the current device state stored in thecontext model 214 or otherwise valid. For example, if the input text is “turn on the kitchen lights,” thecomputing device 102 may determine whether the kitchen lights are currently turned off (i.e., that the request to turn on the kitchen lights is consistent with their current state of being turned off and therefore valid). In block 342, thecomputing device 102 checks whether the command is valid. If so, themethod 300 branches ahead to block 346 as described further below. If the command is not valid, themethod 300 advances to block 344, in which thecomputing device 102 generates an error message for the response. After generating the error message for the response, themethod 300 branches ahead to block 350 to present the response to the user, as described further below. - Referring back to block 342, if the command is valid, the
method 300 branches to block 346, in which thecomputing device 102 generates response text based on thelanguage model 210. The response text may be embodied as any natural language response that acknowledges or otherwise confirms that the user request has been successfully processed. Thecomputing device 102 may determine the response using thelanguage model 210. For example, thelanguage model 210 may include one or more predefined responses associated with the current pattern matching the input text. For example, the response may be embodied as “OK, I turned the lights on,” or simply, “OK.” - In
block 348, thecomputing device 102 generates a home automation command based on thelanguage model 210. The home automation command may be embodied as any command, query, or other action that may be performed using one or more of thehome automation devices 106. In particular, the home automation command may identify particular locations within the home, particularhome automation devices 106, and/or particular requested device states of thehome automation devices 106. For example, the home automation command may adjust lighting in particular rooms of the home, adjust the temperature, humidity, or other climate control settings in particular rooms of the home, activate or deactivate appliances or other equipment in the home, activate or query motion sensors, cameras, locks, or other security systems, or otherwise control thehome automation devices 106. Similar to the response text, thecomputing device 102 may generate the home automation command using thelanguage model 210. For example, thelanguage model 210 may include one or more predefined home automation commands associated with the pattern matching the current input text. The home automation command may be adjusted using parameters and other values included in the input text and/or determined based on thecontext model 214, as described above. - In
block 350, thecomputing device 102 presents the response text to the user. For example, thecomputing device 102 may present a follow-up question, an error message, or a successful response as described above in connection withblocks computing device 102 may use any technique for presenting the response text to the user, including displaying the response text on a display screen. In some embodiments, inblock 352 thecomputing device 102 may perform the home automation command determined as described above in connection withblock 348. Thecomputing device 102 may, for example, send the home automation command to thehome automation controller 104 and/or to one or morehome automation devices 106. By performing the home automation command, thecomputing device 102 executes the action requested by the user in the input text. In some embodiments, inblock 354 thecomputing device 102 outputs the response text using a text-to-speech synthesizer. Thus, by receiving speech input and outputting a speech response, thecomputing device 102 may allow conversational speech control of the home automation system. Additionally, although illustrated as generating response text for each command, it should be understood that in some embodiments thecomputing device 102 may “silently” perform the home automation command without generating response text or presenting any response text to the user. After presenting the response to the user, themethod 300 loops back toblocks FIG. 3A to continue processing input text. - Referring now to
FIG. 4 , in some embodiments thecomputing device 102 may execute amethod 400 for dialog pattern learning. Themethod 400 begins withblock 402, in which thecomputing device 102 receives input text from a user of thecomputing device 102. As described above in connection withblock 302 ofFIG. 3A , the input text may be indicative of a spoken user request. The user request may include part or all of a home automation system command, including one or more parameters associated with the home automation system command or query. Thecomputing device 102 may receive the input text, for example, from an automated speech recognition system. - In
block 404, thecomputing device 102 observes changes to the device state of one or morehome automation devices 106 generated in response to the user request. In some embodiments, thecomputing device 102 may perform learning by example. In those embodiments, the user may manually change the device state, for example by operating manual controls of one or morehome automation devices 106. For example, thecomputing device 102 may be placed into a learning by example mode, and thecomputing device 102 may observe as the user operates lights, climate control systems, or otherhome automation devices 106. As another example, thecomputing device 102 may observe the changes caused by a home automation command executed as described above in connection with themethod 300 ofFIGS. 3A and 3B . - In
block 406, thecomputing device 102 generates a pattern matching rule based on the input text and the observed change in state. The pattern matching rule may associate the particular names, phrasing, or other individualized language that the user associates with a particular location, device, and/or device state. For example, when in the learning mode, the input text may be “I am turning on the kitchen lights,” and thecomputing device 102 may associate the word “kitchen” with the location of thehome automation devices 106 operated by the user. As another example, the input text may be “flip on the lights,” and the home automation command may result in the lights being turned on, for example after presenting one or more follow-up questions to the user. In that example, thecomputing device 102 may respond with a follow-up question such as, “I don't understand, can you rephrase the request,” and the user may respond, “Turn on the lights.” In that example, thecomputing device 102 may generate a pattern matching the original input text “flip on the lights” that is associated with controlling thehome automation devices 106 to turn on the lights. - In
block 408, thecomputing device 102 updates thelanguage model 210 with the pattern matching rule generated inblock 406. Thus, thecomputing device 102 may adapt to the speech patterns of a particular user. After updating thelanguage model 210, themethod 400 loops back to block 402 to continue learning dialog patterns. - Referring now to
FIG. 5 , schematic diagram 500 illustrates a sample home automation control conversational dialog that may be executed by thecomputing device 102. The conversation starts with the user speakinginput text 502, “Homer, can you turn on the lights?” Note that theinput text 502 starts with an illustrative control word “Homer,” which thecomputing device 102 may use to gate the speech input. Thecomputing device 102 processes theinput text 502 and may determine that the input text matches a command to turn on lights, but that the input text does not include a location of the lights. As described above, thecomputing device 102 may determine the location of the lights based on thecontext model 214. In the illustrative embodiment, thecomputing device 102 determines that the user has requested to turn on the kitchen lights based on the location of the user. Thecomputing device 102 turns on the kitchen lights by controlling the appropriatehome automation devices 106. Thecomputing device 102 generates theresponse text 504, “Sure,” to indicate that the user request has been successfully processed. - The user next speaks
input text 506, “No, not those.” Thecomputing device 102 may determine using thecontext model 214 that the word “those” refers to the kitchen lights, which are the subject of the current conversation. Thecomputing device 102 turns off the kitchen lights by controlling the appropriatehome automation devices 106, and generates theresponse text 508, “Okay. Which lights do you want turned on, instead?” Theresponse text 508 thus may act as a follow-up question to determine the location of the lights in the original user request. - The user next speaks
input text 510, “The living room.” Thecomputing device 102 may identify that theinput text 510 includes a location, and may supply the rest of the home automation command (e.g., the requested devices and device state) based on thecontext model 214. Thus, thecomputing device 102 turns on the lights in the living room by controlling the appropriatehome automation devices 106, and generates theresponse text 512, “Sure,” to indicate that the user request has been successfully processed. - The user next speaks
input text 514, “Thanks Homer.” Note that theinput text 514 does not include a home automation command. However, thelanguage model 210 may include one or more patterns matching common conversational words and phrases such as “Thanks.” Thus, in the illustrative embodiment thecomputing device 102 matches theinput text 514 and then generatesresponse text 516, “Glad I could help you.” - Referring now to
FIG. 6 , schematic diagram 600 illustrates another sample home automation control conversational dialog that may be executed by thecomputing device 102. The diagram 600 illustrates configuration of a home automation system using thecomputing device 102. The conversation starts with the user speakinginput text 602, “Homer, I want to add a new device.” Thecomputing device 102 processes theinput text 602 and then puts the home automation system into a pairing mode or otherwise searches for a newhome automation device 106. Thecomputing device 102 further generatesresponse text 604, “Okay, go ahead and plug it in.” - In response to the
response text 604, the user activates a newhome automation device 106. In the illustrative embodiment, thehome automation device 106 is a contact sensor that may be used with thesecurity system 112. Thehome automation device 106 and/or thecomputing device 102 updates thecontext model 214 to indicate that the newhome automation device 106 has been added to the home automation system. In response to the change to thecontext model 214, thecomputing device 102 generatesresponse text 606, “I see a new contact sensor. Where are you going to place it?” Note that thecomputing device 102 may generate theresponse text 606 in response to a change to thecontext model 214 and not directly in response to speech input from the user. - The user next speaks
input text 608, “Backyard door.” Thecomputing device 102 may identify that theinput text 608 includes a location and may supply the rest of the home automation command (e.g., the device and/or requested device state) from thecontext model 214. In response to theinput text 608, thecomputing device 102 configures the newhome automation device 106 and/or thecontext model 214 of thecomputing device 102 based on the location provided by the user. Thecomputing device 102 generates theresponse text 610, “Sure,” to indicate that the user request has been successfully processed. - Next, the
computing device 102 generatesresponse text 612, “What do you want to happen when this is activated?” In some embodiments, thecomputing device 102 may generate theresponse text 612 based on thelanguage model 210. For example, theresponse text 612 may be part of a predefined dialog used for configuring newhome automation devices 106. In some embodiments, thecomputing device 102 may generate theresponse text 612 based on the state of the home automation system. For example, thecomputing device 102 may determine that the newhome automation device 106 does not have an associated action and then generate theresponse text 612 based on that determination. - The user next speaks
input text 614, “Turn on the lights if there is also motion detected in the kitchen.” Thecomputing device 102 processes theinput text 614 and then programs a corresponding rule for turning on the lights. For example, thecomputing device 102 may transmit a command or otherwise control thehome automation controller 104 and/or one or more appropriatehome automation devices 106 to program the rule. In response to theinput text 614, thecomputing device 102 generatesresponse text 616, “You got it!” to indicate that the user request has been successfully processed. - Referring now to
FIG. 7 , in use, thecomputing device 102 may execute amethod 700 for conversational control of a system. Themethod 700 begins withblock 702, in which thecomputing device 102 receives input text from a user of thecomputing device 102. The input text may be indicative of a spoken user request. The user request may include part or all of a system control command, including one or more parameters relating to the system control command or query. The system control command may relate to any system or systems controlled by thecomputing device 102, such as a home automation system (as described above in connection withFIGS. 3A, 3B and 4 ), an application automation system, a data exploration system, or other system. The input text is a natural language request and thus may not conform to any particular predefined format or syntax. - In
block 704, thecomputing device 102 updates thecontext model 214 based on the current state of the system being controlled. Thecomputing device 102 may use any technique to determine the current state of the system, for example receiving asynchronous updates from the system and/or polling the system. Additionally, although illustrated as updating thecontext model 214 sequentially as part of themethod 700, it should be understood that thecomputing device 102 may update thecontext model 214 asynchronously or at any other appropriate time. - In
block 706, thecomputing device 102 matches the input text against dialog patterns defined in a domain-specific language model 210 to identify a matching pattern. As described above, thelanguage model 210 includes a collection of patterns arranged hierarchically to define one or more dialogs. The patterns may be embodied as, for example, one or more regular-expression-like script language statements in a language such as ChatScript. Thelanguage model 210 may include patterns relating to the vocabulary, available system control commands, and other attributes of the particular domain. In some embodiments, inblock 708 thecomputing device 102 may apply an applicationcontrol language model 210. The applicationcontrol language model 210 may include dialog patterns and dictionaries relating to application automation commands, such as commands corresponding to user interface items such as buttons and menu items, as well as commands without a corresponding user interface item (e.g., complicated selections, compound commands, etc.). In some embodiments, inblock 710, thecomputing device 102 may apply a dataexploration language model 210. The dataexploration language model 210 may include dialog patterns and dictionaries relating to a particular data set, including potential queries of the dataset, available tables and data items in the data set, and other attribute of the data set. Parameters of the system control commands may include, for example, variables for comparison and selection of data records, filter commands, relational commands such as joins, aggregations, and other relational commands. - In block 712, the
computing device 102 resolves potential ambiguities or missing parameters in the requested system control command using thecurrent context model 214. In other words, thecomputing device 102 may determine one or more parameters of the system control command based on values selected from thecurrent context model 214. Thecontext model 214 may include conversational context data relating to the content of the current conversation, such as previous user requests and responses. In some embodiments, thecontext model 214 may include context data relating to the system being controlled. For example, for use with an application control system, thecontext model 214 may include data relating to the current context of the application, such as open files, the current view, recent commands, or other application context. As another example, for use with a data exploration system, thecontext model 214 may include parameters or other data relating to previous queries. Inblock 714, thecomputing device 102 determines whether the system control command is sufficiently resolved and valid. The system control command may be sufficiently resolved, for example, if thecomputing device 102 may generate a complete system control command based on the known parameters of the system control command. The system control command may be valid, for example, if the system control command is consistent with the system state described in thecurrent context model 214. If the command is resolved and valid, themethod 700 branches ahead to block 718, described below. If the command is not resolved and/or valid, themethod 700 advances to block 716. - In
block 716, thecomputing device 102 generates a follow-up response and presents the follow-up response to the user. Thecomputing device 102 may present the follow-up response using any appropriate technique, for example, by displaying the follow-up response on a display screen or by using a text-to-speech synthesizer. The follow-up response may be embodied as, for example, a follow-up question that prompts the user to supply additional information that may be used to resolve the system control command. As another example, the follow-up response may be embodied as an error message if the system control command is not valid. Thecomputing device 102 may determine the follow-up response using thelanguage model 210. For example, thelanguage model 210 may include one or more predefined follow-up responses associated with the current pattern matching the input text. After generating and presenting the follow-up response, themethod 700 loops back to block 702 to continue processing input text. - Referring back to block 714, if the system control command is sufficiently resolved and valid, the
method 700 branches to block 718, in which thecomputing device 102 generates response text and presents the response text to the user. The response text may be embodied as any natural language response that acknowledges or otherwise confirms that the user request has been successfully processed. Thecomputing device 102 may determine the response using thelanguage model 210. For example, thelanguage model 210 may include one or more predefined responses associated with the current pattern matching the input text. Thecomputing device 102 may use any technique for presenting the response text to the user, including displaying the response text on a display screen or outputting the response text using a text-to-speech synthesizer. - In
block 720, thecomputing device 102 generates and executes a system control command determined based on thelanguage model 210. The system control command may be embodied as any command, query, or other action that may be performed using the system controlled by thecomputing device 102. Similar to the response text, thecomputing device 102 may generate the system control command using thelanguage model 210. For example, thelanguage model 210 may include one or more predefined system control commands associated with patterns matching the input text. The home automation command may be adjusted using parameters and other values included in the input text and/or determined based on thecontext model 214, as described above. - In some embodiments, in
block 722 thecomputing device 102 may execute an application automation command. The application automation command may be embodied as any script, program, macro, or other command that may control one or more applications executed by thecomputing device 102. For example, the application automation command may be embodied as a Microsoft® Office Interop API command, an AppleScript™ command, a JavaScript™ command, or other application scripting command. In some embodiments, inblock 724, thecomputing device 102 may execute a database query for data exploration. The database query may be embodied as any database query, filter, or other command relating to the data set associated with thelanguage model 210. After executing the system control command, themethod 700 loops back to block 702 to continue processing input text. - Although illustrated as performing home automation, application automation, and data exploration control, it should be understood that in some embodiments the
system 100 may control other devices or collections of devices. For example, in some embodiments, thesystem 100 may be used to provide conversational control of devices participating in a local Internet of Things (IoT). - Illustrative examples of the devices, systems, and methods disclosed herein are provided below. An embodiment of the devices, systems, and methods may include any one or more, and any combination of, the examples described below.
- Example 1 includes a computing device for conversational system control, the computing device comprising a controller module to receive input text indicative of a spoken user request, wherein the spoken user request includes one or more parameters associated with a system control command; a conversational interface module to match the input text against a language model to identify a matching pattern, wherein the language model includes a plurality of patterns and wherein the matching pattern is associated with the system control command; and a context management module to determine one or more of the parameters of the system control command based on a context model in response to a match of the input text; wherein the conversational interface module is further to generate the system control command based on the language model in response to a determination of the one or more parameters; and wherein the controller module is further to perform the system control command.
- Example 2 includes the subject matter of Example 1, and wherein the matching pattern is further associated with text response; the conversational interface module is further to generate the text response based on the language model; and the controller module is further to present the text response to a user of the computing device.
- Example 3 includes the subject matter of any of Examples 1 and 2, and further including an automated speech recognition module to perform automated speech recognition to generate the input text; and a text-to-speech module to generate speech output based on the text response with a text-to-speech engine.
- Example 4 includes the subject matter of any of Examples 1-3, and wherein to perform the automated speech recognition comprises to perform the automated speech recognition in response to detection of a control word.
- Example 5 includes the subject matter of any of Examples 1-4, and wherein the context management module is further to update the context model based on the input text in response to the match of the input text.
- Example 6 includes the subject matter of any of Examples 1-5, and wherein the conversational interface module is further to (i) determine whether the system control command is sufficiently resolved in response to the determination of the one or more parameters of the system control command and (ii) generate a follow-up question for the text response with the language model in response to a determination that the system control command is not sufficiently resolved; wherein to generate the system control command based on the language model comprises to generate the system control command in response to a determination that the system control command is sufficiently resolved.
- Example 7 includes the subject matter of any of Examples 1-6, and wherein the system control command comprises a home automation command; the one or more parameters of the home automation command include a location, a device, or a requested device state; and the context model includes conversational context data, user context data, and home automation device context data.
- Example 8 includes the subject matter of any of Examples 1-7, and wherein to determine the one or more of the parameters of the home automation command based on the context model comprises to determine a location parameter, a device parameter, or a device state parameter of the home automation command based on the conversational context data.
- Example 9 includes the subject matter of any of Examples 1-8, and wherein to determine the one or more of the parameters of the home automation command based on the context model comprises to determine a location parameter of the home automation command based on the user context data.
- Example 10 includes the subject matter of any of Examples 1-9, and wherein to determine the one or more of the parameters of the home automation command based on the context model comprises to determine a device parameter or a device state parameter of the home automation command based on the home automation device context data.
- Example 11 includes the subject matter of any of Examples 1-10, and wherein the context management module is further to update the context model based on data received from a home automation system.
- Example 12 includes the subject matter of any of Examples 1-11, and wherein to update the context model based on the data received from the home automation system comprises to update the home automation device context data based on device data or device state data received from the home automation system.
- Example 13 includes the subject matter of any of Examples 1-12, and wherein the context management module is further to update the context model based on location data associated with a user of the computing device.
- Example 14 includes the subject matter of any of Examples 1-13, and, wherein the controller module is further to determine whether the home automation command is valid based on the home automation device context data of the context model in response to the determination of the one or more parameters of the home automation command; and the conversational interface module is to generate an error message for the text response with the language model in response to a determination that the home automation command is not valid; wherein to generate the system control command based on the language model comprises to generate the home automation command in response to a determination that the home automation command is valid.
- Example 15 includes the subject matter of any of Examples 1-14, and further including a learning module to observe a change in device state of a home automation system in response to performance of the home automation command; generate a new pattern based on the input text, wherein the pattern is associated with the home automation command; and update the language model to include the new pattern.
- Example 16 includes the subject matter of any of Examples 1-15, and wherein to receive the input text indicative of the spoken user request comprises to restrict the input text to a conversation with a single user selected from a plurality of users.
- Example 17 includes the subject matter of any of Examples 1-16, and wherein the controller module is further to receive non-textual input data indicative of a user request; and the conversational interface module is further to match the non-textual input data against the language model to identify the matching pattern.
- Example 18 includes the subject matter of any of Examples 1-17, and wherein the system control command comprises an application control command; the one or more parameters of the application control command include an application user command or an application object; and the context model includes conversational context data and application context data.
- Example 19 includes the subject matter of any of Examples 1-18, and wherein the system control command comprises a data exploration command; the one or more parameters of the data exploration command include a data query parameter; and the context model includes conversational context data and query context data.
- Example 20 includes a method for conversational system control, the method comprising receiving, by a computing device, input text indicative of a spoken user request, wherein the spoken user request includes one or more parameters associated with a system control command; matching, by the computing device, the input text against a language model to identify a matching pattern, wherein the language model includes a plurality of patterns and wherein the matching pattern is associated with the system control command; determining, by the computing device, one or more of the parameters of the system control command using a context model in response to matching the input text; generating, by the computing device, the system control command using the language model in response to determining the one or more parameters; and performing, by the computing device, the system control command.
- Example 21 includes the subject matter of Example 20, and further including generating, by the computing device, a text response using the language model in response to determining the one or more parameters, wherein the matching pattern is associated with the text response; and presenting, by the computing device, the text response to a user of the computing device.
- Example 22 includes the subject matter of any of Examples 20 and 21, and further including performing, by the computing device, automated speech recognition to generate the input text; and generating, by the computing device, speech output based on the text response using a text-to-speech engine.
- Example 23 includes the subject matter of any of Examples 20-22, and wherein performing the automated speech recognition comprises performing the automated speech recognition in response to detecting a control word.
- Example 24 includes the subject matter of any of Examples 20-23, and further including updating, by the computing device, the context model based on the input text in response to matching the input text.
- Example 25 includes the subject matter of any of Examples 20-24, and further including determining, by the computing device, whether the system control command is sufficiently resolved in response to determining the one or more parameters of the system control command; and generating, by the computing device, a follow-up question for the text response using the language model in response to determining that the system control command is not sufficiently resolved; wherein generating the system control command based on the language model comprises generating the system control command in response to determining that the system control command is sufficiently resolved.
- Example 26 includes the subject matter of any of Examples 20-25, and wherein the system control command comprises a home automation command; the one or more parameters of the home automation command include a location, a device, or a requested device state; and the context model includes conversational context data, user context data, and home automation device context data.
- Example 27 includes the subject matter of any of Examples 20-26, and wherein determining the one or more of the parameters of the home automation command using the context model comprises determining a location parameter, a device parameter, or a device state parameter of the home automation command based on the conversational context data.
- Example 28 includes the subject matter of any of Examples 20-27, and wherein determining the one or more of the parameters of the home automation command using the context model comprises determining a location parameter of the home automation command based on the user context data.
- Example 29 includes the subject matter of any of Examples 20-28, and wherein determining the one or more of the parameters of the home automation command using the context model comprises determining a device parameter or a device state parameter of the home automation command based on the home automation device context data.
- Example 30 includes the subject matter of any of Examples 20-29, and further including updating, by the computing device, the context model based on data received from a home automation system.
- Example 31 includes the subject matter of any of Examples 20-30, and wherein updating the context model based on the data received from the home automation system comprises updating the home automation device context data based on device data or device state data received from the home automation system.
- Example 32 includes the subject matter of any of Examples 20-31, and further including updating, by the computing device, the context model based on location data associated with a user of the computing device.
- Example 33 includes the subject matter of any of Examples 20-32, and further including determining, by the computing device, whether the home automation command is valid based on the home automation device context data of the context model in response to determining the one or more parameters of the home automation command; and generating, by the computing device, an error message for the text response using the language model in response to determining that the home automation command is not valid; wherein generating the system control command based on the language model comprises generating the home automation command in response to determining that the home automation command is valid.
- Example 34 includes the subject matter of any of Examples 20-33, and further including observing, by the computing device, a change in device state of a home automation system in response to performing the home automation command; generating, by the computing device, a new pattern based on the input text, wherein the pattern is associated with the home automation command; and updating, by the computing device, the language model to include the new pattern.
- Example 35 includes the subject matter of any of Examples 20-34, and wherein receiving the input text indicative of the spoken user request comprises restricting the input text to a conversation with a single user selected from a plurality of users.
- Example 36 includes the subject matter of any of Examples 20-35, and, further including receiving, by the computing device, non-textual input data indicative of a user request; and matching, by the computing device, the non-textual input data against the language model to identify the matching pattern.
- Example 37 includes the subject matter of any of Examples 20-36, and wherein the system control command comprises an application control command; the one or more parameters of the application control command include an application user command or an application object; and the context model includes conversational context data and application context data.
- Example 38 includes the subject matter of any of Examples 20-37, and wherein the system control command comprises a data exploration command; the one or more parameters of the data exploration command include a data query parameter; and the context model includes conversational context data and query context data.
- Example 39 includes a computing device comprising a processor; and a memory having stored therein a plurality of instructions that when executed by the processor cause the computing device to perform the method of any of Examples 20-38.
- Example 40 includes one or more machine readable storage media comprising a plurality of instructions stored thereon that in response to being executed result in a computing device performing the method of any of Examples 20-38.
- Example 41 includes a computing device comprising means for performing the method of any of Examples 20-38.
- Example 42 includes a computing device for conversational system control, the computing device comprising means for receiving input text indicative of a spoken user request, wherein the spoken user request includes one or more parameters associated with a system control command; means for matching the input text against a language model to identify a matching pattern, wherein the language model includes a plurality of patterns and wherein the matching pattern is associated with the system control command; means for determining one or more of the parameters of the system control command using a context model in response to matching the input text; means for generating the system control command using the language model in response to determining the one or more parameters; and means for performing the system control command.
- Example 43 includes the subject matter of Example 42, and further comprising means for generating a text response using the language model in response to determining the one or more parameters, wherein the matching pattern is associated with the text response; and means for presenting the text response to a user of the computing device.
- Example 44 includes the subject matter of any of Examples 42 and 43, and further including means for performing automated speech recognition to generate the input text; and means for generating speech output based on the text response using a text-to-speech engine.
- Example 45 includes the subject matter of any of Examples 42-44, and wherein the means for performing the automated speech recognition comprises means for performing the automated speech recognition in response to detecting a control word.
- Example 46 includes the subject matter of any of Examples 42-45, and further including means for updating the context model based on the input text in response to matching the input text.
- Example 47 includes the subject matter of any of Examples 42-46, and further including means for determining whether the system control command is sufficiently resolved in response to determining the one or more parameters of the system control command; and means for generating a follow-up question for the text response using the language model in response to determining that the system control command is not sufficiently resolved; wherein the means for generating the system control command based on the language model comprises means for generating the system control command in response to determining that the system control command is sufficiently resolved.
- Example 48 includes the subject matter of any of Examples 42-47, and wherein the system control command comprises a home automation command; the one or more parameters of the home automation command include a location, a device, or a requested device state; and the context model includes conversational context data, user context data, and home automation device context data.
- Example 49 includes the subject matter of any of Examples 42-48, and wherein the means for determining the one or more of the parameters of the home automation command using the context model comprises means for determining a location parameter, a device parameter, or a device state parameter of the home automation command based on the conversational context data.
- Example 50 includes the subject matter of any of Examples 42-49, and wherein the means for determining the one or more of the parameters of the home automation command using the context model comprises means for determining a location parameter of the home automation command based on the user context data.
- Example 51 includes the subject matter of any of Examples 42-50, and wherein the means for determining the one or more of the parameters of the home automation command using the context model comprises means for determining a device parameter or a device state parameter of the home automation command based on the home automation device context data.
- Example 52 includes the subject matter of any of Examples 42-51, and further including means for updating the context model based on data received from a home automation system.
- Example 53 includes the subject matter of any of Examples 42-52, and wherein the means for updating the context model based on the data received from the home automation system comprises means for updating the home automation device context data based on device data or device state data received from the home automation system.
- Example 54 includes the subject matter of any of Examples 42-53, and further including means for updating the context model based on location data associated with a user of the computing device.
- Example 55 includes the subject matter of any of Examples 42-54, and further including means for determining whether the home automation command is valid based on the home automation device context data of the context model in response to determining the one or more parameters of the home automation command; and means for generating an error message for the text response using the language model in response to determining that the home automation command is not valid; wherein the means for generating the system control command based on the language model comprises means for generating the home automation command in response to determining that the home automation command is valid.
- Example 56 includes the subject matter of any of Examples 42-55, and further including means for observing a change in device state of a home automation system in response to performing the home automation command; means for generating a new pattern based on the input text, wherein the pattern is associated with the home automation command; and means for updating the language model to include the new pattern.
- Example 57 includes the subject matter of any of Examples 42-56, and wherein the means for receiving the input text indicative of the spoken user request comprises means for restricting the input text to a conversation with a single user selected from a plurality of users.
- Example 58 includes the subject matter of any of Examples 42-57, and further including means for receiving non-textual input data indicative of a user request; and means for matching the non-textual input data against the language model to identify the matching pattern.
- Example 59 includes the subject matter of any of Examples 42-58, and wherein the system control command comprises an application control command; the one or more parameters of the application control command include an application user command or an application object; and the context model includes conversational context data and application context data.
- Example 60 includes the subject matter of any of Examples 42-59, and wherein the system control command comprises a data exploration command; the one or more parameters of the data exploration command include a data query parameter; and the context model includes conversational context data and query context data.
Claims (13)
1-20. (canceled)
21. A consumer electronic device for user voice command-related processing for multiple users, the user voice command-related processing being associated with control of one or more home automation devices, the consumer electronic device also being usable with one or more servers, the consumer electronic device comprising:
audio sensor circuitry to generate, when the consumer electronic device is in operation, audio signal data for use in detecting voice commands by the multiple users, the audio signal data to be used to generate respective text associated with respective of the multiple users, the respective text being indicative of the voice commands, the voice commands including one or more voice commands by one or more of the multiple users, the respective text including certain text indicative of the one or more voice commands;
processor circuitry for use in updating, at least in part, when the consumer electronic device is in the operation, particularized speech pattern modeling data associated with the multiple users to accommodate individualized respective speech patterns of the multiple users, the particularized speech pattern modeling data to be used in speech pattern matching operations associated with the multiple users;
audio output circuitry to generate one or more synthesized speech responses related to the one or more voice commands;
communication circuitry for use, when the consumer electronic device is in the operation, in communicating with the one or more home automation devices and/or the one or more servers;
wherein:
the certain text includes one or more parameters associated with one or more control commands for the one or more home automation devices;
the certain text is to be matched against dialog patterns to identify a matching pattern;
the matching pattern is associated with the one or more control commands; and
the one or more control commands, when executed by the one or more home automation devices, result in one or more changes to:
one or more settings of the one or more home automation devices; and/or
one or more activation states of the one or more home automation devices.
22. The consumer electronic device of claim 21 , wherein:
the one or more servers are to:
generate the certain text;
match the certain text against the dialog patterns to identify the matching pattern; and
generate the one or more control commands.
23. The consumer electronic device of claim 21 , wherein:
the voice commands are to be gated based upon control word detection.
24. The consumer electronic device of claim 21 , wherein:
the consumer electronic device comprises:
a cellular telephone; and
a display screen for use in displaying, when the consumer electronic device is in the operation, text response related to the one or more voice commands;
the communication circuitry is for use, when the consumer electronic device is in the operation, in wireless communication; and
the wireless communication comprises:
WiFi communication;
Bluetooth communication;
Zigbee communication; and/or
Zwave communication.
25. A consumer electronic device for user voice command-related processing for multiple users, the user voice command-related processing being associated with control of one or more home automation devices, the consumer electronic device also being usable with one or more servers, the consumer electronic device comprising:
audio sensor means for generating, when the consumer electronic device is in operation, audio signal data for use in detecting voice commands by the multiple users, the audio signal data to be used to generate respective text associated with respective of the multiple users, the respective text being indicative of the voice commands, the voice commands including one or more voice commands by one or more of the multiple users, the respective text including certain text indicative of the one or more voice commands;
processor means for updating, at least in part, when the consumer electronic device is in the operation, particularized speech pattern modeling data associated with the multiple users to accommodate individualized respective speech patterns of the multiple users, the particularized speech pattern modeling data to be used in speech pattern matching operations associated with the multiple users;
audio output means for generating one or more synthesized speech responses related to the one or more voice commands;
communication means for communicating, when the consumer electronic device is in the operation, with the one or more home automation devices and/or the one or more servers;
wherein:
the certain text includes one or more parameters associated with one or more control commands for the one or more home automation devices;
the certain text is to be matched against dialog patterns to identify a matching pattern;
the matching pattern is associated with the one or more control commands; and
the one or more control commands, when executed by the one or more home automation devices, result in one or more changes to:
one or more settings of the one or more home automation devices; and/or
one or more activation states of the one or more home automation devices.
26. The consumer electronic device of claim 25 , wherein:
the one or more servers are to:
generate the certain text;
match the certain text against the dialog patterns to identify the matching pattern; and
generate the one or more control commands.
27. The consumer electronic device of claim 25 , wherein:
the voice commands are to be gated based upon control word detection.
28. The consumer electronic device of claim 25 , wherein:
the consumer electronic device comprises:
a cellular telephone; and
display means for use in displaying, when the consumer electronic device is in the operation, text response related to the one or more voice commands;
the communication means is for use, when the consumer electronic device is in the operation, in wireless communication; and
the wireless communication comprises:
WiFi communication;
Bluetooth communication;
Zigbee communication; and/or
Zwave communication.
29. One or more non-transitory machine-readable storage media storing instructions for being executed by a consumer electronic device, the consumer electronic device being for user voice command-related processing for multiple users, the user voice command-related processing being associated with control of one or more home automation devices, the consumer electronic device also being usable with one or more servers, the instructions when executed by the consumer electronic device resulting in the consumer electronic device being configured for performance of operations comprising:
generating, by audio sensor circuitry of the consumer electronic device, audio signal data for use in detecting voice commands by the multiple users, the audio signal data to be used to generate respective text associated with respective of the multiple users, the respective text being indicative of the voice commands, the voice commands including one or more voice commands by one or more of the multiple users, the respective text including certain text indicative of the one or more voice commands;
updating, at least in part, by processor circuitry of the consumer electronic device, particularized speech pattern modeling data associated with the multiple users to accommodate individualized respective speech patterns of the multiple users, the particularized speech pattern modeling data to be used in speech pattern matching operations associated with the multiple users;
generating, by audio output circuitry of the consumer electronic device, one or more synthesized speech responses related to the one or more voice commands;
communicating, via communication circuitry of the consumer electronic device, with the one or more home automation devices and/or the one or more servers;
wherein:
the certain text includes one or more parameters associated with one or more control commands for the one or more home automation devices;
the certain text is to be matched against dialog patterns to identify a matching pattern;
the matching pattern is associated with the one or more control commands; and
the one or more control commands, when executed by the one or more home automation devices, result in one or more changes to:
one or more settings of the one or more home automation devices; and/or
one or more activation states of the one or more home automation devices.
30. The one or more non-transitory machine-readable storage media of claim 29 , wherein:
the one or more servers are to:
generate the certain text;
match the certain text against the dialog patterns to identify the matching pattern; and
generate the one or more control commands.
31. The one or more non-transitory machine-readable storage media of claim 29 , wherein:
the voice commands are to be gated based upon control word detection.
32. The one or more non-transitory machine-readable storage media of claim 29 , wherein:
the consumer electronic device comprises:
a cellular telephone; and
a display screen for use in displaying, when the consumer electronic device is in operation, text response related to the one or more voice commands;
the communication circuitry is for use, when the consumer electronic device is in the operation, in wireless communication; and
the wireless communication comprises:
WiFi communication;
Bluetooth communication;
Zigbee communication; and/or
Zwave communication.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/352,018 US20190391541A1 (en) | 2015-06-25 | 2019-03-13 | Technologies for conversational interfaces for system control |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/750,482 US10274911B2 (en) | 2015-06-25 | 2015-06-25 | Conversational interface for matching text of spoken input based on context model |
US16/352,018 US20190391541A1 (en) | 2015-06-25 | 2019-03-13 | Technologies for conversational interfaces for system control |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/750,482 Continuation US10274911B2 (en) | 2015-06-25 | 2015-06-25 | Conversational interface for matching text of spoken input based on context model |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190391541A1 true US20190391541A1 (en) | 2019-12-26 |
Family
ID=57585204
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/750,482 Active 2035-12-12 US10274911B2 (en) | 2015-06-25 | 2015-06-25 | Conversational interface for matching text of spoken input based on context model |
US16/352,018 Abandoned US20190391541A1 (en) | 2015-06-25 | 2019-03-13 | Technologies for conversational interfaces for system control |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/750,482 Active 2035-12-12 US10274911B2 (en) | 2015-06-25 | 2015-06-25 | Conversational interface for matching text of spoken input based on context model |
Country Status (4)
Country | Link |
---|---|
US (2) | US10274911B2 (en) |
EP (1) | EP3314876B1 (en) |
CN (1) | CN107667318B (en) |
WO (1) | WO2016209489A1 (en) |
Families Citing this family (71)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10564614B2 (en) | 2014-01-31 | 2020-02-18 | Vivint, Inc. | Progressive profiling in an automation system |
US11044114B2 (en) | 2014-01-31 | 2021-06-22 | Vivint, Inc. | Rule-based graphical conversational user interface for security and automation system |
US10572810B2 (en) | 2015-01-07 | 2020-02-25 | Microsoft Technology Licensing, Llc | Managing user interaction for input understanding determinations |
US10274911B2 (en) * | 2015-06-25 | 2019-04-30 | Intel Corporation | Conversational interface for matching text of spoken input based on context model |
US10249297B2 (en) | 2015-07-13 | 2019-04-02 | Microsoft Technology Licensing, Llc | Propagating conversational alternatives using delayed hypothesis binding |
JP6461738B2 (en) * | 2015-07-24 | 2019-01-30 | シャープ株式会社 | Cooperation system and device control server |
US10541958B2 (en) * | 2015-08-05 | 2020-01-21 | Facebook, Inc. | Controlling a device cloud |
US10567479B2 (en) | 2015-08-05 | 2020-02-18 | Facebook, Inc. | Managing a device cloud |
US9946862B2 (en) * | 2015-12-01 | 2018-04-17 | Qualcomm Incorporated | Electronic device generating notification based on context data in response to speech phrase from user |
EP3395019B1 (en) * | 2015-12-21 | 2022-03-30 | Google LLC | Automatic suggestions and other content for messaging applications |
CN108781175B (en) | 2015-12-21 | 2021-09-21 | 谷歌有限责任公司 | Method, medium, and system for automatic suggestion of message exchange contexts |
US10567312B2 (en) | 2016-04-11 | 2020-02-18 | Facebook, Inc. | Techniques for messaging bot controls based on machine-learning user intent detection |
US10831802B2 (en) * | 2016-04-11 | 2020-11-10 | Facebook, Inc. | Techniques to respond to user requests using natural-language machine learning based on example conversations |
US11226598B2 (en) | 2016-05-04 | 2022-01-18 | Johnson Controls Technology Company | Building system with user presentation composition based on building context |
US11774920B2 (en) | 2016-05-04 | 2023-10-03 | Johnson Controls Technology Company | Building system with user presentation composition based on building context |
WO2017192770A1 (en) | 2016-05-04 | 2017-11-09 | Johnson Controls Technology Company | Systems and methods for agent interaction with building management system |
US11226597B2 (en) * | 2016-07-11 | 2022-01-18 | Johnson Controls Technology Company | Systems and methods for interaction with a building management system |
US10387461B2 (en) | 2016-08-16 | 2019-08-20 | Google Llc | Techniques for suggesting electronic messages based on user activity and other context |
CN107765838A (en) * | 2016-08-18 | 2018-03-06 | 北京北信源软件股份有限公司 | Man-machine interaction householder method and device |
US10573299B2 (en) * | 2016-08-19 | 2020-02-25 | Panasonic Avionics Corporation | Digital assistant and associated methods for a transportation vehicle |
US10446137B2 (en) * | 2016-09-07 | 2019-10-15 | Microsoft Technology Licensing, Llc | Ambiguity resolving conversational understanding system |
US10511450B2 (en) | 2016-09-20 | 2019-12-17 | Google Llc | Bot permissions |
US10015124B2 (en) | 2016-09-20 | 2018-07-03 | Google Llc | Automatic response suggestions based on images received in messaging applications |
CN117634495A (en) | 2016-09-20 | 2024-03-01 | 谷歌有限责任公司 | Suggested response based on message decal |
US10356029B2 (en) | 2016-09-21 | 2019-07-16 | Facebook, Inc. | Methods and systems for presenting modules in an inbox interface |
US11233760B2 (en) | 2016-09-21 | 2022-01-25 | Facebook, Inc. | Module ranking for a modular inbox |
US10416846B2 (en) | 2016-11-12 | 2019-09-17 | Google Llc | Determining graphical element(s) for inclusion in an electronic communication |
US11354089B2 (en) * | 2016-12-09 | 2022-06-07 | Robert Bosch Gmbh | System and method for dialog interaction in distributed automation systems |
US10586528B2 (en) * | 2017-02-02 | 2020-03-10 | Adobe Inc. | Domain-specific speech recognizers in a digital medium environment |
US20180231954A1 (en) * | 2017-02-14 | 2018-08-16 | Honeywell International Inc. | Apparatus and method supporting an interactive chat feature for relaying on-demand information to a user from an industrial process control and automation system |
US10057125B1 (en) * | 2017-04-17 | 2018-08-21 | Essential Products, Inc. | Voice-enabled home setup |
US10353480B2 (en) * | 2017-04-17 | 2019-07-16 | Essential Products, Inc. | Connecting assistant device to devices |
US10013979B1 (en) * | 2017-04-17 | 2018-07-03 | Essential Products, Inc. | Expanding a set of commands to control devices in an environment |
US10176807B2 (en) | 2017-04-17 | 2019-01-08 | Essential Products, Inc. | Voice setup instructions |
US10355931B2 (en) | 2017-04-17 | 2019-07-16 | Essential Products, Inc. | Troubleshooting voice-enabled home setup |
US10860854B2 (en) | 2017-05-16 | 2020-12-08 | Google Llc | Suggested actions for images |
US10348658B2 (en) | 2017-06-15 | 2019-07-09 | Google Llc | Suggested items for use with embedded applications in chat conversations |
US10404636B2 (en) | 2017-06-15 | 2019-09-03 | Google Llc | Embedded programs and interfaces for chat conversations |
EP3646319A4 (en) * | 2017-06-28 | 2020-12-23 | Mars Advertising, Inc. D/B/A The Mars Agency Group | Voice interface shopping system |
CN107179700A (en) * | 2017-07-03 | 2017-09-19 | 杭州善居科技有限公司 | A kind of intelligent home control system and method based on Alljoyn and machine learning |
WO2019010309A1 (en) * | 2017-07-05 | 2019-01-10 | Johnson Controls Technology Company | Systems and methods for conversational interaction with a building automation system |
US11120803B2 (en) | 2017-07-05 | 2021-09-14 | Johnson Controls Tyco IP Holdings LLP | Building automation system with NLP service abstraction |
WO2019032994A1 (en) * | 2017-08-10 | 2019-02-14 | Facet Labs, Llc | Oral, facial and gesture communication devices and computing architecture for interacting with digital media content |
KR20190019748A (en) * | 2017-08-18 | 2019-02-27 | 삼성전자주식회사 | Method and apparatus for generating natural language |
US10938858B2 (en) * | 2017-09-29 | 2021-03-02 | Endgame, Inc. | Chatbot interface for network security software application |
US10891526B2 (en) | 2017-12-22 | 2021-01-12 | Google Llc | Functional image archiving |
GB2572175B (en) * | 2018-03-21 | 2022-10-12 | Emotech Ltd | Processing a command |
CN112313924A (en) * | 2018-05-07 | 2021-02-02 | 谷歌有限责任公司 | Providing a composite graphical assistant interface for controlling various connected devices |
US10997963B1 (en) * | 2018-05-17 | 2021-05-04 | Amazon Technologies, Inc. | Voice based interaction based on context-based directives |
KR20190133100A (en) * | 2018-05-22 | 2019-12-02 | 삼성전자주식회사 | Electronic device and operating method for outputting a response for a voice input, by using application |
US10607610B2 (en) | 2018-05-29 | 2020-03-31 | Nortek Security & Control Llc | Audio firewall |
US10832010B2 (en) * | 2018-06-05 | 2020-11-10 | International Business Machines Corporation | Training of conversational agent using natural language |
US20190390866A1 (en) | 2018-06-22 | 2019-12-26 | Honeywell International Inc. | Building management system with natural language interface |
US10719066B2 (en) * | 2018-08-29 | 2020-07-21 | Rockwell Automation Technologies, Inc. | Audio recognition-based industrial automation control |
CN109065048A (en) * | 2018-09-12 | 2018-12-21 | 深圳市卓翼科技股份有限公司 | Sound control method and device |
US11784845B2 (en) | 2018-09-28 | 2023-10-10 | Qualcomm Incorporated | System and method for disambiguation of Internet-of-Things devices |
KR102543676B1 (en) * | 2018-10-08 | 2023-06-14 | 구글 엘엘씨 | Summarily conveying smart appliance statuses |
CN109379261B (en) * | 2018-11-30 | 2021-09-14 | 北京小米智能科技有限公司 | Control method, device, system, equipment and storage medium of intelligent equipment |
CN111312253A (en) * | 2018-12-11 | 2020-06-19 | 青岛海尔洗衣机有限公司 | Voice control method, cloud server and terminal equipment |
US20220012547A1 (en) * | 2018-12-24 | 2022-01-13 | Samsung Electronics Co., Ltd. | Method and apparatus for controlling smart device to perform corresponding operations |
US11132510B2 (en) * | 2019-01-30 | 2021-09-28 | International Business Machines Corporation | Intelligent management and interaction of a communication agent in an internet of things environment |
US11955120B1 (en) * | 2019-01-31 | 2024-04-09 | Alan AI, Inc. | Systems and methods for integrating voice controls into applications |
US11954453B2 (en) | 2019-03-12 | 2024-04-09 | International Business Machines Corporation | Natural language generation by an edge computing device |
EP3836043A1 (en) * | 2019-12-11 | 2021-06-16 | Carrier Corporation | A method and an equipment for configuring a service |
WO2021140447A1 (en) * | 2020-01-06 | 2021-07-15 | 7Hugs Labs | System and method for controlling a plurality of devices |
CN113823280A (en) * | 2020-06-19 | 2021-12-21 | 华为技术有限公司 | Intelligent device control method, electronic device and system |
US20220215833A1 (en) * | 2021-01-07 | 2022-07-07 | Lenovo (Singapore) Pte. Ltd. | Method and device for converting spoken words to text form |
US11790168B2 (en) * | 2021-01-29 | 2023-10-17 | Ncr Corporation | Natural language and messaging system integrated group assistant |
US11856038B2 (en) | 2021-05-27 | 2023-12-26 | International Business Machines Corporation | Cognitively learning to generate scripts that simulate live-agent actions and responses in synchronous conferencing |
CN114301723A (en) * | 2021-12-21 | 2022-04-08 | 珠海格力电器股份有限公司 | Intelligent household control system and method |
WO2023170075A1 (en) * | 2022-03-09 | 2023-09-14 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Contextual control |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6408272B1 (en) * | 1999-04-12 | 2002-06-18 | General Magic, Inc. | Distributed voice user interface |
US20130110511A1 (en) * | 2011-10-31 | 2013-05-02 | Telcordia Technologies, Inc. | System, Method and Program for Customized Voice Communication |
US20130183944A1 (en) * | 2012-01-12 | 2013-07-18 | Sensory, Incorporated | Information Access and Device Control Using Mobile Phones and Audio in the Home Environment |
US20130297313A1 (en) * | 2010-05-26 | 2013-11-07 | Google Inc. | Acoustic model adaptation using geographic information |
US10274911B2 (en) * | 2015-06-25 | 2019-04-30 | Intel Corporation | Conversational interface for matching text of spoken input based on context model |
Family Cites Families (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE69712485T2 (en) * | 1997-10-23 | 2002-12-12 | Sony Int Europe Gmbh | Voice interface for a home network |
US6754626B2 (en) * | 2001-03-01 | 2004-06-22 | International Business Machines Corporation | Creating a hierarchical tree of language models for a dialog system based on prompt and dialog context |
US8068881B2 (en) | 2002-08-09 | 2011-11-29 | Avon Associates, Inc. | Voice controlled multimedia and communications system |
WO2004045154A1 (en) * | 2002-11-13 | 2004-05-27 | Intel Corporation | Multi-modal web interaction over wireless network |
US8595642B1 (en) * | 2007-10-04 | 2013-11-26 | Great Northern Research, LLC | Multiple shell multi faceted graphical user interface |
US9129601B2 (en) * | 2008-11-26 | 2015-09-08 | At&T Intellectual Property I, L.P. | System and method for dialog modeling |
US20120089392A1 (en) * | 2010-10-07 | 2012-04-12 | Microsoft Corporation | Speech recognition user interface |
US9214157B2 (en) * | 2011-12-06 | 2015-12-15 | At&T Intellectual Property I, L.P. | System and method for machine-mediated human-human conversation |
US9324323B1 (en) * | 2012-01-13 | 2016-04-26 | Google Inc. | Speech recognition using topic-specific language models |
US9026441B2 (en) | 2012-02-29 | 2015-05-05 | Nant Holdings Ip, Llc | Spoken control for user construction of complex behaviors |
US8346563B1 (en) | 2012-04-10 | 2013-01-01 | Artificial Solutions Ltd. | System and methods for delivering advanced natural language interaction applications |
US9053708B2 (en) * | 2012-07-18 | 2015-06-09 | International Business Machines Corporation | System, method and program product for providing automatic speech recognition (ASR) in a shared resource environment |
US8831957B2 (en) * | 2012-08-01 | 2014-09-09 | Google Inc. | Speech recognition models based on location indicia |
US9230560B2 (en) * | 2012-10-08 | 2016-01-05 | Nant Holdings Ip, Llc | Smart home automation systems and methods |
US9085303B2 (en) * | 2012-11-15 | 2015-07-21 | Sri International | Vehicle personal assistant |
US20140156281A1 (en) | 2012-12-03 | 2014-06-05 | Qualcomm Incorporated | Voice-controlled configuration of an automation system |
KR101759009B1 (en) * | 2013-03-15 | 2017-07-17 | 애플 인크. | Training an at least partial voice command system |
WO2014197335A1 (en) | 2013-06-08 | 2014-12-11 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US9189742B2 (en) * | 2013-11-20 | 2015-11-17 | Justin London | Adaptive virtual intelligent agent |
US9939823B2 (en) * | 2014-06-05 | 2018-04-10 | Wise Spaces Ltd. | Home automation control system |
CN104657104A (en) * | 2015-01-22 | 2015-05-27 | 苏州工业职业技术学院 | PC (personal computer)-oriented embedded non-specific voice communication system |
-
2015
- 2015-06-25 US US14/750,482 patent/US10274911B2/en active Active
-
2016
- 2016-05-25 WO PCT/US2016/034053 patent/WO2016209489A1/en unknown
- 2016-05-25 CN CN201680030149.1A patent/CN107667318B/en active Active
- 2016-05-25 EP EP16814948.2A patent/EP3314876B1/en active Active
-
2019
- 2019-03-13 US US16/352,018 patent/US20190391541A1/en not_active Abandoned
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6408272B1 (en) * | 1999-04-12 | 2002-06-18 | General Magic, Inc. | Distributed voice user interface |
US20130297313A1 (en) * | 2010-05-26 | 2013-11-07 | Google Inc. | Acoustic model adaptation using geographic information |
US20130110511A1 (en) * | 2011-10-31 | 2013-05-02 | Telcordia Technologies, Inc. | System, Method and Program for Customized Voice Communication |
US20130183944A1 (en) * | 2012-01-12 | 2013-07-18 | Sensory, Incorporated | Information Access and Device Control Using Mobile Phones and Audio in the Home Environment |
US10274911B2 (en) * | 2015-06-25 | 2019-04-30 | Intel Corporation | Conversational interface for matching text of spoken input based on context model |
Also Published As
Publication number | Publication date |
---|---|
CN107667318B (en) | 2022-02-15 |
US10274911B2 (en) | 2019-04-30 |
EP3314876B1 (en) | 2023-05-24 |
WO2016209489A1 (en) | 2016-12-29 |
EP3314876A4 (en) | 2019-02-20 |
CN107667318A (en) | 2018-02-06 |
US20160378080A1 (en) | 2016-12-29 |
EP3314876A1 (en) | 2018-05-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20190391541A1 (en) | Technologies for conversational interfaces for system control | |
US11422772B1 (en) | Creating scenes from voice-controllable devices | |
US11699448B2 (en) | Intelligent assistant for home automation | |
JP6683893B2 (en) | Processing voice commands based on device topology | |
CN112074898B (en) | Machine generation of context-free grammar for intent reasoning | |
JP6752870B2 (en) | Methods and systems for controlling artificial intelligence devices using multiple wake words | |
JP7262532B2 (en) | VOICE INTERACTIVE PROCESSING METHOD, APPARATUS, ELECTRONIC DEVICE, STORAGE MEDIUM AND PROGRAM | |
Ruslan et al. | Development of multilanguage voice control for smart home with IoT | |
KR102487078B1 (en) | The hub device, multi device system comprising the hub device and a plurality of devices and method operating the same |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |