US10657963B2 - Method and system for processing user command to provide and adjust operation of electronic device by analyzing presentation of user speech - Google Patents

Method and system for processing user command to provide and adjust operation of electronic device by analyzing presentation of user speech Download PDF

Info

Publication number
US10657963B2
US10657963B2 US15/969,394 US201815969394A US10657963B2 US 10657963 B2 US10657963 B2 US 10657963B2 US 201815969394 A US201815969394 A US 201815969394A US 10657963 B2 US10657963 B2 US 10657963B2
Authority
US
United States
Prior art keywords
user
option
voice input
options
user command
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US15/969,394
Other languages
English (en)
Other versions
US20180322872A1 (en
Inventor
Seijin Cha
Eonjoung Choi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ly Corp
Naver Corp
Original Assignee
Line Corp
Naver Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Line Corp, Naver Corp filed Critical Line Corp
Assigned to NAVER CORPORATION, LINE CORPORATION reassignment NAVER CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHOI, EONJOUNG, CHA, SEIJIN
Publication of US20180322872A1 publication Critical patent/US20180322872A1/en
Application granted granted Critical
Publication of US10657963B2 publication Critical patent/US10657963B2/en
Assigned to A HOLDINGS CORPORATION reassignment A HOLDINGS CORPORATION CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: LINE CORPORATION
Assigned to LINE CORPORATION reassignment LINE CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: A HOLDINGS CORPORATION
Assigned to LINE CORPORATION reassignment LINE CORPORATION CORRECTIVE ASSIGNMENT TO CORRECT THE SPELLING OF THE ASSIGNEES CITY IN THE ADDRESS SHOULD BE TOKYO, JAPAN PREVIOUSLY RECORDED AT REEL: 058597 FRAME: 0303. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: A HOLDINGS CORPORATION
Assigned to A HOLDINGS CORPORATION reassignment A HOLDINGS CORPORATION CORRECTIVE ASSIGNMENT TO CORRECT THE THE CITY SHOULD BE SPELLED AS TOKYO PREVIOUSLY RECORDED AT REEL: 058597 FRAME: 0141. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: LINE CORPORATION
Assigned to Z INTERMEDIATE GLOBAL CORPORATION reassignment Z INTERMEDIATE GLOBAL CORPORATION CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: LINE CORPORATION
Assigned to LY CORPORATION reassignment LY CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Z INTERMEDIATE GLOBAL CORPORATION
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/04—Segmentation; Word boundary detection
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/26—Devices for calling a subscriber
    • H04M1/27—Devices whereby a plurality of signals may be stored simultaneously
    • H04M1/271—Devices whereby a plurality of signals may be stored simultaneously controlled by voice recognition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/26—Devices for calling a subscriber
    • H04M1/27—Devices whereby a plurality of signals may be stored simultaneously
    • H04M1/274—Devices whereby a plurality of signals may be stored simultaneously with provision for storing more than one subscriber number at a time, e.g. using toothed disc
    • H04M1/2745—Devices whereby a plurality of signals may be stored simultaneously with provision for storing more than one subscriber number at a time, e.g. using toothed disc using static electronic memories, e.g. chips
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L2015/088—Word spotting
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223—Execution procedure of a spoken command

Definitions

  • One or more example embodiments relate to a user command processing method and system to provide and adjust an operation of a device or a range of providing contents according to an analysis of a presentation of a user speech, a computer program stored in a computer-readable recording medium for causing a computer to execute the user command processing method in combination with the computer, and the recording medium.
  • Korean Patent Application Publication No. 10-2011-0139797 discloses technology related to a home media device and a home network system and method using the same.
  • This technology provides a home network service using a second communication network such as Wireless-Fidelity (Wi-Fi), other than a mobile communication network in the home network service, and enables a user to control a plurality of in-house multimedia devices through a voice command without a separate button control.
  • Wi-Fi Wireless-Fidelity
  • an electronic device configured to receive an input of a voice command of a user may receive a voice input “Turn off the light” of the user.
  • the electronic device analyzes the received voice input “Turn off the light” and considers turning off the power of a smart lighting device.
  • the electronic device may confront a situation in which the electronic device needs to determine which smart lighting device is to be turned off.
  • One or more example embodiments provide a user command processing method and system that may determine an option according to a presentation of a voice input received from a user although the voice input does not include a direct keyword for determining an option, and perform a selected operation in association with the determined option, with respect to various options of a pre-defined operation of an electronic device capable of performing an operation according to a user command, a computer program stored in a non-transitory computer-readable recording medium for causing a computer to execute the user command processing method in combination with the computer, and the non-transitory recording medium.
  • One or more example embodiments provide a user command processing method and system that may determine an option by utilizing a social relationship between a user and a person associated with an operation selected according to a presentation of a voice input of the user, or an urgency of a user command and/or a user emotion analyzed based on at least one of a manner of speaking and a tone of the user obtained through the voice input, although the voice input does not include a direct keyword for determining an option, a computer program stored in a non-transitory computer-readable recording medium for causing a computer to execute the user command processing method in combination with the computer, and the non-transitory recording medium.
  • a user command processing method including managing at least one pre-defined operation to be performed according to a user command, a plurality of options being preset in relation to each of the at least one pre-defined operation, receiving a user command at least including a voice input received from a user, selecting an operation corresponding to a keyword extracted from the voice input, determining at least one option corresponding to the extracted keyword among a plurality of options preset in relation to the selected operation, according to a presentation of the voice input, and performing the selected operation in association with the determined at least one option
  • a non-transitory computer-readable recording medium storing instructions that, when executed by a processor, cause the processor to perform the user command processing method.
  • a computer program stored in a non-transitory computer-readable recording medium for causing a computer to execute the user command processing method in combination with the computer.
  • a user command processing system including at least one processor implemented to execute computer-readable instructions, and the at least one processor may be configured to manage at least one pre-defined operation to be performed according to a user command, a plurality of options being preset in relation to each of the at least one pre-defined operation, receive a user command at least including a voice input received from a user, select an operation corresponding to a keyword extracted from the voice input, determine at least one option corresponding to the extracted keyword among a plurality of options preset in relation to the selected operation, according to a presentation of the voice input, and perform the selected operation in association with the determined at least one option.
  • FIG. 1 is a diagram illustrating an example of a service environment utilizing a voice-based interface according to one embodiment
  • FIG. 2 is a diagram illustrating another example of a service environment utilizing a voice-based interface according to one embodiment
  • FIG. 3 is a diagram illustrating an example of a cloud artificial intelligence (AI) platform according to one embodiment
  • FIG. 4 is a block diagram to describe internal configurations of an electronic device and a server according to one embodiment
  • FIG. 5A is a flowchart illustrating an example of a user command processing method to be performed by an electronic device according to one embodiment
  • FIG. 5B is a block diagram illustrating an example of constituent elements included in at least one processor of the electronic device 410 according to one embodiment.
  • FIG. 6 is a flowchart illustrating an example of a user command processing method in a case in which an option is indeterminable according to a presentation of a voice input according to one embodiment
  • FIG. 7 is a flowchart illustrating an example of a user command processing method for recommending an automatically selected option or receiving a confirmation thereof according to one embodiment.
  • FIG. 8 is a flowchart illustrating an example of a user command processing method in a case of utilizing an external platform according to one embodiment.
  • Example embodiments will be described in detail with reference to the accompanying drawings.
  • Example embodiments may be embodied in various different forms, and should not be construed as being limited to only the illustrated embodiments. Rather, the illustrated embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the concepts of this disclosure to those skilled in the art. Accordingly, known processes, elements, and techniques, may not be described with respect to some example embodiments. Unless otherwise noted, like reference characters denote like elements throughout the attached drawings and written description, and thus descriptions will not be repeated.
  • first,” “second,” “third,” etc. may be used herein to describe various elements, components, regions, layers, and/or sections, these elements, components, regions, layers, and/or sections, should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or section, from another region, layer, or section. Thus, a first element, component, region, layer, or section, discussed below may be termed a second element, component, region, layer, or section, without departing from the scope of this disclosure.
  • spatially relative terms such as “beneath,” “below,” “lower,” “under,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below,” “beneath,” or “under,” other elements or features would then be oriented “above” the other elements or features. Thus, the example terms “below” and “under” may encompass both an orientation of above and below.
  • the device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
  • the element when an element is referred to as being “between” two elements, the element may be the only element between the two elements, or one or more other intervening elements may be present.
  • Example embodiments may be described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented in conjunction with units and/or devices discussed in more detail below.
  • a function or operation specified in a specific block may be performed differently from the flow specified in a flowchart, flow diagram, etc.
  • functions or operations illustrated as being performed serially in two consecutive blocks may actually be performed simultaneously, or in some cases be performed in reverse order.
  • Units and/or devices may be implemented using hardware, software, and/or a combination thereof.
  • hardware devices may be implemented using processing circuitry such as, but not limited to, a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a System-on-Chip (SoC), a programmable logic unit, a microprocessor, or any other device capable of responding to and executing instructions in a defined manner.
  • processing circuitry such as, but not limited to, a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a System-on-Chip (SoC), a programmable logic unit, a microprocessor, or any other device capable of responding to and executing instructions in a defined manner.
  • Software may include a computer program, program code, instructions, or some combination thereof, for independently or collectively instructing or configuring a hardware device to operate as desired.
  • the computer program and/or program code may include program or computer-readable instructions, software components, software modules, data files, data structures, and/or the like, capable of being implemented by one or more hardware devices, such as one or more of the hardware devices mentioned above.
  • Examples of program code include both machine code produced by a compiler and higher-level program code that is executed using an interpreter.
  • a hardware device is a computer processing device (e.g., a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a microprocessor, etc.)
  • the computer processing device may be configured to carry out program code by performing arithmetical, logical, and input/output operations, according to the program code.
  • the computer processing device may be programmed to perform the program code, thereby transforming the computer processing device into a special purpose computer processing device.
  • the processor becomes programmed to perform the program code and operations corresponding thereto, thereby transforming the processor into a special purpose processor.
  • Software and/or data may be embodied permanently or temporarily in any type of machine, component, physical or virtual equipment, or computer storage medium or device, capable of providing instructions or data to, or being interpreted by, a hardware device.
  • the software also may be distributed over network coupled computer systems so that the software is stored and executed in a distributed fashion.
  • software and data may be stored by one or more computer readable recording mediums, including the tangible or non-transitory computer-readable storage media discussed herein.
  • computer processing devices may be described as including various functional units that perform various operations and/or functions to increase the clarity of the description.
  • computer processing devices are not intended to be limited to these functional units.
  • the various operations and/or functions of the functional units may be performed by other ones of the functional units.
  • the computer processing devices may perform the operations and/or functions of the various functional units without sub-dividing the operations and/or functions of the computer processing units into these various functional units.
  • Units and/or devices may also include one or more storage devices.
  • the one or more storage devices may be tangible or non-transitory computer-readable storage media, such as random-access memory (RAM), read-only memory (ROM), a permanent mass storage device (such as a disk drive), solid state (e.g., NAND flash) device, and/or any other like data storage mechanism capable of storing and recording data.
  • the one or more storage devices may be configured to store computer programs, program code, instructions, or some combination thereof, for one or more operating systems and/or for implementing the example embodiments described herein.
  • the computer programs, program code, instructions, or some combination thereof may also be loaded from a separate computer readable storage medium into the one or more storage devices and/or one or more computer processing devices using a drive mechanism.
  • a separate computer readable storage medium may include a Universal Serial Bus (USB) flash drive, a memory stick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other like computer readable storage media.
  • the computer programs, program code, instructions, or some combination thereof may be loaded into the one or more storage devices and/or the one or more computer processing devices from a remote data storage device via a network interface, rather than via a local computer readable storage medium.
  • the computer programs, program code, instructions, or some combination thereof may be loaded into the one or more storage devices and/or the one or more processors from a remote computing system that is configured to transfer and/or distribute the computer programs, program code, instructions, or some combination thereof, over a network.
  • the remote computing system may transfer and/or distribute the computer programs, program code, instructions, or some combination thereof, via a wired interface, an air interface, and/or any other like medium.
  • the one or more hardware devices, the one or more storage devices, and/or the computer programs, program code, instructions, or some combination thereof, may be specially designed and constructed for the purposes of the example embodiments, or they may be known devices that are altered and/or modified for the purposes of example embodiments.
  • a hardware device such as a computer processing device, may run an operating system (OS) and one or more software applications that run on the OS.
  • the computer processing device also may access, store, manipulate, process, and create data in response to execution of the software.
  • OS operating system
  • a hardware device may include multiple processing elements and multiple types of processing elements.
  • a hardware device may include multiple processors or a processor and a controller.
  • other processing configurations are possible, such as parallel processors.
  • a user command processing system may be implemented through an electronic device that provides an interface operating based on a voice.
  • a user command may at least include a voice input received from a user, and the electronic device may process the user command by recognizing and analyzing the voice input directly or through an external platform connected through a separate interface connect such as an Application Programming Interface (API) call.
  • API Application Programming Interface
  • a user command processing method may be performed through the electronic device.
  • a computer program may be installed and run on the electronic device, and the electronic device may perform the user command processing method according to a control of the computer program.
  • the above computer program may be stored in a computer-readable recording medium for causing a computer to execute the user command processing method in combination with the electronic device.
  • FIG. 1 is a diagram illustrating an example of a service environment utilizing a voice-based interface according to one embodiment.
  • FIG. 1 illustrates an example in which an electronic device 120 providing an interface that operates based on a voice recognizes and analyzes a voice input “Turn off the light” received according to a speech of a user 110 and controls a light power of an in-house lighting device 130 linked to the electronic device 120 through an internal network in a house, in technology that connects and controls in-house devices, such as a smart home or a home network service.
  • in-house devices may include various devices that may be connected and controlled online, for example, home appliances such as the in-house lighting device 130 , a television, a personal computer (PC), peripheral equipment, an air conditioner, a refrigerator, and a robot cleaner, energy consuming devices such as water supply, electricity, and heating and cooling equipment, and security devices such as a door lock and a surveillance camera.
  • the internal network may utilize wired network technology such as the Ethernet, home phoneline networking alliance (HomePNA), and the institute of electrical and electronics engineers (IEEE) 1394, or wireless network technology such as Bluetooth, ultra-wideband (UWB), ZigBee, Wireless 1394, and home radio frequency (HomeRF).
  • home appliances such as the in-house lighting device 130 , a television, a personal computer (PC), peripheral equipment, an air conditioner, a refrigerator, and a robot cleaner
  • energy consuming devices such as water supply, electricity, and heating and cooling equipment
  • security devices such as a door lock and a surveillance camera.
  • the internal network may utilize wired network technology
  • the electronic device 120 may be one of the in-house devices.
  • the electronic device 120 may be one of devices such as an artificial intelligence (AI) speaker or a robot cleaner provided in the house.
  • the electronic device 120 may be a mobile device of the user 110 , for example, a smart phone, a mobile phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), or a tablet PC.
  • PDA personal digital assistant
  • PMP portable multimedia player
  • the electronic device 120 is not particularly limited thereto, and may include any type of device having a function to be connected to the in-house devices to receive a voice input of the user 110 and control the in-house devices.
  • FIG. 2 is a diagram illustrating another example of a service environment utilizing a voice-based interface according to one embodiment.
  • FIG. 2 illustrates an example in which the electronic device 120 providing an interface that operates based on a voice recognizes and analyzes a voice input “The weather today” received according to a speech of the user 110 , obtains information related to the weather for today from an external server 210 through an external network, and outputs the obtained information by voice.
  • the external network may include at least one predetermined network among networks such as a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), and the Internet.
  • PAN personal area network
  • LAN local area network
  • CAN campus area network
  • MAN metropolitan area network
  • WAN wide area network
  • BBN broadband network
  • the electronic device 120 may be one of in-house devices or one of mobile devices of the user 110 .
  • the electronic device 120 is not particularly limited thereto, and may include any type of device having a function to receive and process a voice input of the user 110 and a function to access the external server 210 through the external network and provide the user 110 with contents or a service provided by the external server 210 .
  • the electronic device 120 is not particularly limited, and may include any type of device that may process a user command at least including a voice input received according to a speech of the user 110 through the voice-based interface.
  • the electronic device 120 may process a user command by directly recognizing and analyzing a voice input of a user and performing an operation suitable for the voice input.
  • the electronic device 120 may perform a process of recognizing the voice input of the user, analyzing the recognized voice input, and synthesizing a voice to be provided to the user through an external platform linked to the electronic device 120 .
  • FIG. 3 is a diagram illustrating an example of a cloud AI platform according to one embodiment.
  • FIG. 3 illustrates electronic devices 310 , a cloud AI platform 320 , and “contents-services” 330 .
  • the electronic devices 310 may refer to devices provided in a house, and include at least the electronic device 120 described above.
  • the electronic devices 310 or applications, hereinafter, apps, installed and run on the electronic devices 310 may be linked to the cloud AI platform 320 through an interface connect 340 such as an Application Programming Interface (API) call.
  • the interface connect 340 may provide developers with development documents and/or a software development kit (SDK) to be used to develop the electronic devices 310 or the apps installed and run on the electronic devices 310 .
  • SDK software development kit
  • the interface connect 340 may provide an application program interface (API) by which the electronic devices 310 or the apps installed and run on the electronic devices 310 may utilize functions provided by the cloud AI platform 320 .
  • API application program interface
  • devices or apps developed by the developers using the development documents and/or the SDK provided by the interface connect 340 may utilize the functions provided by the cloud AI platform 320 using the API provided by the interface connect 340 .
  • the cloud AI platform 320 may provide a function to provide a voice-based service.
  • the cloud AI platform 320 may include various modules to provide the voice-based service, for example, a voice processing module 321 to recognize a received voice and synthesize a voice to be output, a vision processing module 322 to analyze and process a received image or video, a conversation processing module 323 to determine a suitable conversation to output an appropriate voice according to the received voice, a recommendation module 324 to recommend a function suitable for the received voice, and a neural machine translation (NMT) 325 to support an AI to translate a language in a unit of sentence through data learning.
  • NMT neural machine translation
  • the electronic device 120 may transmit the voice input of the user 110 to the cloud AI platform 320 using the API provided by the interface connect 340 .
  • the cloud AI platform 320 may recognize and analyze the received voice input by utilizing the modules 321 through 325 described above, and synthesize and provide a suitable answer voice according to the received voice input or recommend a suitable operation.
  • an extension kit 350 may be a software development kit that enables third-party content developers or companies to implement a new voice-based function based on the cloud AI platform 320 and provides by provider of the cloud AI platform 320 .
  • the electronic device 120 may transmit the received voice input of the user 110 to the external server 210 , and the external server 210 may transmit the voice input to the cloud AI platform 320 through an API provided through the extension kit 350 .
  • the cloud AI platform 320 may synthesize and provide a suitable answer voice or provide the external server 210 with recommendation information related to a function to be processed through the voice input by recognizing and analyzing the received voice input.
  • FIG. 1 the example of FIG.
  • the external server 210 may transmit the voice input “The weather today” to the cloud AI platform 320 , and receive keywords “the weather” and “today” extracted through recognition of the voice input “The weather today” from the cloud AI platform 320 .
  • the external server 210 may generate textual information such as “The weather for today is . . . ” through the keywords “the weather” and “today,” and transmit the generated textual information back to the cloud AI platform 320 .
  • the cloud AI platform 320 may synthesize the textual information into a voice and provide the synthesized voice to the external server 210 .
  • the external server 210 may transmit the synthesized voice to the electronic device 120 , and the electronic device 120 may output the synthesized voice “The weather for today is . . . ” through a speaker, thereby processing the voice input “The weather today” received from the user 110 .]
  • FIG. 4 is a block diagram to describe internal configurations of an electronic device and a server according to one embodiment.
  • An electronic device 410 of FIG. 4 may correspond to the electronic device 120 described above, and the server 420 may correspond to the external server 210 described above or a computer device that implements the cloud AI platform 320 .
  • the electronic device 410 and the server 420 may include memories 411 and 421 , processors 412 and 422 , communication modules 413 and 423 , and input/output interfaces 414 and 424 , respectively.
  • the memories 411 and 421 may be computer-readable recording media, and include permanent mass storage devices such as a random-access memory (RAM), a read-only memory (ROM), and a disk drive.
  • the permanent mass storage devices such as the ROM and the disk drive may be included in the electronic device 410 or the server 420 as separate permanent storage devices distinct from the memories 411 and 421 .
  • an operating system and at least one program code may be stored in the memories 411 and 421 .
  • Such software components may be loaded from computer-readable recording media separate from the memories 411 and 421 .
  • the separate computer-readable recording media may include computer-readable recording media such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card.
  • the software components may be loaded from the memories 411 and 421 through the communication modules 413 and 423 , other than the computer-readable recording media.
  • At least one program may be loaded from the memory 411 of the electronic device 410 based on a computer program, for example, the application described above, installed by files provided by a file distribution system that distributes a setup file of an application or developers through a network 430 .
  • the processors 412 and 422 may be configured to process instructions of the computer program by performing basic arithmetic, logical, and input/output operations.
  • the instructions may be provided by the memories 411 and 421 or the communication modules 413 and 423 to the processors 412 and 422 , respectively.
  • the processors 412 and 422 may be configured to execute instructions received according to program codes stored in recording devices such as the memories 411 and 421 .
  • the communication modules 413 and 423 may provide a function for the electronic device 410 and the server 420 to communicate with each other through the network 430 , and provide a function for the electronic device 410 and/or the server 420 to communicate with another electronic device or another server.
  • a request generated by the processor 412 of the electronic device 410 according to the program code stored in a recording device such as the memory 411 may be transmitted to the server 420 through the network 430 according to a control of the communication module 413 .
  • a control signal, an instruction, contents, or a file provided according to a control of the processor 422 of the server 420 may be received by the electronic device 410 through the communication module 413 of the electronic device 410 via the communication module 423 and the network 430 .
  • the control signal, the instruction, the contents, or the file of the server 420 received through the communication module 413 may be transmitted to the processor 412 or the memory 411 , and the contents or the file may be stored in a storage medium, for example, the permanent storage device described above, that is further included in the electronic device 410 .
  • the input/output interface 414 may be a means for interface with an input/output device 415 .
  • an input device may include a device such as a microphone, a keyboard, or a mouse
  • an output device may include a device such as a display or a speaker.
  • the input/output interface 414 may be a means for interfacing with a device in which functions for an input and an output are unified, such as a touch screen.
  • the input/output device 415 and the electronic device 410 may be configured as a single device.
  • the input/output interface 424 of the server 420 may be a means for interface with a device (not shown) for an input or an output, the device to be connected to the server 420 or included in the server 420 .
  • the electronic device 410 and the server 420 may include fewer or more elements or components than the elements or components of FIG. 4 .
  • the electronic device 410 may be implemented to include at least a portion of the input/output device 415 or may further include other elements such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database.
  • GPS global positioning system
  • the electronic device 410 may be implemented to further include various elements that a general smart phone includes, for example, an acceleration sensor or gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration.
  • the electronic device 410 may basically include a microphone to receive a voice input of a user as the input/output device 415 , and may further include, as necessary, a speaker to output an answer voice corresponding to the voice input of the user or a sound such as audio contents as the input/output device 415 .
  • a single operation to be performed by the electronic device 410 according to a user command may include a plurality of options according to the operation. For example, with respect to a voice input “Turn off the light” included in a user command, the electronic device 410 may perform an operation of turning off the power of a lighting device linked thereto. However, if a plurality of lighting devices is linked to the electronic device 410 , there may be an option to determine which lighting device is to be turned off.
  • the voice input of the user may directly include information to select the option, for example, “this room” of “Turn off the light in this room,” or may simply include only information related to the operation. At least one example embodiment describes technologies for selecting such an option based on a presentation of an existing voice input, without receiving an additional voice input from the user.
  • FIG. 5A is a flowchart illustrating an example of a user command processing method to be performed by an electronic device according to one embodiment
  • FIG. 5B is a block diagram illustrating an example of constituent elements included in at least one processor of the electronic device 410 according to one embodiment
  • a user command processing system may be implemented in a form of a computer device such as the electronic device 410 described above.
  • the electronic device 410 may perform operations 510 through 570 of the user command processing method of FIG. 5A
  • the processor 412 may be implemented to execute control instructions according to codes of an operating system or codes of at least one program included in the memory 411 .
  • the processor 412 may control the electronic device 410 to perform operations 510 through 570 of FIG.
  • the constituent elements of the processor 412 such as an operation manager 581 , a user command receiver 582 , an operation selector 583 , an option existence determinator 584 , an option determinator 585 , an operation performer 586 , a voice input transmitter 587 , and an information receiver 588 of FIG. 5B , may be representations of different functions performed at the processor 412 .
  • the operation manager 581 may manage at least one pre-defined operation to be performed according to a user command.
  • a plurality of options may be preset in relation to each of the at least one pre-defined operation.
  • options preset with respect to a single operation may include an option for selecting a device, for example, selecting a type of a device to be controlled according to the corresponding operation or selecting a predetermined device to be controlled according to the operation.
  • the options preset with respect to the single operation may include an option for controlling a range of contents to be provided or a providing scheme such as a characteristic of the contents to be provided according to the operation or an order of providing the contents. Detailed examples of such options will be described further later.
  • the user command receiver 582 may receive a user command at least including a voice input received from a user.
  • the electronic device 410 may include a microphone as the input/output device 415 , and the user command receiver 582 may receive the user command including the voice input of the user input through the microphone.
  • the electronic device 410 may include a camera, and receive an image input through the camera together with the voice input as the user command. Such image utilization will be described further later.
  • the operation selector 583 may select an operation corresponding to a keyword extracted from the voice input. For example, the operation selector 583 may obtain a character string corresponding to the voice input by analyzing the received voice input, and select a corresponding operation using a specific keyword by analyzing the obtained character string. As another example, the operation selector 583 may receive a keyword or select an operation corresponding to the keyword by means of an external platform such as the cloud AI platform 320 of FIG. 3 . In detail, the operation selector 583 may transmit the voice input of the user to the cloud AI platform 320 through the interface connect 340 , and select an operation suitable for the user command through a keyword or a recommended operation corresponding to the keyword provided by the cloud AI platform 320 by analyzing the voice input.
  • the option existence determinator 584 may determine whether at least one option corresponding to the extracted keyword exists among a plurality of options preset in related to the selected operation. For example, in relation to an operation of turning off a light power of a lighting device, if the electronic device 410 is connected to a plurality of lighting devices, an option for selecting a specific lighting device from among the plurality of lighting devices or all the lighting devices may be preset for the corresponding operation. In this example, the electronic device 410 may determine whether an option is selectable through the keyword extracted from the voice input of the user.
  • the electronic device 410 may determine whether one of the plurality of lighting devices is selectable as at least one of a plurality of options preset with respect to an operation, like “the small bedroom” of “Turn off the light in the small bedroom” or “this room” of “Turn off the light in this room.”
  • a person to which an instant message is to be sent may be determined through a selection of an option.
  • different accounts registered in relation to an account of a user in a messenger service may be preset as the plurality of options in relation to the corresponding operation.
  • phone numbers stored in a smart phone of the user or names stored with respect to the phone numbers may be preset as the plurality of options with respect to the corresponding operation.
  • the electronic device 410 may determine whether a person to which an instant message is to be sent is selectable as at least one of the plurality of options preset with respect to the operation, like “daughter” of “Text my daughter to come home early” or “my mother” of “Text my mother to please come home early.”
  • the electronic device 410 may perform operation 550 when an option corresponding to the extracted keyword is present, and perform operation 560 when an option corresponding to the extracted keyword is absent.
  • the option determinator 585 may determine at least one option corresponding to the extracted keyword among the plurality of options. That is, the option determinator 585 may determine a corresponding operation and at least one of the options preset with respect to the operation using a keyword included in the voice input.
  • At least one of the options preset with respect to the operation is indeterminable through the keyword included in the voice input.
  • the voice input “Turn off the light” does not include a keyword regarding which lighting device is to be turned off
  • the voice input “Text come home early” does not include a keyword regarding to whom the message is to be sent.
  • a selection of which lighting device or whom the message should be sent needs to be received from the user.
  • the user may have more difficulties in selecting one of various options in a voice-based interface, when compared to selecting one of various options in a visual information-based interface. Accordingly, at least one example embodiment may provide a function to select an option according to a presentation of the voice input.
  • the option determinator 585 may determine at least one of the plurality options according to a presentation of the voice input. For example, the option determinator 585 may determine an option of the operation based on honorifics used in the voice input, how exploratory a speech of the user is, for example, an urgency level of the user command, or a user emotion analyzed based on a manner of speaking or a tone of the user.
  • the presentation of the voice input may include an expression representing a social relationship between a person associated with the selected operation and the user.
  • the option determinator 585 may determine at least one of the plurality of options based on the social relationship with the user which is analyzed according to the expression.
  • the plurality of options may correspond to a plurality of devices interoperating with the electronic device 410 .
  • the electronic device 410 may determine, to be an option, at least one device associated with a person selected based on the social relationship with the user among the plurality of devices.
  • a voice input “Turn off the light” and a voice input “Please turn off the light” (“please” used herein to carry a honorific connotation in Korean language)
  • the electronic device 410 may verify that a request of the user is for a person in a higher social position than the user according to a presentation of the latter voice input “Please turn off the light,” and select an option based on the social relationship. For example, in the above example, the electronic device 410 may automatically select a lighting device positioned in a room of a person having a higher social status than the user, for example, a parent, rather than requesting the user to select the lighting device. Further, the electronic device 410 may automatically select a lighting device in a room where the user is currently present with respect to the voice input “Turn off the light.”
  • the plurality of options may correspond to messenger accounts or phone numbers of other users having a personal relationship with the user.
  • the electronic device 410 may determine, to be an option, a messenger account or phone number of another user selected based on the social relationship with the user among the messenger accounts or phone numbers.
  • the electronic device 410 may verify that a recipient of the message is a person who lives together with the user and has a lower social status than the user, for example, a child, through a presentation of the voice input.
  • the electronic device 410 may automatically determine the recipient of the message to be a “daughter” or “son” of the user, rather than requesting the user to determine the recipient.
  • the electronic device 410 may perform an operation of recommending the recipient to the user, such as “Do you want to send the message to your daughter?” and send a text message saying “come home early” to the “daughter” of the user using a phone number or messenger account of the “daughter” according to a confirmation input of the user.
  • the electronic device 410 may verify that a recipient of the message is a person who lives together with the user and has a higher social status than the user, for example, a parent, through a presentation of the voice input.
  • the electronic device 410 may automatically determine the “mother” or the “father” of the user to be the recipient of the message based on such a social relationship, and perform an operation of recommending the recipient of the message to the user, such as “Do you want to send the message to your mother?”
  • the electronic device 410 may send a text message saying “please come home early” to the “mother” of the user using the phone number or the messenger account of the “mother” according to the confirmation input of the user.
  • the electronic device 410 may preset options with respect to an operation of sending a text based on information related to the social relationship of the user.
  • the presentation of the voice input may include at least one of an urgency of a user command and a user emotion analyzed based on at least one of a manner of speaking and the tone of the user obtained through the voice input.
  • the electronic device 410 may determine at least one option among the plurality of options based on at least one of the urgency of the user command and the user emotion.
  • the plurality of options may correspond to characteristic classifications of contents to be associated with the selected operation or schemes of providing information related to the contents.
  • the electronic device 410 may determine at least one classification among the characteristic classifications of the contents to be associated with the selected operation or determine at least one scheme among the schemes of providing information related to the contents based on at least one of the urgency of the user command and the user emotion.
  • a voice input “Find me a good restaurant” and a voice input “Would you find me a good restaurant?” may each have a different urgency of a user command according to the presentation of the voice input.
  • the electronic device 410 may preset an urgency level according to the presentation of a voice input, such as a manner of speaking and a tone of the user, like “Find me” and “Would you find me?” and determine a characteristic of contents to be provided or a providing scheme to be the option according to the urgency level. For example, if a voice input is determined to have a relatively high urgency level, like the voice input “Find me a good restaurant,” the electronic device 410 may provide the user with high-accuracy search results including restaurants close to the user and reflecting a taste of the user.
  • the electronic device 410 may provide the user with search results reflecting a taste of the user irrespective of a distance or various search results such as restaurants that recently attract a public interest.
  • the priority of the distance between the user and a restaurant included in a search result may be determined to be the option with respect to the operation in relation to the search.
  • a characteristic of contents or a providing scheme may be selected as the option based on an emotion of the user analyzed through the manner of speaking or the tone of the user. For example, if the user is in a bad mood, the electronic device 410 may first provide the user with information related to a match won by a sports team that the user supports, among information related to the corresponding sports team. As another example, if the user feels good, the electronic device 410 may sequentially provide the user with the latest information related to the sports team that the user supports. That is, the electronic device 410 may select the characteristic of the contents to be provided or the providing scheme as the option with respect to an operation of providing contents according to the emotion of the user.
  • the electronic device 410 may determine at least one option among the plurality of options by further utilizing situation data of the user.
  • the situation data may include at least one of an instant message sent and received in association with a phone number or messenger account of the user and a facial expression of the user recognized through a camera included in the electronic device 410 .
  • instant messages recently sent and received by the user for example, within the last one hour, may be utilized to accurately measure the emotion of the user described above.
  • the facial expression of the user recognized through the camera may be utilized to accurately measure the emotion of the user or the urgency of the user command.
  • the operation performer 586 may perform the selected operation in association with the determined at least one option. As already described above, the operation performer 586 may perform an operation of recommending a recipient selected as the option or sending a message to the selected recipient. Also, the electronic device 410 may provide the user with the contents according to the characteristic of the contents and/or the providing scheme selected as the option.
  • the option may be automatically selected according to a presentation of the voice input of the user and the corresponding operation may be performed, without requesting the user to additionally select the option.
  • FIG. 6 is a flowchart illustrating an example of a user command processing method in a case in which an option is indeterminable according to a presentation of a voice input according to one embodiment. Operations 610 and 620 of FIG. 6 may be performed by being included in operation 560 of FIG. 5A .
  • the option determinator 585 may output information to receive, from a user, a selection of at least one option among a plurality of options. That is, the option determinator 585 may attempt to automatically determine the option through the presentation of the voice input, and then request the user to additionally select the option if the option is indeterminable through the presentation. For example, if a recipient of a message is indeterminable through only the presentation of the voice input, the option determinator 585 may output information to request the user to select the recipient, like “To whom do you want to send the message?”
  • the option determinator 585 may determine at least one option among the plurality of options according to an additional voice input received in response to the output information. For example, when an additional voice input of “my mother” is received from the user after outputting a voice saying “To whom do you want to send the message?”, the recipient of the message may be determined based on the additional voice input.
  • FIG. 7 is a flowchart illustrating an example of a user command processing method for recommending an automatically selected option or receiving a confirmation thereof according to one embodiment. Operations 710 through 780 of FIG. 7 may be performed by being included in operation 570 of FIG. 5A .
  • the operation performer 586 may output information to receive, from a user, a confirmation on whether to continue performing an operation in association with a determined at least one option.
  • the output information may be utilized to receive, from the user, a confirmation on whether the determined option is suitable, or to recommend the determined option to the user. For example, in the example described above, “Do you want to send the message to your daughter?” may be output to receive, from the user, a confirmation on whether the recipient “daughter” determined to be the option is suitable and/or to recommend the recipient “daughter” determined to be the option to the user.
  • the operation performer 586 may receive a first additional voice input in response to the output information.
  • the first additional voice input may be received as an answer for continuation consent or continuation refusal of the user with respect to the output information for confirmation and/or recommendation.
  • the operation performer 586 may analyze the first additional voice input to extract the continuation consent or the continuation refusal.
  • the operation performer 586 may determine whether the continuation consent is extracted. In this example, the operation performer 586 may perform operation 750 when an answer for continuation consent is extracted, and perform operation 760 , 770 , and 780 when an answer for continuation refusal is extracted.
  • the operation performer 586 may output information to receive, from the user, a selection of at least one other option among the plurality of options when an answer for continuation refusal is extracted from the first additional voice input. That is, the operation performer 586 may request the user to select another option when the user shows a negative response to the information output for confirmation and/or recommendation. For example, when an answer for continuation refusal, like “No,” is extracted with respect to “Do you want to send the message to your daughter?” although the “daughter” is recommended as the recipient, the operation performer 586 may request the user to select another option.
  • the operation performer 586 may determine at least one other option among the plurality of options according to a second additional voice input received in response to the output information.
  • the user may select another option while providing the answer for continuation refusal.
  • the user may simultaneously provide the answer for continuation refusal and a selection of another option, like “No, send it to my son,” with respect to the output information “Do you want to send the message to your daughter?”
  • operation 750 may be omitted, and the operation performer 586 may determine another option from the first additional voice input, in operation 760 .
  • the operation performer 586 may perform the selected operation in association with the determined at least one option.
  • FIG. 8 is a flowchart illustrating an example of a user command processing method in a case of utilizing an external platform according to one embodiment.
  • Operations 810 and 820 of FIG. 8 may be performed after operation 520 of FIG. 5A , and an operation following operation 820 may be connected to one of operations 530 through 570 of FIG. 5A according to information received in operation 820 .
  • the voice input transmitter 587 of FIG. 5B may transmit a voice input to a cloud AI platform connected to the electronic device 410 through an interface connect.
  • a cloud AI platform connected to the electronic device 410 through an interface connect.
  • FIG. 3 a case in which an external platform provides a voice-based service related to processing the voice input, synthesizing an answer voice, and/or processing a conversation was described.
  • a voice input of a user received for processing the voice input according to the voice-based service may be transmitted to the cloud AI platform.
  • the information receiver 588 of FIG. 5B may receive, from the cloud AI platform, information related to at least one of a keyword extracted from the voice input, an operation recommended based on the extracted keyword, an option corresponding to the extracted keyword, and an option determined according to a presentation of the voice input.
  • the cloud AI platform may recognize and analyze the voice input, extract a keyword therefrom, and transmit the extracted keyword to the electronic device 410 .
  • the electronic device 410 may perform the selected operation in association with the determined option, by performing operations of FIG. 5A , sequentially from operation 530 of FIG. 5A .
  • the cloud AI platform may recognize and analyze the voice input, extract a keyword therefrom, and recommend an operation based on the extracted keyword.
  • the electronic device 410 may perform the selected operation in association with the determined option, by performing operations of FIG. 5A , sequentially from operation 540 of FIG. 5A based on the recommended operation.
  • the cloud AI platform may recognize and analyze the voice input, extract a keyword therefrom, select an operation based on the extracted keyword, determine an option corresponding to the extracted keyword, and provide the selected operation and the determined option to the electronic device 410 .
  • the electronic device 410 may determine the option provided from the cloud AI platform to be at least one option with respect to the selected operation. Further, the electronic device 410 may perform the selected operation in association with the determined option through operation 570 .
  • the cloud AI platform may recognize and analyze the voice input, extract a keyword, select an operation based on the extracted keyword, and determine an option according to a presentation of the voice input when an option corresponding to the extracted keyword is absent.
  • the cloud AI platform may provide the selected operation and the determined option to the electronic device 410 .
  • the electronic device 410 may determine the option provided from the cloud AI platform to be at least one option with respect to the selected operation in operation 560 of FIG. 5A , and perform the selected operation in association with the determined option through operation 570 .
  • the cloud AI platform may provide the voice-based service to the electronic device 410 , and the voice-based service may include a different function according to at least one example embodiment.
  • an option it is possible to determine an option according to a presentation of a voice input received from a user although the voice input does not include a direct keyword for determining an option, and perform a selected operation in association with the determined option, with respect to various options of a pre-defined operation of an electronic device capable of performing an operation according to a user command.
  • an option by utilizing a social relationship between a user and a person associated with an operation selected according to a presentation of a voice input of the user, or an urgency of a user command and/or a user emotion analyzed based on at least one of a manner of speaking and a tone of the user obtained through the voice input, although the voice input does not include a direct keyword for determining an option.
US15/969,394 2017-05-02 2018-05-02 Method and system for processing user command to provide and adjust operation of electronic device by analyzing presentation of user speech Active 2038-10-31 US10657963B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2017-0056128 2017-05-02
KR1020170056128A KR101949497B1 (ko) 2017-05-02 2017-05-02 사용자 발화의 표현법을 파악하여 기기의 동작이나 컨텐츠 제공 범위를 조정하여 제공하는 사용자 명령 처리 방법 및 시스템

Publications (2)

Publication Number Publication Date
US20180322872A1 US20180322872A1 (en) 2018-11-08
US10657963B2 true US10657963B2 (en) 2020-05-19

Family

ID=64015405

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/969,394 Active 2038-10-31 US10657963B2 (en) 2017-05-02 2018-05-02 Method and system for processing user command to provide and adjust operation of electronic device by analyzing presentation of user speech

Country Status (3)

Country Link
US (1) US10657963B2 (ja)
JP (1) JP2018190413A (ja)
KR (1) KR101949497B1 (ja)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI651714B (zh) * 2017-12-22 2019-02-21 隆宸星股份有限公司 語音選項選擇系統與方法以及使用其之智慧型機器人
EP3783867A1 (en) 2018-05-07 2021-02-24 Google LLC Providing composite graphical assistant interfaces for controlling various connected devices
KR102088322B1 (ko) * 2018-11-16 2020-03-12 엘지전자 주식회사 인공지능 기반의 어플라이언스 제어 장치 및 그를 포함하는 어플라이언스 제어 시스템
WO2020217318A1 (ja) * 2019-04-23 2020-10-29 三菱電機株式会社 機器制御装置、および、機器制御方法
WO2020241923A1 (ko) * 2019-05-29 2020-12-03 엘지전자 주식회사 사용자 환경에서의 음성 인식 모델의 성능을 예측하는 인공 지능 장치 및 그 방법
US20210027409A1 (en) * 2019-07-23 2021-01-28 Ola Electric Mobility Private Limited Methods and Systems for Facilitating Safety and Security of Users
CN110519373B (zh) * 2019-08-28 2022-04-26 百度在线网络技术（北京）有限公司 用于推送信息的方法和装置
KR20210059367A (ko) * 2019-11-15 2021-05-25 삼성전자주식회사 음성 입력 처리 방법 및 이를 지원하는 전자 장치
GB2590473B (en) * 2019-12-19 2022-07-27 Samsung Electronics Co Ltd Method and apparatus for dynamic human-computer interaction
CN116866872A (zh) * 2020-01-09 2023-10-10 博泰车联网科技(上海)股份有限公司 用于信息处理的方法、设备和计算机存储介质
KR20210111423A (ko) * 2020-03-03 2021-09-13 삼성전자주식회사 사용자 입력을 처리하는 전자 장치 및 방법
US11917092B2 (en) * 2020-06-04 2024-02-27 Syntiant Systems and methods for detecting voice commands to generate a peer-to-peer communication link
US11769489B2 (en) 2021-01-28 2023-09-26 Samsung Electronics Co., Ltd. Electronic device and method for performing shortcut command in electronic device
KR20220108919A (ko) * 2021-01-28 2022-08-04 삼성전자주식회사 전자 장치 및 전자 장치의 단축 명령어 수행 방법
KR20220125566A (ko) * 2021-03-05 2022-09-14 삼성전자주식회사 전자 장치 및 전자 장치의 기기 연결 방법
KR20220133414A (ko) 2021-03-25 2022-10-05 삼성전자주식회사 음성 어시스턴트 서비스 제공 방법 및 이를 지원하는 전자 장치
CN113643701B (zh) * 2021-07-27 2024-04-12 广州市威士丹利智能科技有限公司 一种智能识别语音控制家居的方法及统

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH01119791A (ja) 1987-11-04 1989-05-11 Toshiba Corp 核融合装置
JPH11119791A (ja) 1997-10-20 1999-04-30 Hitachi Ltd 音声感情認識システムおよび方法
US20070216764A1 (en) * 2006-01-09 2007-09-20 Samsung Electronics Co., Ltd. Access authentication system and method using smart communicator
KR20070106488A (ko) 2007-10-12 2007-11-01 (주)인피니티 텔레콤 자연어 음성 인식 기능을 이용한 휴대폰 서비스 방법
US7590536B2 (en) * 2005-10-07 2009-09-15 Nuance Communications, Inc. Voice language model adjustment based on user affinity
JP2009282936A (ja) 2008-05-26 2009-12-03 Nippon Telegr & Teleph Corp <Ntt> 選択式情報提示装置および選択式情報提示処理プログラム
JP2011028460A (ja) 2009-07-24 2011-02-10 J&K Car Electronics Corp 情報検索装置、制御方法及びプログラム
KR20110139797A (ko) 2010-06-24 2011-12-30 에스케이플래닛 주식회사 홈 미디어 디바이스 및 이를 이용한 홈 네트워크 시스템, 방법
KR20130125367A (ko) 2010-11-12 2013-11-18 마이크로소프트 코포레이션 콘텐츠의 시청자 기반 제공 및 맞춤화
US20130311997A1 (en) * 2012-05-15 2013-11-21 Apple Inc. Systems and Methods for Integrating Third Party Services with a Digital Assistant
US20140100848A1 (en) * 2012-10-05 2014-04-10 Avaya Inc. Phrase spotting systems and methods
US20150170643A1 (en) * 2013-12-17 2015-06-18 Lenovo (Singapore) Pte, Ltd. Verbal command processing based on speaker recognition
US9189196B2 (en) * 2013-03-14 2015-11-17 Google Inc. Compartmentalized self registration of external devices
WO2015196063A1 (en) * 2014-06-19 2015-12-23 Robert Bosch Gmbh System and method for speech-enabled personalized operation of devices and services in multiple operating environments
US20160125879A1 (en) * 2014-11-03 2016-05-05 Microsoft Technology Licensing, Llc Augmentation of key phrase user recognition
US20160248768A1 (en) * 2015-02-20 2016-08-25 Sri International Joint Speaker Authentication and Key Phrase Identification
US20160262017A1 (en) * 2015-03-04 2016-09-08 Microsoft Technology Licensing, Llc Personal assistant authentication
KR20160141595A (ko) 2015-06-01 2016-12-09 삼성전자주식회사 메시지를 출력하는 전자 장치 및 그 제어 방법
JP2017016683A (ja) 2011-09-30 2017-01-19 アップル インコーポレイテッド ハンズフリーインタラクションに対するユーザインターフェースの自動適合化
US20180074785A1 (en) * 2015-03-31 2018-03-15 Sony Corporation Information processing device, control method, and program

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000187776A (ja) * 1998-06-09 2000-07-04 Digicube Co Ltd 情報提供装置、情報提供システム、情報提供処理方法、及び記録媒体
JP2002320037A (ja) * 2001-04-19 2002-10-31 Atr Onsei Gengo Tsushin Kenkyusho:Kk 翻訳電話システム
JP2003216690A (ja) * 2002-01-18 2003-07-31 Fujita Corp 情報管理システム
JP2004219714A (ja) * 2003-01-15 2004-08-05 Will Being:Kk 人間からの指示に基づいてそれぞれ予め定めた特定のシーンに属する対話のシーンを識別し、シーンに即した自然対話を構成する応答文を作成して、それを音声合成することにより、音声対話を行うコンピュータによる音声対話方法及び音声対話システム
JP2005275601A (ja) * 2004-03-23 2005-10-06 Fujitsu Ltd 音声による情報検索システム
JP2010210847A (ja) * 2009-03-10 2010-09-24 Clarion Co Ltd 楽曲選択方法、楽曲選択プログラム、及び楽曲選択装置
US9484025B2 (en) * 2013-10-15 2016-11-01 Toyota Jidosha Kabushiki Kaisha Configuring dynamic custom vocabulary for personalized speech recognition
JP2016076799A (ja) * 2014-10-03 2016-05-12 シャープ株式会社 家電管理システム、家電、リモコン装置、ロボット
JP6739907B2 (ja) * 2015-06-18 2020-08-12 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 機器特定方法、機器特定装置及びプログラム
JP2017009826A (ja) * 2015-06-23 2017-01-12 トヨタ自動車株式会社 グループ状態判定装置およびグループ状態判定方法

Patent Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH01119791A (ja) 1987-11-04 1989-05-11 Toshiba Corp 核融合装置
JPH11119791A (ja) 1997-10-20 1999-04-30 Hitachi Ltd 音声感情認識システムおよび方法
US7590536B2 (en) * 2005-10-07 2009-09-15 Nuance Communications, Inc. Voice language model adjustment based on user affinity
US20070216764A1 (en) * 2006-01-09 2007-09-20 Samsung Electronics Co., Ltd. Access authentication system and method using smart communicator
KR20070106488A (ko) 2007-10-12 2007-11-01 (주)인피니티 텔레콤 자연어 음성 인식 기능을 이용한 휴대폰 서비스 방법
JP2009282936A (ja) 2008-05-26 2009-12-03 Nippon Telegr & Teleph Corp <Ntt> 選択式情報提示装置および選択式情報提示処理プログラム
JP2011028460A (ja) 2009-07-24 2011-02-10 J&K Car Electronics Corp 情報検索装置、制御方法及びプログラム
KR20110139797A (ko) 2010-06-24 2011-12-30 에스케이플래닛 주식회사 홈 미디어 디바이스 및 이를 이용한 홈 네트워크 시스템, 방법
KR20130125367A (ko) 2010-11-12 2013-11-18 마이크로소프트 코포레이션 콘텐츠의 시청자 기반 제공 및 맞춤화
JP2017016683A (ja) 2011-09-30 2017-01-19 アップル インコーポレイテッド ハンズフリーインタラクションに対するユーザインターフェースの自動適合化
US20130311997A1 (en) * 2012-05-15 2013-11-21 Apple Inc. Systems and Methods for Integrating Third Party Services with a Digital Assistant
US20140100848A1 (en) * 2012-10-05 2014-04-10 Avaya Inc. Phrase spotting systems and methods
US9189196B2 (en) * 2013-03-14 2015-11-17 Google Inc. Compartmentalized self registration of external devices
US20150170643A1 (en) * 2013-12-17 2015-06-18 Lenovo (Singapore) Pte, Ltd. Verbal command processing based on speaker recognition
WO2015196063A1 (en) * 2014-06-19 2015-12-23 Robert Bosch Gmbh System and method for speech-enabled personalized operation of devices and services in multiple operating environments
US20160125879A1 (en) * 2014-11-03 2016-05-05 Microsoft Technology Licensing, Llc Augmentation of key phrase user recognition
US20160248768A1 (en) * 2015-02-20 2016-08-25 Sri International Joint Speaker Authentication and Key Phrase Identification
US20160262017A1 (en) * 2015-03-04 2016-09-08 Microsoft Technology Licensing, Llc Personal assistant authentication
US20180074785A1 (en) * 2015-03-31 2018-03-15 Sony Corporation Information processing device, control method, and program
KR20160141595A (ko) 2015-06-01 2016-12-09 삼성전자주식회사 메시지를 출력하는 전자 장치 및 그 제어 방법

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
1) Takuhiro Ega, 1 hour delivery, all you can do, point doubling, omni channel . . . more convenient! You can get more! New way of buying convenience store × net mail order best service decision title Head-to-head: CVS vs E-commerce, Nikkei trendy No. 395, Nikkei Business Publications, Inc. Nikkei Business Publications, Inc., 20 Jun. 20 (with partial translation).
Japanese Office Action dated Feb. 19, 2019 by the Japanese Patent Office in Japanese patent application No. 2018-85215.
Japanese Office Action issued in corresponding Japanese patent application No. 2018-85215, dated Jun. 25, 2019.
Nikkei Communication No. 624 Nikkei Communications, Japan, Nikkei Business Publications, Inc. Nikkei Business Publications, Inc., Dec. 28, 2015 (with partial translation).

Also Published As

Publication number Publication date
US20180322872A1 (en) 2018-11-08
KR20180122106A (ko) 2018-11-12
KR101949497B1 (ko) 2019-02-18
JP2018190413A (ja) 2018-11-29

Similar Documents

Publication Publication Date Title
US10657963B2 (en) Method and system for processing user command to provide and adjust operation of electronic device by analyzing presentation of user speech
US20230186915A1 (en) Processing voice commands based on device topology
US10970605B2 (en) Electronic apparatus and method of operating the same
US11450353B2 (en) Video tagging by correlating visual features to sound tags
JP6752870B2 (ja) 複数のウェイクワードを利用して人工知能機器を制御する方法およびシステム
JP6731894B2 (ja) デバイス制御方法及び電子機器
JP6567727B2 (ja) 受信した音声入力の入力音量に基づいて出力される音の出力音量を調節するユーザ命令処理方法およびシステム
JP6728319B2 (ja) 人工知能機器で複数のウェイクワードを利用したサービス提供方法およびそのシステム
JP6607999B2 (ja) 音声要請に対応する情報提供のためのメディア選択
US11341966B2 (en) Output for improving information delivery corresponding to voice request
US11662976B2 (en) Electronic device and method for sharing voice command thereof
KR102209092B1 (ko) 복수의 호출 용어를 이용하여 인공지능 기기를 제어하는 방법 및 시스템
WO2018133307A1 (zh) 一种实现语音控制的方法和终端
US11030479B2 (en) Mapping visual tags to sound tags using text similarity
US20200090652A1 (en) System and method for providing voice assistant service
US11341965B2 (en) System for processing user utterance and operating method thereof
CN110868471A (zh) 新型设备构建方法、装置及存储介质
US11881215B2 (en) Electronic device and operation method of same
KR20220165993A (ko) 인공지능 캐릭터 생성 방법 및 그 시스템
US20230154462A1 (en) Electronic device and method of restoring device state
US20230186031A1 (en) Electronic device for providing voice recognition service using user data and operating method thereof
US20230169973A1 (en) Electronic device and method for providing list of suggested utterances
KR20240020141A (ko) 언어 모델에 기초하여 사용자 발화를 처리하는 방법 및 전자 장치
JP2020022158A (ja) 音源フィルタリングのための方法およびシステム

Legal Events

Date Code Title Description
AS Assignment

Owner name: LINE CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHA, SEIJIN;CHOI, EONJOUNG;SIGNING DATES FROM 20180420 TO 20180430;REEL/FRAME:045696/0891

Owner name: NAVER CORPORATION, KOREA, REPUBLIC OF

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHA, SEIJIN;CHOI, EONJOUNG;SIGNING DATES FROM 20180420 TO 20180430;REEL/FRAME:045696/0891

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: LINE CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:A HOLDINGS CORPORATION;REEL/FRAME:058597/0303

Effective date: 20211118

Owner name: A HOLDINGS CORPORATION, JAPAN

Free format text: CHANGE OF NAME;ASSIGNOR:LINE CORPORATION;REEL/FRAME:058597/0141

Effective date: 20210228

AS Assignment

Owner name: A HOLDINGS CORPORATION, JAPAN

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE THE CITY SHOULD BE SPELLED AS TOKYO PREVIOUSLY RECORDED AT REEL: 058597 FRAME: 0141. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:LINE CORPORATION;REEL/FRAME:062401/0328

Effective date: 20210228

Owner name: LINE CORPORATION, JAPAN

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE SPELLING OF THE ASSIGNEES CITY IN THE ADDRESS SHOULD BE TOKYO, JAPAN PREVIOUSLY RECORDED AT REEL: 058597 FRAME: 0303. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:A HOLDINGS CORPORATION;REEL/FRAME:062401/0490

Effective date: 20211118

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: Z INTERMEDIATE GLOBAL CORPORATION, JAPAN

Free format text: CHANGE OF NAME;ASSIGNOR:LINE CORPORATION;REEL/FRAME:067069/0467

Effective date: 20231001

AS Assignment

Owner name: LY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:Z INTERMEDIATE GLOBAL CORPORATION;REEL/FRAME:067091/0109

Effective date: 20240329