JP6655081B2 - Electronic device, method for augmenting information requirements, and computer program product for augmenting information requirements - Google Patents

Description

  The present disclosure relates to the field of information retrieval, and specifically to the field of information retrieval based on user inquiries. Even more specifically, the present disclosure relates to the field of interpreting user generated queries.

  Many mobile devices today have the ability to receive verbal questions from a user and then respond with answers using speech synthesis.

  For example, US Patent Application Publication No. 2012/0016678 describes an intelligent automated assistant that interacts with a user using natural language dialogs to answer specific questions from the user. However, such systems are limited to voice input from the user, which can result in false responses by the mobile device. For example, a user may ask a question, but the question may either be misinterpreted by the system, or the system may simply not understand the question at all. This requires the user to follow up with additional voice commands, instructions, etc.

  Other devices provide information about non-verbal (ie, non-speech) input, but do not allow a user to generate and raise questions about non-verbal input.

  For example, U.S. Patent Application Publication No. 2012/0124461 describes a technique for annotating a street-level image with contextual information that is overlaid on the image. Similarly, U.S. Patent Application Publication No. 2013/0346068 describes a system for tagging digital photographs with a number of terms provided by a user. However, this type of technique is limited to automatically populating images with predefined information, regardless of the specific questions that the user may have.

US Patent Application Publication No. 2012/0016678 U.S. Patent Application Publication No. 2012/0124461 U.S. Patent Application Publication No. 2013/0346068

  Thus, alone or in combination, the prior art fails to provide a system that allows a user to ask a question and then augment this question using non-verbal inputs that define the question. That is, the prior art does not provide a system that allows the user to present to the device questions generated by the user regarding non-verbal inputs such as visual images and scents and sounds. The present invention provides one or more solutions to this need that has been felt for many years.

  In one embodiment of the present disclosure, the electronic device comprises a microphone for receiving an oral request for information about an unspecified object. A sensor receives augmentation information about the unspecified subject. Combining logic combines information from the verbal request with the augmented information into a combined format information request. The information retrieval logic then retrieves a response to the combined format information request.

  In one embodiment of the present disclosure, the sensor in the electronic device is a chemical sensor that can “sniff” the subject of a question issued by a user.

  In one embodiment of the present disclosure, the sensor in the electronic device is a camera that captures a video image, and the unspecified object is an object that does not appear in the video image. The electronic device accesses metadata retrieval logic to retrieve metadata describing the video image from the video image captured by the camera. Correlation logic correlates information from the metadata to identify information about unspecified objects and to generate a response to the combined format information request.

  In one embodiment of the present disclosure, a method of augmenting an information request includes receiving, by an information server, a request for information about an object. The request for information is from an electronic device and is in a first format. The information server receives the augmented information about the object. The augmentation information is generated by the electronic device and is of a second format different from the first format. The information server generates a response to the request for information based on the augmented information.

  In one embodiment of the present disclosure, the first format is for speech recognition and the second format is for photographs. The object for which information is required is shown in the photograph. The information server retrieves embedded information in the photo related to the object shown in the photo. The information server locates additional information about the object in the photo based on the embedded information.

  In one embodiment of the present disclosure, the request for information is generated by a particular user. The method further includes associating the request for information with a user profile for a particular user and adapting a response to the request for information based on the user profile.

  In one embodiment of the present disclosure, the information server receives the user's body gesture and uses the body gesture as augmentation information about the subject.

  In one embodiment of the present disclosure, a computer program product augments information requirements. The computer program product includes a computer readable storage medium having the program code embodied therein, wherein the computer readable storage medium is not the temporary signal itself, the program code is readable by a processor to perform the method, and It is feasible. The method includes receiving a request for information about an object, wherein the request for information is from an electronic device and the request for information is in a first format. Receiving the augmented information, wherein the augmented information is generated by the electronic device, and wherein the augmented information is of a second format different from the first format. Generating a response to the request for information.

  Thus, the systems disclosed herein provide technical advantages over the prior art in improving the efficiency and accuracy of information requests. More specifically, without the teachings of the present disclosure, many, if not most, verbal questions would be meaningless and therefore unanswerable by intelligent assistant devices.

  Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings.

FIG. 1 depicts a cloud computing node according to one embodiment of the present disclosure. FIG. 1 illustrates a cloud computing environment according to one embodiment of the present disclosure. FIG. 3 depicts an abstraction model layer according to one embodiment of the present disclosure. FIG. 1 illustrates an exemplary system in which the present invention may be utilized. FIG. 3 is a high-level flowchart of one or more operations performed by one or more processors or other hardware devices to respond to a request for information from an electronic device generated by a user.

  The invention can be a system, method, or computer program product, or a combination thereof. A computer program product may include one or more computer readable storage media having computer readable program instructions for causing a processor to perform aspects of the present disclosure.

  The computer readable storage medium may be a tangible device capable of holding and storing instructions used by the instruction execution device. The computer readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above, but Not limited. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable reads. Dedicated memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy (R) -) Discs, mechanically encoded devices (punched cards or raised structures in grooves, with instructions recorded), and any suitable combination of the above. Computer readable storage media herein refers to radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, optical pulses through fiber optic cables), or wires. It should not be construed as a transient signal itself, such as an electrical signal transmitted over

  The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing / processing device, or may be downloaded from a network (eg, the Internet, a local area network, a wide area network, Or a wireless network, or a combination thereof) to an external computer or external storage device. A network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from a network and transfers these computer readable program instructions to a computer readable storage medium in the respective computing / processing device. Transfer as stored in.

  The computer readable program instructions for performing the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or Write in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk® and C ++, and conventional procedural programming languages such as the “C” programming language and similar programming languages. It may be either source code or object code written. The computer readable program instructions may be executed entirely on the user's computer, partially executed on the user's computer as a stand-alone software package, or partially executed on the user's computer. It may run on a remote computer or run entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or The connection may be made to an external computer (eg, via the Internet using an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), for implementing aspects of the present disclosure, Computer readable program instructions can be executed by personalizing the electronic circuit using state information of the computer readable program instructions.

  Aspects of the disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart description and / or block diagram, and combinations of blocks in the flowchart description and / or block diagram, can be implemented by computer readable program instructions.

  These computer readable program instructions correspond to functions / acts in which instructions executed via a processor of a computer or other programmable data processing device are specified in one or more blocks of a flowchart and / or block diagram. May be provided to the processor of a general purpose computer, special purpose computer, or other programmable data processing device to create a machine to provide a means for implementing These computer-readable program instructions also include instructions that cause the computer-readable storage medium on which the instructions are stored to implement aspects of the function / act specified in one or more blocks in the flowcharts and / or block diagrams. One that is stored on a computer readable storage medium and that can direct a computer, programmable data processor, or other device, or a combination thereof, to function in a particular manner to constitute an article of manufacture. You may.

  The computer readable program instructions also describe the instructions that are executed on the computer, other programmable devices, or other devices, when the instructions / functions are specified in one or more blocks in a flowchart and / or block diagram. As implemented, loaded into a computer, other programmable data processing device, or other device to perform a series of operational steps on the computer, other programmable device, or other device, to create a computer-implemented process. It may be the one that causes it.

  The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, a segment, or a portion of an instruction, which may include one or more executable instructions for implementing the specified logical function (s). including. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, depending on the functions involved, two blocks shown in succession may actually be performed substantially simultaneously, or they may be performed in the reverse order. In addition, blocks in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may perform specified functions or actions, or may contain dedicated hardware and computer instructions. It should also be noted that it can be implemented by a dedicated hardware-based system that performs a combination of.

  It is to be understood that in one or more embodiments, the invention may be implemented in a cloud computing environment.

  Cloud computing is a configurable computing resource that can be quickly provisioned and released with minimal administrative effort or with minimal interaction with service providers (eg, networks, network bandwidth, servers, processing, A model of service delivery to allow convenient on-demand network access to a shared pool of memory, storage, applications, virtual machines, and services). The cloud model may include at least five features, at least three service models, and at least four deployment models.

The features are as follows.
On-demand self-service: Cloud consumers automatically unidirectionally compute computing power, such as server time and network storage, as needed, without the need for human interaction with service providers Can be provisioned.
Broad Network Access: Capabilities are available over the network, and capabilities also facilitate use by heterogeneous thin or thick client platforms (eg, mobile phones, laptops, and PDAs). Accessed via standard mechanisms.
Resource pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, and different physical and virtual resources are dynamically allocated and reallocated according to demand. You. Consumers generally have no control or knowledge about the exact location of the provided resources, but may be able to specify locations at a higher level of abstraction (eg, country, state, or data center). In that respect, there is a sense of place independence.
Rapid resilience: Capabilities can be provisioned quickly and resiliently, sometimes automatically, and scaled out quickly, released quickly and scaled in quickly. To the consumer, the capacity available for provisioning often looks endless and can be purchased at any time and in any quantity.
Measured services: The cloud system automates resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (eg, storage, processing, bandwidth, and active user accounts). Control and optimization. Resource usage can be monitored, controlled, and reported, resulting in transparency for both the provider and the consumer of the service used.

The service model is as follows.
Software as a Service (SaaS): The ability provided to consumers is to use a provider's application running on a cloud infrastructure. Access to the application is possible from various client devices (eg, web-based email) via a thin client interface, such as a web browser. Consumers do not manage or control the underlying cloud infrastructure, including network, server, operating system, storage, and even individual application capabilities, but have limited user-specific applications Configuration settings can be an exception.
Platform as a Service (PaaS): The ability provided to consumers is the ability to create and obtain applications created or obtained by consumers using programming languages and tools supported by the provider. To deploy on infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but control over deployed applications, and possibly an application hosting environment Has control over configuration.
Infrastructure as a Service (IaaS): The ability provided to the consumer is to provision processing, storage, networking, and other basic computing resources, where the consumer , Any software that can include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but control over the operating system, storage, deployed applications, and possibly selected networking components (eg, host firewalls). ) With limited control.

The deployment model is as follows.
Private cloud: The cloud infrastructure operates only for the organization. The cloud infrastructure may be managed by this organization or a third party and may be on-premises or off-premises.
Community Cloud: The cloud infrastructure is shared by several organizations and supports specific communities with shared interests (eg, mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by these organizations or third parties and may be on-premises or off-premises.
Public Cloud: The cloud infrastructure is made available to the general public or large industry associations and is owned by the organization that sells the cloud services.
Hybrid cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public). These clouds remain their own entities, but are brought together by standardized or proprietary technologies that enable data and application portability (e.g., cloud load balancing for cloud-to-cloud load balancing). Bursting).

  Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure that makes up a network of interconnected nodes.

  Referring now to FIG. 1, a schematic diagram of an example of a cloud computing node is shown. Cloud computing node 10 is merely an example of a suitable cloud computing node, and is not meant to imply any limitations as to the scope of use or functionality of the embodiments of the invention described herein. do not do. Regardless, the cloud computing node 10 can be implemented with any of the foregoing functions and / or perform any of the above functions.

  Within the cloud computing node 10 is a computer system / server 12, which also operates in many other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12, include personal computer systems, server computer systems, thin computers, and the like. Clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, main Including, but not limited to, frame computer systems, and distributed cloud computing environments that include any of these systems or devices.

  Computer system / server 12 may be described in the general context of computer system executable instructions executed by a computer system, such as program modules. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

  As shown in FIG. 1, a computer system / server 12 in a cloud computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 include one or more processors or processing units 16, a system memory 28, a bus 18 that couples various system components including the system memory 28 to the processor 16, , But is not limited thereto.

  Bus 18 may use any of a number of types, including memory buses or controllers, peripheral buses, accelerated graphics ports, and processor or local buses, using any of a variety of bus architectures. Represents one or more of the following bus structures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) buses, MicroChannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and PCI (Peripheral Component Interconnects) bus.

  Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

  System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading and writing to non-removable, non-volatile magnetic media (not shown; typically referred to as a “hard drive”). . Although not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (eg, a "floppy (R) disk") and a CD-ROM, DVD-ROM, or other An optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk, such as the optical media described above. In such a case, each drive may be connected to bus 18 by one or more data media interfaces. As will be further shown and described below, memory 28 stores at least one program product having a set (eg, at least one) of program modules configured to perform the functions of the embodiments of the present invention. Can be included.

  A program / utility 40 having a set of (at least one) program module 42 may be stored in memory 28 by way of example and not limitation, and may include an operating system, one or more application programs, other Program modules and program data may also be stored. The operating system, one or more application programs, other program modules, and program data, or some combination thereof, may each include an implementation of a networking environment. Program module 42 generally performs the functions and / or methods of the embodiments of the invention described herein.

  Computer system / server 12 can also communicate with one or more external devices 14. External device 14 may be one or more devices that allow a user to interact with computer system / server 12, such as a keyboard, pointing device, display 24, or one or more devices that computer system / server 12 may have. Any device (eg, network card, modem, etc.) that allows it to communicate with other computing devices, or a combination thereof. Such communication can be performed via an input / output (I / O) interface 22. Further, the computer system / server 12 may be connected via a network adapter 20 to a local area network (LAN), a general wide area network (WAN), or a public network (eg, the Internet), or any of them. It can communicate with one or more networks, such as a combination. As shown, network adapter 20 communicates with other components of computer system / server 12 via bus 18. Although not shown, it should be understood that other hardware components and / or software components may be used with computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

  In one or more embodiments of the present disclosure, external device 14 utilizes the architecture of computer system / server 12 shown in FIG. Similarly, the architecture of the computer system / server 12 may be implemented in the electronic device 402 and / or the information cloud server 404 shown in FIG.

  Referring now to FIG. 2, an illustrative cloud computing environment 50 is depicted. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 and a local computing device used by a cloud consumer, such as a personal digital assistant ( A PDA) or cellular telephone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or a combination thereof, etc., can communicate with the cloud computing node 10. Nodes 10 can communicate with each other. These may be physically or virtually grouped in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud as described above, or a combination thereof. (Not shown). This allows the cloud computing environment 50 to offer infrastructure, platforms, or software, or a combination thereof, as a service that does not require the cloud consumer to maintain resources on the local computing device. . The types of computing devices 54A-N shown in FIG. 2 are merely exemplary, and computing node 10 and cloud computing environment 50 may include any type of network and / or network-capable connection. It should be appreciated that any type of computerized device can be communicated via (eg, using a web browser).

  Referring now to FIG. 3, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 2) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 3 are merely exemplary and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided.

  Hardware and software layer 60 includes hardware and software components. Examples of hardware components are mainframes, in one example an IBM (R) zSeries (R) system, a server based on a reduced instruction set computer (RISC) architecture, in one example an IBM pSeries (R) system, an IBM xSeries (R) ) Systems, IBM BladeCenter (R) systems, storage devices, networks and networking components. Examples of software components include network application server software, in one example, IBM WebSphere® application server software, and database software, in one example, IBM DB2® database software. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.)

  The virtualization layer 62 provides an abstraction layer from which the following examples of virtual entities can be provided. That is, virtual servers, virtual storage, virtual networks including virtual private networks, virtual applications and operating systems, and virtual clients may be provided.

  In one example, management layer 64 can provide the functions described below. Resource provisioning provides for the dynamic sourcing of computing and other resources utilized to perform tasks within a cloud computing environment. Metering and pricing provides the cost tracking as resources are utilized within a cloud computing environment, and the billing or invoicing of the consumption of these resources. In one example, these resources may include an application software license. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. A user portal provides consumers and system administrators with access to a cloud computing environment. Service level management provides cloud computing resource allocation and management so that required service levels are met. Service Level Agreement (SLA) planning and fulfillment provides for the pre-configuration and procurement of cloud computing resources that are expected to be needed in the future according to the SLA.

  Workload layer 66 provides examples of functions for which a cloud computing environment can be utilized. Examples of workloads and functions that can be provided from this layer include mapping and navigation, software development and life cycle management, virtual classroom education delivery, data analysis processing, transaction processing, and as described herein, and Handling information queries from electronic devices, as represented by the "information query process" found in the workload layer 66.

  Referring now to FIG. 4, an exemplary system 400 illustrates various new and novel technical features, technical characteristics and / or interoperability not found in the prior art and in which the present invention may be utilized. Is drawing. More specifically, the elements depicted in FIG. 4 and further described with respect to the operations depicted in FIG. 5, the query logic defines the nature and scope of the information query generated by the user with clear and efficient Solve the problem of not being able to identify. The present disclosure solves this problem through the use of augmented sensors (visual, acoustic, chemical, etc.) that define the nature of the subject of the query.

  An electronic device 402, such as a smart phone, a personal digital assistant, a tablet computer, etc., can communicate with the information cloud server 404 via a network 406. Network 406 may be wired (eg, the Internet), wireless (eg, a Wi-Fi system), or a combination of both, or a combination thereof.

  Electronic device 402 includes a user interface 408, which may prompt the user to “ask a question”, as shown at “ask a question” button 410. The "Ask a Question" button 410 is highlighted when the user presses the "Ask a Question" button 410 (which can be a mechanical button or a touch-sensitive area of the user interface 408). Well, thus indicating that the audio logic 414 has been enabled to receive the user's question via the microphone 412. Speech logic 414 converts the user's question into a digitized format. This format may be processed locally by processor 416 or remotely by information cloud server 404, or both.

  To better understand the user's question, additional information is provided to processor 416 and / or information cloud server 404 as "augmented information." This augmentation information may be in the form of a photograph, video, sound, chemical signature, reference to a document, reference to a graphic file, and the like.

  In one embodiment of the present disclosure, the user's question (i.e., verbal request for information) is initially about an unspecified subject. For example, the question may be "What kind of flower is this?" If only linguistic questions are used, the system cannot "know" which "flower" the user is asking about, so the "flower" in the question is unspecified (i.e., "unspecified" Subject of "). According to one or more embodiments of the present invention, sensors on electronic device 402 provide augmentation information about unspecified objects.

  In one embodiment of the present disclosure, the sensor used to provide this augmentation information is a camera 420, which may be a photograph (ie, a still photo) and / or a video (ie, a visual recording of a moving image) or both. Create Camera 420 is selected by the user pressing camera selector button 418, which activates camera 420 on electronic device 402. After taking the photo / video of the flower with camera 420, the system now has augmented information about the query. That is, when the user asks, "What kind of flower is this?", The processor 416 and / or the information cloud server 404 use verbal logic (eg, part of the processor 416) to verbally Combine the information from the question ("What kind of flower is this?") With the augmentation information provided by the flower picture to create a combined format information request. Using this information binding, information retrieval logic (eg, processor 416) can now retrieve the answer to the binding format information request (eg, from storage device 430). Thus, both verbal / language questions and visual information from photos / videos are provided to the data query logic (eg, part of the “information query process” shown in workload 66 of FIG. 3). You. This allows the data query logic to both "listen" to the question and "see" the subject of the question.

  In one embodiment of the present disclosure, processor 416 and / or information cloud server 404 use image recognition software to identify objects (eg, “flowers”) in the photograph. That is, the processor 416 and / or the information cloud server 404 generate a digital representation of the flower, including the shape, color, number of petals, etc. of the flower, and compare this digital representation with a database of digital information about the flower. , Identify flowers as roses. This information (i.e., the flower type is "rose") is combined with the question ("What kind of flower is this?") To create a combined format information request. In this example, the system simply returns the answer "This flower is a rose" from the augmented information.

  In one embodiment of the present disclosure, the combined format information request allows the data query logic to consult the relevant database. For example, if the query from the user continues to be "What kind of flower is this?" And the augmentation information created by the photo from camera 420 reveals that this flower is a "rose" Assume. The data query logic will then use these keywords (ie, "roses", "flowers") to determine where the roses need to grow and thrive (water, nutrients, sunlight, etc.) You can look up other databases about Roses, including their Latin two-name names. The audio logic 414 can then prompt the user for an offer to provide additional information from the consulted database. In one embodiment, the offer is based on the user's profile stored in a database (not shown), such as that found in storage device 430 of FIG. For example, if the storage device 430 includes a record that the user is a gardener, a horticultural recommendation may be offered, and if the storage device 430 includes a record that the user is a botanical student, the Latin word may be provided. Two-name names may be offered. Similarly, if the user is a chemist (due to the user's stored profile), a statement as to which fragrance is made from this type of rose may be offered.

  In one embodiment of the present disclosure, data query logic (eg, processor 416 and / or information cloud server 404) can provide information about the user himself. For example, assume that storage device 430 includes an entry for a user of electronic device 402 that indicates that the user has had a knee surgery. Further assume that the user takes a picture of his knee with camera 420 and then asks the question "Is my knee OK?" Processor 416 will then compare the current picture of the knee with the previous picture of the knee (stored in the profile for this user in storage device 430). If the knee looks the same as immediately after the user had previously injured the knee, the data query logic can return the answer "Your knee appears to be injured again". However, if the visual image of the knee appears to be "normal" when compared to other photographs of the injured user's knee (or the knee of another patient), the data query logic may return the answer " His knees look good. "

  Similarly, a user may ask data query logic about food options for this particular user. For example, assume that the user of electronic device 402 is on a low salt diet and is standing in front of a food buffet. The user can take a picture of the food buffet with the camera 420 and send this picture to the data query logic with the question "Which food should I eat?" Data query logic uses visual recognition logic to identify various types of food, queries a database of nutritional data on the identified types of food, and allows the user to select specific low salt foods from the photos. Will be returned.

  Clicking on the camera selector button 418 enables image capture, but in one embodiment of the present invention, actuating the camera selector button 418 causes a live stream of data to be sent along with the user's language query. It can be sent to inquiry logic. Similarly, the camera 420 may be always active in one embodiment, so that the user does not need to activate the camera selector button 418.

  Thus, as described above, one or more embodiments of the present disclosure may be implemented in such a manner that the data query logic associated with the electronic device 402 "listens" for a question (by a user clicking a "question" button 410). As well as "see" what the user is asking about. Although the above embodiments have been presented as using photos, the same process may be utilized for video streams.

  In one embodiment of the present disclosure, the electronic device 402 can not only "listen" for a question (by the user clicking the "Ask a Question" button 410), but also allow the user to press the microphone selector button 422. May also "hear" what the user is asking about, and the microphone selector button 422 activates the microphone 412 on the electronic device 402. For example, suppose the user asked the question "What kind of bird is that?" If only linguistic questions are used, the system cannot "know" what "birds" the user is asking about. However, a microphone 412 (audio / hearing sensor) on the electronic device 402 provides an augmentation on the unspecified object ("bird") according to its call. In one embodiment of the present disclosure, the processor 416 and / or the information cloud server 404 use sound pattern recognition software to identify "birds" heard during sound capture as "Makibirds." That is, the processor 416 and / or the information cloud server 404 generate a digital representation of the bird call, including the pitch, pattern, timbre, etc. of the bird call, and convert the sound signature to digital information about the bird call. Birds are identified as madrids, compared to a database. This information (i.e., the bird type is "Makibird") is combined with the question ("What kind of bird is that?") And the answer "This bird is a Makibird" is returned. Both the / language query and the augmented information from the recording / sound capture are provided to the data query logic (eg, part of the "information query process" shown in workload 66 of FIG. 3). , The data query logic can both "listen" to the question and "listen" to the subject of the question.In one example, the system simply responds to the augmented information found in the recording " Is returned. "

  In one embodiment of the present disclosure, the combined format information request allows the data query logic to consult an associated database based on what the data query logic has “heard” from the object being queried. For example, suppose that the query from the user continues to be "What type of bird is this?", And that the augmentation information created by the recording from microphone 412 reveals that this bird is a dragonfly. . In this case, the data query logic can track other databases relating to the squid, including where the squid is inhabited, whether it is a game, and its Latin two-name. The voice logic 414 can then prompt the user for an offer to provide such additional information. In one embodiment, the offer is based on the user's profile. For example, if the storage device 430 contains a record that the user is a huntsman, the data query logic will offer (or automatically reveal) to the huntsman that the dragon is not a gamebird (and therefore should not be shot). )be able to. Similarly, if the storage device 430 includes a record that the user is a bird observer (ornithologist), a Latin two-name may be offered to the user by the data query logic.

  In one embodiment of the present disclosure, once an object is identified by its image and / or sound, a suggested action may be presented to a user. For example, assume that a user of electronic device 402 has taken an audio / visual recording of a motor and has asked the question "Is this motor running properly?" Image recognition software will identify the type of motor from the video component of the recording. Sound pattern recognition will identify any motor abnormalities by comparing the audio component of the recording to a digital database of normal and / or abnormal sounds for this motor. That is, the digitized sound pattern from the audio portion of the recording is used to identify any problems with the motor, which translates the digitized sound pattern into a known sound signature for this motor (normal And anomalies). The system can then access a database of solutions to the identified problems and present these solutions to the user. For example, if the sound signature for this motor indicates that the main bearing has failed, the system will provide a procedure for replacing the main bearing, which part number is associated with the required main bearing. Or an estimate of how long the task will take (and therefore how long the system using the motor will be idle).

  In one embodiment of the present disclosure, the electronic device 402 can not only "listen" for a question (by the user clicking the "Ask a Question" button 410), but also the user presses the chemical sensor selector button 424. This may also "sniff" what the user is asking about, and the chemical sensor selector button 424 activates the chemical sensor 426 on the electronic device 402. For example, suppose a user is standing next to a wheelbarrow in a food store and asks the question "What type of food is this?" If only linguistic questions are used, the system cannot "know" what "food" the user is asking about. However, chemical sensors 426 on the electronic device 402 provide augmentation for unspecified subjects according to their aroma. In one embodiment of the present disclosure, the processor 416 and / or the information cloud server 404 use chemical pattern recognition software to convert the “smell” received by the chemical sensor capture into a “hot dog”. Identify. That is, the processor 416 and / or the information cloud server 404 may use the hot dog aroma generated by one or more of the sensors in the chemical sensor 426 that may detect chemical signatures such as beef, salt, and some spices. A descriptive digital file is generated and the chemical signature is compared to a database of digital information about the food to identify the food as a hot dog. This information (ie, the food type is "hot dog") is combined with the question ("What type of food is this?") And the answer "This food is a hot dog" is returned. In this manner, both the verbal / language questions and the augmented information from the chemical signature are provided to the data query logic (eg, part of the “information query process” shown in workload 66 of FIG. 3). . This allows the data query logic to both "listen" to the question and "sniff" the subject of the question. In this example, the system simply returns the answer "This food is a hot dog" from the augmented information found in the chemical signature. In additional embodiments, this information is then used, such as based on the user's profile, and sold by the store whether the user should eat hot dogs in the first place (based on the user's medical history). Suggestions may be provided as to which hot dogs are the highest rated, higher rated hot dog shops nearby, and the like.

  In one embodiment of the present disclosure, and as shown in FIG. 4, the electronic device 402 includes a transceiver 434, which can send and receive electronic messages either wirelessly or over a wire. it can. Transceiver 434 may send a combined format information request (discussed above) to an information server.

  In one embodiment of the present disclosure, the information server is a local device on electronic device 402, such as processor 416. Thus, any queries are handled locally by the electronic device 402.

  In one embodiment of the present disclosure, the information server is a remote device such as the information cloud server 404, and the information cloud server 404 can handle information requests from multiple electronic devices.

  In one embodiment of the present disclosure, the information server is a combination of a local device (eg, processor 416) and a remote device (eg, information cloud server 404).

  In one or more embodiments of the present disclosure, electronic device 402 comprises a positioning system, such as illustrated Global Positioning System (GPS) 432. GPS 432 utilizes a series of earth-synchronous satellites to mark the position of electronic device 402 at any given time. Other positioning logic (not shown) includes a triangulation system (locating the device using signals from a transmitter in the room), a Doppler shift system (a positioning transmitter / receiver where the signal is known). To the electronic device 402 and use the time it takes to return to the positioning transmitter / receiver). Thus, metadata about the real-time location of the electronic device 402 when a question is asked and / or when augmented information (eg, photos, recordings, chemical readings, etc.) are generated, or both, Created by GPS432.

  In one embodiment of the present invention, data query logic (eg, processor 416 and / or information cloud server 404) is used to answer questions about the object being viewed / sniffed / other that is being queried. Not enough information may be stored locally. In this scenario, the data query logic can access an information resource server 436, such as a system that provides access to information resources such as websites and databases. Such information resources are selectively accessed / retrieved using web crawlers, search engines, database search logic, etc. to access particular information. For example, suppose a user asks, "What fragrance is made from this flower?" And the data query logic receives both the question and a picture of the flower, and determines that the flower is a rose. The data query logic will in this case crawl / query the information resources provided by the information resource server 436 to determine which fragrances (including product names, manufacturers, etc.) use the rose essence in the formulation. Will be identified.

  Referring now to FIG. 5, a user generated information request from an electronic device is implemented by one or more processors or other hardware devices to respond via an augmentation of the information request. A high-level flowchart of one or more operations is presented.

  After the initiator block 502, the information server receives a request for information about the subject (block 504). As described in FIG. 4, the information server may be local (eg, processor 416), remote (eg, information cloud server 404), or a combination thereof, or both. The request for information is from an electronic device (eg, electronic device 402 in FIG. 4). The request for information is in a first format, such as the format used in voice conversion / digitization by voice logic 414 in FIG. For example, the request for information may be from a user who “asks” the electronic device 402 for questions regarding a particular subject / object. However, what the user is asking about is initially not clear to the information server.

  Accordingly, at block 506, the information server receives augmentation information about the subject. This augmentation information is generated by the electronic device and is of a second format different from the first format. For example, the first format may be a format used by a voice digitizer to convert a user's verbal question to a digitized format. The second format may be the format used by photos, video files, chemical signatures, text documents, graphic files, and the like.

  As described in block 508, the information server may then generate a response to the request for information based on the augmented information. That is, as described above, subjects such as "flowers", "motors", and "knees" are now described with better specificity by attached photos / recordings / chemical signatures / etc. To respond better.

  Thus, in one embodiment of the present disclosure, the first format is for speech recognition and the second format is for a video file generated by a camera on an electronic device.

  Thus, in one embodiment of the present disclosure, the first format is for speech recognition and the second format is for an audio file generated by a microphone on the electronic device.

  Thus, in one embodiment of the present disclosure, the first format is for speech recognition and the second format is for a chemical signature generated by a chemical sensor on the electronic device.

  In one embodiment of the present disclosure, the first format is for speech recognition and the second format is for a text document received from an electronic device. For example, assume that the user has pressed the document selector button 428 of FIG. This causes the processor 416 and / or the information cloud server 404 to retrieve the text document, which was previously stored (eg, on the storage device 430) or on the user interface 408. That is currently created by the user drawing at (eg, using the user's fingertip on a touch-sensitive portion of the screen dedicated to receiving free-form text that is converted to ASCII characters). There is either. The user can then ask questions such as "What is this text document about?", "Is this document suitable for sending to customers?" Similarly, the query may be "What does this text document mean?" The data query logic can then compare the text in the selected text document (or meta tags from the selected text document) to a database describing the document to answer such questions.

  In one embodiment of the present disclosure, the first format is for speech recognition and the second format is for graph documents stored on the electronic device. For example, assume that the user has pressed the document selector button 428 of FIG. This causes processor 416 and / or information cloud server 404 to retrieve the graphics file, which may have been previously stored (eg, on storage device 430) or • currently created by the user drawing on the interface 408 (eg, using the user's fingertips on a touch-sensitive portion of the screen dedicated to receiving free-form text that is converted to ASCII characters) That is either. The user can then ask questions such as "What does this diagram represent?", "What does this graph tell about current sales?" The data query logic can then extract information from the diagrams / graphs (including meta tags) using a database describing other graphs / diagrams to answer such questions.

  In one embodiment of the present disclosure, the first format is for speech recognition, the second format is for photos, and the object being asked is shown in the photos. In this embodiment, the information server retrieves embedded information about the object shown in the picture from the picture, and the embedded information describes the object shown in the picture. The information server then locates additional information about the object in the photo based on the embedded information, so that the additional information is superior to the embedded information. That is, as described above, additional information about the object shown in the photo may be extracted using the embedded information in the photo, such as a meta tag. For example, assume that a photo was taken by the camera 420 shown in FIG. 4 and that GPS 432 or similar positioning logic placed a meta tag on the photo indicating the time, date, and location where the photo was taken. Further assume that the photo is the user and friend's "selfie" and the question (which was asked immediately or at a later date and time) is "When is my friend's birthday to the right of the picture?" . By correlating metadata from photos (eg, tags identifying friends) with a database of information (eg, from social media websites to which the user has access), then the friend's birthday May be derived by data query logic (eg, the “information query process” shown in workload 66 of FIG. 3).

  In one embodiment of the present disclosure, the first format is for speech recognition, the second format is for photos, and the object being asked is not shown in the photos. In this embodiment, the information server utilizes the embedded information in the photo to retrieve additional information about objects not shown in the photo, and this additional information overrides the embedded information. For example, assume that a photo was taken by the camera 420 shown in FIG. 4 and that GPS 432 or similar positioning logic placed a meta tag on the photo indicating the time, date, and location where the photo was taken. In addition, the photo is the user and friend's "selfie" at the music festival, and the question (submitted immediately or at a later date and time) is "What band was playing when this picture was taken?" Suppose, however, that the band being asked is not in the picture. By correlating the metadata from the photo with the band's schedule at any particular time (and even songs from the playlist), the name of the band at the time of the photo (and / or the song being played) May be derived by data query logic (eg, the “information query process” shown in workload 66 of FIG. 3).

  In one embodiment of the present disclosure, the request for information is generated by a particular user, and the method further comprises associating the request for information with a user profile for the particular user, and generating the information based on the user profile. Tailoring the response to the requested request. For example, as described above, information on birds differs depending on whether the user is a huntsman or an ornithologist.

  In one embodiment of the present disclosure, the information server receives the user's body gesture and then uses the body gesture as additional augmentation information about the subject. For example, assume that a user of electronic device 402 is taking video and / or photos of a group of cars. The user may be interested in a particular car make and model (or, in the case of a race car on a track, a particular race car driver) in the video / photo. The user may point at or point with a thumb or thumb to specify which car is the subject of the user's question "What kind of car is that?" Or "Who is the driver of that race car?" Hand gestures, such as a circle made with your index finger, can be included in the photo / video to specify which car you are interested in.

  As described herein, in one or more embodiments, the present disclosure provides that a user can request a search using voice recognition software and more regarding a search request through the use of non-voice input. Allows you to provide details. The present disclosure enables intelligent automation assistants with voice recognition capabilities to analyze drawings, photos, sounds, chemical signatures, etc. to better understand the nature of search requests. As described herein, a user may direct a mobile device (eg, a smart phone) to a subject in real-time, to learn more about a property of the subject (eg, visual, tonal, scent, etc.). Can be found. Thus, the present disclosure presents technical improvements over the prior art that have heretofore been unavailable and / or known to those skilled in the art.

  The terms used in the description are intended to describe certain embodiments only, and shall by no means restrict the present invention. As used herein, the singular forms “a”, “an”, and “the” include plural forms unless the context clearly dictates otherwise. Furthermore, when the term “comprising” is used herein (“comprises” and / or “comprising”), these terms denote the recited feature, integer, step, act, element or component, It is also understood that specifying the presence of, or combinations thereof, does not preclude the existence or addition of one or more other features, integers, steps, acts, elements, components, or groups thereof, or combinations thereof. Will be done.

  The corresponding structures, materials, acts and equivalents of all means or step-plus-function elements in the following claims perform their functions in combination with the other specifically claimed elements Or any structure, material, or act of doing so. The description of various embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the present disclosure. The embodiments describe the invention in other ways in order to best explain the principles and practical application of the present disclosure, and in the case of various embodiments with various modifications to suit the particular use contemplated. It has been chosen and described so that it can be understood by the trader.

  Any of the methods described in this disclosure may be implemented through the use of VHDL (VHSIC Hardware Description Language) programs and VHDL chips. VHDL is an exemplary design input language for field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and other similar electronic devices. Accordingly, any of the software implementation methods described herein may be emulated by a hardware-based VHDL program, which is then applied to a VHDL chip, such as an FPGA.

  Thus, while embodiments of the present disclosure of the present application have been described in detail and with reference to exemplary embodiments thereof, they depart from the scope of the present disclosure as defined in the appended claims. Obviously, modifications and variations are possible.

Claims (12)

An electronic device,
A microphone for receiving verbal requests for information about unspecified subjects;
A sensor for receiving augmentation information about the unspecified subject;
Combining logic for combining information from the verbal request with the augmented information into a combined format information request;
Information retrieval logic for retrieving a response to the combined format information request ,
An electronic device , wherein the sensor is a camera . An electronic device,
A microphone for receiving verbal requests for information about unspecified subjects;
A sensor for receiving augmentation information about the unspecified subject;
Combining logic for combining information from the verbal request with the augmented information into a combined format information request;
Information extraction logic for extracting a response to the combined format information request;
With
The sensor is a camera that captures a video image, the unspecified object is an object that does not appear in the video image, and the electronic device includes:
Metadata retrieval logic for retrieving metadata describing the video image from the video image captured by the camera;
The meta and information correlated from the data, the identifying information about the unspecified object, the correlation logic and access to that electronic device to generate an answer to the binding format information request. 3. The electronic device according to claim 1, further comprising a transmitter for transmitting the combined format information request to an information server. A method to augment information requirements,
Receiving, by an information server, a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format; ,
Receiving, by the information server, augmentation information about the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is in a second format different from the first format; Receiving and
By the information server based on said augmented information, see containing and generating a response to the request for information,
The method wherein the first format is for speech recognition and the second format is for a video file generated by a camera on the electronic device . A method to augment information requirements,
Receiving, by an information server, a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format; ,
Receiving, by the information server, augmentation information about the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is in a second format different from the first format; Receiving and
Generating, by the information server, a response to the request for information based on the augmented information;
Including
The first format is for a voice recognition, the second format, der Ru way intended for audio files generated by a microphone on the electronic device. A method to augment information requirements,
Receiving, by an information server, a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format; ,
Receiving, by the information server, augmentation information about the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is in a second format different from the first format; Receiving and
Generating, by the information server, a response to the request for information based on the augmented information;
Including
The first format is for speech recognition, the second format is for a photo, the object is indicated in the photo,
By the information server, for the object shown in the picture, extracting embedded information in the picture, wherein the embedded information describes the object shown in the picture, the extracting,
By the information server, further including METHODS and this and the <br/> to locate additional information about the object in the photograph on the basis of the embedded information. A method to augment information requirements,
Receiving, by an information server, a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format; ,
Receiving, by the information server, augmentation information about the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is in a second format different from the first format; Receiving and
Generating, by the information server, a response to the request for information based on the augmented information;
Including
The first format is for speech recognition, the second format is for a photo, the object is not shown in the photo,
Wherein the information server, by using the embedded information in the photo, further including METHODS that you take out the additional information about the object that is not represented by the in photographs. Said request for information is generated by a specific user;
Associating the request for information with a user profile for the particular user;
8. The method of any one of claims 4 to 7 , further comprising adapting the response to the request for information based on the user profile. Receiving, by the information server, a user's body gesture;
The method according to any one of claims 4 to 8 , further comprising utilizing the body gesture as additional augmentation information about the subject by the information server. A computer program for augmenting an information request, said program readable and executable by a processor to perform a method, said method comprising:
Receiving a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format;
Receiving the augmentation information for the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is of a second format different from the first format;
Based on the augmented information, see containing and generating a response to the request for information,
A computer program , wherein the first format is for speech recognition and the second format is for a video file . A computer program for augmenting an information request, said program readable and executable by a processor to perform a method, said method comprising:
Receiving a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format ;
Receiving the augmentation information for the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is of a second format different from the first format;
Generating a response to the request for information based on the augmented information;
Including
The first format is for speech recognition, the second format is for a photo, the object is indicated in the photo, and the method further comprises:
Regarding the object shown in the picture, extracting embedded information in the picture, wherein the embedded information describes the object shown in the picture;
The buried including computer program to <br/> and this to locate additional information about the object in the photograph on the basis of the information. A computer program for augmenting an information request, said program readable and executable by a processor to perform a method, said method comprising:
Receiving a request for information about an object, wherein the request for information is from an electronic device, and wherein the request for information is in a first format;
Receiving the augmentation information for the subject, wherein the augmentation information is generated by the electronic device, and wherein the augmentation information is of a second format different from the first format;
Generating a response to the request for information based on the augmented information;
Including
The first format is for speech recognition, the second format is for a photograph, the object is not shown in the photograph, and the method further comprises:
It said using the embedded information in the photograph, including computer programs that you take out additional information about the object that is not represented by the above in the photograph.

Images