CN112818141A

CN112818141A - Searching method and device

Info

Publication number: CN112818141A
Application number: CN202110149087.9A
Authority: CN
Inventors: 朱佳贤; 姜旼廷; 姜友植; 孔元根
Original assignee: Samsung Electronics Co Ltd
Current assignee: Samsung Electronics Co Ltd
Priority date: 2014-05-23
Filing date: 2015-05-25
Publication date: 2021-05-18
Also published as: CN111046197A; CN110825928A; CN105095341A

Abstract

A method of an electronic device is provided, comprising: presenting an image including at least one product on a display of an electronic device; receiving a voice query from a user of the electronic device related to the at least one product in the image; generating a search query for information related to the at least one product based on a voice query including the image of the at least one product and related to the at least one product; obtaining search results including information related to the at least one product using the search query; and presenting the obtained search results including information related to the at least one product on a display of the electronic device.

Description

Searching method and device

The present application is a divisional application of an invention patent application having an application date of 2015, 05 and 25, application number of 201510272556.0 and an invention name of "search method and apparatus".

Cross reference to related applications

This application claims the priority of korean patent application No. 10-2014-0062568, issued to the korean intellectual property office on day 5 and 23 2014, korean patent application No. 10-2014-0167818, issued to the korean intellectual property office on day 11 and 27 2014, and korean patent application No. 10-2015-0025918, issued to the korean intellectual property office on day 24 and 2015, and is a partial continuation of us non-temporary patent application No. 14/588,275, issued to the us patent and trademark office on day 12 and 31 2014, the disclosures of which are incorporated herein by reference in their entirety.

Technical Field

Apparatuses and methods consistent with exemplary embodiments relate to searching and acquiring information in a computing environment, and more particularly, to performing a search based on various requirements of a user.

Background

Various methods of searching for and obtaining information have been developed. Generally, text-based searches are performed to search for information in a computing environment. Text-based searches use search queries that include one or more text components, such as words or phrases. The text components are matched against each other or against an index or data to identify documents, such as web pages, that include similar text content, metadata, filenames, or textual representations as the text components.

As the technology advances, information to be searched for is further diversified, and the amount of such information increases. Thus, in addition to text components, components of different modalities may be used to perform a search.

Disclosure of Invention

Some aspects of one or more exemplary embodiments provide a method and apparatus that receives a single-modal query or a multi-modal query and performs a search using the received query.

Additional aspects will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the exemplary embodiments.

According to an aspect of the exemplary embodiments, there is provided an apparatus, including: at least one input device configured to receive a first query input of a first query type and a second query input of a second query type; and a controller configured to output a query input window including a first display item corresponding to a first query input and a second display item corresponding to a second query input, automatically switch the apparatus from a first state to receive the first query input of the first query type to a second state to receive the second query input of the second query type in response to receiving the first query input, and obtain a search result according to a query based on the first query input and the second query input.

The second query type may be an audio query type; and in response to receiving the first query input, the controller may be further configured to automatically activate a microphone configured to receive a second query input.

The second query type may be an image query type; and in response to receiving the first query input, the controller may be further configured to automatically activate a camera configured to receive a second query input.

In response to receiving the mode switch input, the controller may be further configured to switch the search mode from a multi-modal input mode, in which the first query input and the second query input are received via the query input window and combined to generate the query, to a single input mode, in which an input of one query type is received to generate the query.

The at least one input device may include a first input device configured to receive a first query input and a second input device distinct from the first input device and configured to receive a second query input.

According to an aspect of another exemplary embodiment, there is provided an apparatus including: a display configured to display a query input window; at least one input device configured to receive a first query input of a first query type and a second query input of a second query type; and a controller configured to obtain search results from a query based on the first query input and the second query input, wherein the display is further configured to simultaneously display a first region corresponding to the first query type and a second region corresponding to the second query type on the query input window.

The controller may be further configured to determine a first query type for the first query input and a second query type for the second query input; and the display may be further configured to display the first region according to the determined first query type and the second region according to the determined second query type.

The display may be further configured to display a query input window in which a first display item corresponding to the first query input and a second display item corresponding to the second query input are simultaneously displayed, such that the first query type and the second query type are distinguishable from each other.

According to an aspect of another exemplary embodiment, there is provided an apparatus including: a display; a microphone configured to acquire voice information; a camera configured to acquire image data; a memory configured to store text data, image data, and audio data; and a controller configured to display a display item for selecting a query type, display a query input window corresponding to the query type selected by the display item, obtain a search result based on a query input received through the query input window, and control the display to display the search result, wherein the query input includes at least one of image data obtained by the camera, text data stored in the memory, image data stored in the memory, and audio data stored in the memory.

The query type may be from among a plurality of query types including a text query, an image query, and an audio query; and when the selected query type is an audio query, the controller may be further configured to control the display to display at least one of a display item for receiving voice information obtained through the microphone as the query input and a display item for receiving audio data stored in the memory as the query input on the query input window.

The query type may be from among a plurality of query types including a text query, an image query, and an audio query; and when the selected query type is an image query, the controller may be further configured to control the display to display at least one of a display item for receiving image data obtained by the camera and a display item for receiving image data stored in the memory as a query input on the query input window.

The apparatus may further comprise: a handwriting input unit configured to receive a handwritten image, wherein the query type may be from among a plurality of query types including a text query, an image query, an audio query, and a handwritten query, and wherein when the selected query type is the handwritten query, the controller may be further configured to control the display to display a display item for receiving the handwritten image on the query input window.

When the plurality of query types are selected through the display item, the controller may be further configured to control the display to display the display item for receiving the plurality of query inputs on the query input window.

According to an aspect of another exemplary embodiment, there is provided a method including: receiving a first query input of a first query type and a second query input of a second query type; outputting, by an apparatus, a query input window comprising a first region corresponding to a first query input and a second region corresponding to a second query input; automatically switching the apparatus from a first state to receive a first query input of a first query type to a second state to receive a second query input of a second query type in response to receiving the first query input; and obtaining search results from a query based on the first query input and the second query input.

The method may further comprise: in response to receiving the second query input, simultaneously displaying a second display item corresponding to the second query input on the second region and a first display item corresponding to the first query input on the first region.

The second query type may be an audio query type; and automatically switching may include automatically activating a microphone for receiving a second query input in response to receiving the first query input.

The second query type may be an image query type; and automatically switching may include automatically activating a camera for receiving the second query input in response to receiving the first query input.

According to an aspect of another exemplary embodiment, there is provided a method of obtaining a search result by an apparatus, the method including: displaying a display item for selecting a query type; receiving user input based on the displayed display items; selecting at least one query type based on the received user input; displaying a query input window corresponding to the selected at least one query type; and obtaining search results based on query input received through the displayed query input window.

The query type may be from among a plurality of query types including a text query, an image query, and an audio query; and the method may further comprise: when the selected query type is an audio query, display items for receiving voice data obtained through a microphone included in or connected to the apparatus and audio data stored in a memory included in or connected to the apparatus as a query input are displayed on a query input window.

According to an aspect of another exemplary embodiment, there is provided a method including: displaying a query input window; receiving text data and a handwritten image through the displayed query input window; and obtaining a search result based on a combined result of the received text data and the received handwritten image.

More specifically, according to an aspect of another exemplary embodiment, there is provided a method of an electronic device, including: presenting an image including at least one product on a display of an electronic device; receiving a voice query from a user of the electronic device related to the at least one product in the image; generating a search query for information related to the at least one product based on a voice query including the image of the at least one product and related to the at least one product; obtaining search results including information related to the at least one product using the search query; and presenting the obtained search results including information related to the at least one product on a display of the electronic device.

Drawings

These and/or other aspects will become apparent and more readily appreciated from the following description of the exemplary embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a flow chart illustrating a process of performing a search in accordance with an exemplary embodiment;

FIG. 2 is a conceptual diagram illustrating a configuration of a user interface according to an exemplary embodiment;

FIG. 3 is a conceptual diagram illustrating a structure for constructing a query input device according to an exemplary embodiment;

FIG. 4 is a conceptual diagram illustrating a network environment for carrying out the illustrative embodiments;

fig. 5 to 7 are exemplary diagrams illustrating a method of receiving a query for a single search according to an exemplary embodiment;

fig. 8 to 18 are exemplary diagrams illustrating a method of receiving a query for a multi-modal search according to an exemplary embodiment;

FIGS. 19 and 20 are exemplary diagrams illustrating a method of receiving a query for a multi-modal search according to another exemplary embodiment;

fig. 21 and 22 are exemplary diagrams illustrating a method of generating a query input window according to an exemplary embodiment;

FIG. 23 is an exemplary diagram illustrating a method of receiving a query for a multi-modal search according to another exemplary embodiment;

FIG. 24 is an exemplary diagram illustrating a list of multimodal query types in accordance with another exemplary embodiment;

fig. 25A to 25E are exemplary diagrams illustrating a method of receiving a query for a multi-modal search according to another exemplary embodiment;

FIG. 26 is a flowchart illustrating a process of displaying a query input window according to an exemplary embodiment;

fig. 27 and 28 are exemplary diagrams illustrating a method of displaying a query input window according to an exemplary embodiment;

FIG. 29 is a flowchart illustrating a process of receiving a query in accordance with an illustrative embodiment;

fig. 30 to 32 are exemplary diagrams illustrating a method of receiving a query according to an exemplary embodiment;

FIG. 33 is a flowchart illustrating a process of receiving a query in accordance with an illustrative embodiment;

FIG. 34 is an exemplary diagram illustrating a method of receiving a query component with a query type that is text in accordance with exemplary embodiments;

fig. 35 to 38 are exemplary diagrams illustrating a method of receiving a query component in which a query type is an image according to an exemplary embodiment;

FIG. 39 is an exemplary diagram illustrating a method of receiving a query component whose query type is sketch information according to an exemplary embodiment;

FIG. 40 is an exemplary diagram illustrating a method of receiving a query component in which the query type is audio information according to an exemplary embodiment;

FIG. 41 is an exemplary diagram illustrating a method of receiving a query including a plurality of query components having a plurality of query types and displaying search results in accordance with an exemplary embodiment;

FIG. 42 is a flowchart illustrating a process of performing a search in accordance with an exemplary embodiment;

FIG. 43 is an exemplary diagram illustrating results obtained by performing a multimodal search in accordance with an exemplary embodiment;

FIG. 44 is a flowchart illustrating a process of displaying search results according to another exemplary embodiment;

FIG. 45 is an exemplary diagram illustrating the priority of search results according to another exemplary embodiment;

FIG. 46 is an exemplary diagram illustrating results obtained by performing a multi-modal search in accordance with another exemplary embodiment;

fig. 47 is an exemplary diagram illustrating a method of receiving a query using a plurality of devices according to an exemplary embodiment and a method of acquiring an image of an area viewed by a user from an entire screen of a Television (TV) by smart glasses according to an exemplary embodiment:

FIG. 48 is an exemplary diagram illustrating a method of receiving query components via a plurality of devices according to another exemplary embodiment;

FIG. 49 is a flowchart illustrating a process of performing a search in accordance with an exemplary embodiment;

FIG. 50 is a flowchart illustrating a process of performing a search in accordance with an exemplary embodiment;

FIG. 51 is a block diagram illustrating the structure of a query input device according to an exemplary embodiment;

fig. 52 to 57 are exemplary diagrams illustrating a method of performing a multi-modal search according to another exemplary embodiment;

FIG. 58 is an exemplary diagram illustrating a method of displaying a list of query types according to another exemplary embodiment;

FIG. 59 is an exemplary diagram illustrating a method of displaying a list of query types according to another exemplary embodiment;

FIG. 60 is an exemplary diagram illustrating a method of generating a list of multimodal query types in accordance with an illustrative embodiment;

fig. 61 to 65 are exemplary diagrams illustrating a method of performing a multi-modal search according to another exemplary embodiment;

FIG. 66 is a block diagram illustrating a framework for performing multi-modal searches based on queries having multiple query types, in accordance with an illustrative embodiment;

FIG. 67 illustrates a diagram of a system for implementing one or more exemplary embodiments; and is

Fig. 68 to 72 are exemplary diagrams illustrating a process of extracting image features of a query component having an image query type according to one or more exemplary embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. The exemplary embodiments may have different forms herein, and should not be construed as limited to the description set forth herein. Accordingly, the following description of the exemplary embodiments is made merely for the purpose of illustrating aspects of the present description, with reference to the accompanying drawings. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of …" modify the entire list of elements when preceding the list of elements, but do not modify the individual elements in the list. The expression "at least one" modifies the list of all elements without modifying each element comprised in the list.

In the following disclosure, when a component (or element, device, etc.) is referred to as being "connected" to another component (or element, device, etc.), it should be understood that the former may be "directly connected" to the latter, or "electrically connected" to the latter via intervening components (or elements, devices, etc.). In addition, when a component (or element, device, etc.) is described as including (or containing or having) some elements, it should be understood that it may include (or contain or have) only those elements, or it may include (or contain or have) other elements in addition to those elements, if not specifically limited.

In this specification, a query denotes a command for performing a search. The query may include information to be obtained as a search result. The query may include at least one query component (i.e., query input). The query component represents the units of information that make up the query. In addition, a query component input to a device (e.g., a query input device) may be referred to as a query input. For example, the query components may include at least one of keywords, image data, sketch information, video data, and audio data included in text input by the user to the query input device. The audio data may include voice information. The query type may represent a modality (modality) of the query component. Here, a modality is defined to include a source of information about a search database algorithmically used for searching, in addition to the sensory aspects of humans. For example, the query type may indicate which type of text, image data, sketch information, video data, and audio data the query component corresponds to. A query representation that includes a plurality of query types indicates that a plurality of query components included in the query correspond to the plurality of query types. For example, when a query includes a first query component whose query type is text and a second query component whose query type is an image, the query includes a plurality of query types. That is, the query type may be at least one of a text query, an image query, an audio type, and a handwritten query.

The audio data may include at least one of sound, speech, audio, and music. In this specification, a search mode may mean an operation mode with respect to a method of performing a search. The search mode may include a single input mode (i.e., a single input mode) and a multi-modal input mode (i.e., a multi-modal input mode). The single input mode may represent an operational mode in which a search is performed based on a query that includes one or more query components having one query type. The multimodal input mode may represent a mode of operation in which a search is performed based on a query that includes a plurality of query components having a plurality of query types.

Hereinafter, exemplary embodiments will be described in detail.

Fig. 1 is a flowchart illustrating a process of performing a search according to an exemplary embodiment.

Referring to fig. 1, in operation S110, a computing device may display a query input window enabling a user to input a query. The query input window represents a user interface as follows: the computing device receives a query input by a user through the user interface. The query input window may include a query input tool for inputting query components. For example, the query input window may include at least one of a text box, a Uniform Resource Locator (URL) input window, a file upload button, a sketch input window, an application execution button, and so on. A text box represents a box in which text entered using an input device (e.g., a keyboard, keypad, virtual keypad, etc.) is displayed. The URL input window represents a user interface through which a URL indicating a location of information to be input as a query is input. The file upload button denotes a button for performing an operation of the computing device to select a file when the file is selected by the user and an operation to upload the selected file. The sketch input window represents a user interface that displays information input by a user using an input device (e.g., a tablet, touch panel, track pad, mouse, stylus, etc.). The application execution button denotes a button that enables an application corresponding to the selected button to be executed. For example, when an application execution button corresponding to photographing is selected, an application for photographing may be executed. As another example, when an application execution button corresponding to recording of sound is selected, an application for recording sound may be executed. As another example, the query input device may extract information such as a Quick Response (QR) code from an image acquired by operating a camera, and receive a query component using the QR code.

According to an exemplary embodiment, when the search mode corresponds to a multi-modal search (i.e., a multi-modal input mode), the query input device may display a query input window for receiving a plurality of query components (i.e., query inputs) having a plurality of query types. Alternatively, the query input device may display a single query input window when the search mode corresponds to a single search (i.e., a single input mode). A single query input window represents a query input window that receives queries that include only one query type.

Subsequently, the query input device may receive a query through the displayed query input window in operation S120. Here, the query input device may receive a query including a plurality of query types, i.e., a query including a plurality of query components of a plurality of query types. That is, a query may be received that includes a plurality of query components of a plurality of query types.

The query input device may then select at least one search result based on the received query. Here, a search engine may be utilized to select at least one search result. A search engine represents hardware, software, or a combination thereof that searches for information based on a query. The search engine may be included in the query input device or may be included in a separate device (e.g., a web server, media server, web server, etc.). When the search engine is included in a separate device, the query input device may transmit the received query to the separate device. The query input device may obtain the selected search results from the separate device in response to the transmitted query.

In operation S130, the query input device may display the selected search result. Here, the query input device may prioritize the search results. The query input device may display the search results based on the priority of the search results.

According to an exemplary embodiment, the query input device may convert text included in the search result into voice (or communicate with a server or another device to perform the conversion), and output the converted voice. Alternatively, the query input device may convert speech included in the search result into text and output the converted text. Thus, the query input device enables blind or hearing impaired people to view search results.

Fig. 2 is a conceptual diagram illustrating a configuration of a user interface according to an exemplary embodiment. The query input device may display the query input window 210 and the results display area 220 shown in fig. 2, although it is understood that one or more other exemplary embodiments are not limited thereto. The query input device may separately display the query input window 210 and the results display area 220.

The query input window 210 may receive a first query component 211 (i.e., a first query input) corresponding to a first query type and a second query component 212 (i.e., a second query input) corresponding to a second query type. The query input window 210 may be displayed differently based on the first query type and the second query type. In addition, the query input window 210 may receive a single query or a query that includes two or more queries of different types.

The results display area 220 may include a list of response results 221 and 222 (i.e., search results). The first response result 211 may include summary information regarding the identified information responsive to the search. For example, the first response result 211 may include a thumbnail of the image document, some of the text included in the document, a link of the searched document, an icon, and the like.

Fig. 3 is a conceptual diagram illustrating a structure for constructing a query input device 300 according to an exemplary embodiment. FIG. 3 illustrates an exemplary environment for implementing exemplary embodiments. Query input device 300 may be implemented as a computing device. For example, the query input device 300 may be implemented in types such as: personal Computers (PCs), smart phones, connected Televisions (TVs), notebook (laptop) computers, Personal Digital Assistants (PDAs), tablet PCs, portable multimedia players, netbooks, workstations, and the like.

The exemplary embodiments may be described in connection with machine-useable instructions, or computer code comprising computer-executable instructions, such as program modules, being executed by a device such as a computer, personal portable terminal, or handheld device. Generally, program modules including routines, programs, objects, components, and data structures, represent code that perform particular tasks or abstract data types. Exemplary embodiments can be implemented in a variety of systems including hand-held devices, consumer electronics devices, general-purpose computers, and special-purpose computing devices. In addition, the exemplary embodiments may be implemented in a distributed computing environment.

Query input device 300 may include a memory 320, at least one processor 330, at least one output device 340, at least one input/output (I/O) port 350, at least one I/O component 360, a power supply 370, and a bus 310 connecting these elements. Bus 310 may include one or more types of bus, such as an address bus, a data bus, or a combination thereof. The functional blocks of fig. 3 are illustrated as lines for simplicity. However, the query input device 300 may be substantially configured with various components. For example, an output device 340 such as a display device can be considered an I/O component 360. Additionally, the processor 330 may include a memory 320.

Query input device 300 may include a variety of computer-readable media. Computer readable media can be any available media that can be accessed by query input device 300 and can include volatile media, nonvolatile media, removable media, and non-removable media. Computer storage media may include volatile, nonvolatile, removable, or non-removable media implemented in any method or technology for storage of computer readable instructions, data structures, program modules, or information such as data. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory, memory technology, CD-ROM, DVD, holographic memory, magnetic cassettes, magnetic tape, magnetic disk, or other magnetic storage devices, may be used to encode desired information, or may include any medium accessible by query input device 300. In an exemplary embodiment, the computer storage medium may be selected from various types of computer storage media. In another embodiment, the computer storage medium may be selected from non-transitory computer storage media.

The memory 320 may include computer storage media of the volatile and/or nonvolatile memory type. The memory 320 may be removable memory, non-removable memory, or a combination thereof. For example, the memory 320 may include a semiconductor memory, a hard disk drive, an optical disk drive, and the like. Query input device 300 may include one or more processors 330 that read data from various entities such as memory 320 or I/O components 360. The output device 340 provides data instructions to the user or another device. For example, the output device 340 may include at least one of a display device, a speaker, a printing component, a vibration motor, a communication device, and so forth.

I/O port 350 allows query input device 300 to be logically connected to other devices, including I/O component 360. For example, the I/O components 360 may include at least one of a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, keyboard, track pad, touch screen, rotatable dials, camera, and handwriting input unit. The handwriting input unit may be a touch screen and may be used to input a handwritten image. The touch screen may include an Electrode Magnetic Reaction (EMR) panel that senses touch by an active stylus (hereinafter, referred to as a pen), although one or more other exemplary embodiments are not limited thereto.

The pen may include a coil, and the coil may generate a magnetic field at a particular point of the EMR plate. The EMR plate can detect the location of the generated magnetic field to detect the location of the pen touching the EMR plate. The EMR plate may detect the position of the pen-generated magnetic field to detect the pen touch position on the touch screen. The controller may detect the pen touch position and thereby receive the handwritten image.

Fig. 4 is a conceptual diagram illustrating a network environment for performing exemplary embodiments. This network environment is only one example of an environment that is applicable to the exemplary embodiments and may include any number of components in terms of board configurations.

The network environment may include a network 410, a query input device 400, and a search engine server 420. Network 410 may include any computer network, such as the Internet, intranets, non-public and public local area networks, non-public and public wide area networks, wireless data or telephony networks, and the like. Query input device 400 is a device that provides a query. According to an exemplary embodiment, the query input apparatus 400 may output the search result as well as the query input.

Search engine server 420 may include any computing device, such as query input device 400. The search engine server 420 may provide at least some operations to provide search services.

Fig. 5 to 7 are exemplary diagrams illustrating a method of receiving a query for a single search according to an exemplary embodiment.

When the search mode is a single search mode (i.e., a single input mode), the query input apparatus according to an exemplary embodiment may display a single query input window 510 through which a query input including one query type is input. The query input device may display at least one search mode selection object for selecting a search mode. Referring to fig. 5, the query input device may display a single input mode selection icon 501 (i.e., a single input mode selection icon) and a multi-modal input mode selection icon 502 (i.e., a multi-modal input mode selection icon). The query input device may display a single query input window 510 when a user input for selecting the single input mode selection icon 501 is received from the user 1. Here, the user input may vary depending on the type of query input device or I/O device. For example, when the query input device is a PC that outputs a Graphical User Interface (GUI), an event of clicking or selecting the single input mode selection icon 501 with a mouse cursor may be a user input of selecting the single input mode selection icon 501. As another example, when the query input device is a handheld device including a touchscreen, a touch of a portion of a human body or an electronic pen on the touchscreen at a location where the single input mode selection icon 501 is displayed may be a user input to select the single input mode selection icon 501.

Additionally, the query input device may display a single query type list 520. The single query type list 520 may be a display item for selecting a query type. The display item may be a UI element displayed on a screen. The single query type list 520 may include multiple objects. Each object included in the single query type list 520 may correspond to a query type, respectively. That is, the single query type list 520 may include queries of at least one of a text type, an image type, a handwriting type, a video type, and an audio type. In FIG. 5, the single query type list 520 is illustrated as a set of button-type objects, but in one or more other exemplary embodiments each object may be replaced in various forms, such as icons, gadgets, and so forth.

The query input device may determine the query type of the query components to be received by the single query input window 510 based on a selection of the single query type list 520. Referring to fig. 6, user 1 may determine the query type included in the single query input window 510 using dragging and dropping one of the objects included in the single query type list 520 to an area where the single query input window 510 is displayed. Drag-and-drop represents one such input scheme: one object is selected in the GUI using a mouse click or touch input, the position is moved, and then the mouse click or touch input ends. For example, when a drag-and-drop event for an object 521 corresponding to sketch information is received by the single query input window 510 as shown in fig. 7, a sketch input window for inputting sketch information into the single query input window 510 may be displayed. Subsequently, when an object corresponding to another query type is selected from among the objects included in the single query type list 520, the sketch input window may be changed to an input window for inputting a query component corresponding to the another query type. Here, the drag and drop may be variously replaced with various input schemes. For example, according to another exemplary embodiment, user 1 may utilize inputs such as those described above with reference to selection of the single input mode selection icon 501 to determine the query type. According to an exemplary embodiment, the area or position of the single query input window 510 may be changed or varied depending on the type of query to be displayed in the single query input window 510 (i.e., the display items corresponding to the query input of the query type). Alternatively, the size of the query components (i.e., display items) displayed in the query input window 810 may be changed or varied depending on the area of the query component input region. For example, as the number of characters input to the text entry window increases, e.g., to more than a predetermined number, the font size of each character displayed in the text entry window may be decreased. For example, according to one or more exemplary embodiments, user 1 may select a query type via selection on the single input mode selection icon 501. According to one or more exemplary embodiments, the area or position of the single query input window 510 may be changed based on the query type of the query to be displayed on the query input window (i.e., the display items corresponding to the query input of the query type). On the other hand, the size of the query components (i.e., display items) of the query to be displayed on the single query input window 510 may be changed based on the query component input area. For example, when inputting characters, if more than a predetermined number of characters are input, the text input window may be enlarged, or the characters in the text input window may be changed to a smaller size.

The controller may be operable to display a query input window 510 corresponding to a type of query selected through the display item 520, obtain search results based on the query input through the query input window 520, and display the search results. For example, when the query type is text, the controller may be operable to display a display item enabling entry of text.

When the selected query type is audio, the controller may be operative to display a display item for inputting at least one of voice information acquired through the microphone and audio data stored in the memory as a query on the query input window 510. For example, an icon for selecting an icon enabling the microphone to operate and audio data stored in the memory may be displayed on the query input window 510. The microphone may receive a voice input to output an electrical signal and may acquire voice information from the electrical signal. The audio data may be stored as a file in the memory.

When the selected query type is an image, the controller may be operative to display a display item for inputting at least one selected from image data acquired by the camera and image data stored in the memory as a query on the query input window 510. For example, an icon for selecting an icon enabling the camera to operate and image data stored in the memory may be displayed on the query input window 510. The camera may acquire image data. The image data may be stored in the memory as a file.

Additionally, the query input device may include a search button 530 for entering commands that allow a search to be performed based on the query entered into the single query input window 510. According to one or more other exemplary embodiments, the search button 530 may not be displayed or may be changed to another form.

Fig. 8 to 18 are exemplary diagrams illustrating a method of receiving a query for a multi-modal search according to an exemplary embodiment.

When the search mode is a multi-modal search mode (i.e., a multi-modal input mode), the query input device according to an exemplary embodiment may display a query input window 810 through which a query including one query type is input. When a plurality of query types are selected through the display item 820, the controller may operate to display the

display items

811 and 812 for receiving a plurality of queries on the query input window 810. For example, referring to FIG. 8, the query input window 810 may include a first query component input area 811 (i.e., a first region) and a second query component input area 812 (i.e., a second region).

The query input device may display at least one search mode selection object for selecting a search mode. Referring to fig. 8, the query input device may display a single input mode selection icon 801 (i.e., a single input mode selection icon) and a multi-modal input mode selection icon 802 (i.e., a multi-modal input mode selection icon). Upon receiving a user input for selecting the multimodal input mode selection icon 801 from the user 1, the query input device may display a query input window 810. A query input window that can receive multiple query components (i.e., query inputs) having multiple query types can be referred to as a multimodal query input window. Here, the user input may vary depending on the type of query input device or I/O device. For example, when the query input device is a PC outputting a GUI, the event of clicking or selecting the multimodal input mode selection icon 801 with a mouse cursor may be a user input of selecting the multimodal input mode selection icon 801. As another example, when the query input device is a handheld device that includes a touchscreen, a touch of a portion of a human body or an electronic pen (e.g., a stylus) on the touchscreen at a location where the multimodal input mode selection icon 801 is displayed may be a user input that selects the multimodal input mode selection icon 801.

In addition, the query input device may display a single query type list 820. The single query type list 820 may include multiple objects. Each object included in the single query type list 820 may correspond to a query type, respectively. In FIG. 8, the single query type list 820 is illustrated as a simple button type, but in one or more other exemplary embodiments each object may be replaced in various forms, such as icons, gadgets, and so forth.

The query input device can determine the query type included in the multimodal query input window 810 based on the single query type list 820. Referring to FIG. 9, user 1 can determine the query type included in the query input window 810 by dragging and dropping one of the objects included in the single query type list 820 to an area where the multimodal query input window 810 is displayed. Drag-and-drop represents one such input scheme: an object is selected in the GUI using a mouse click or touch input, the position is moved, and then the mouse click or touch input is ended. For example, when a drag-and-drop event for the object 821 corresponding to text is input to the first query component input area 811 of the query input window 810 as shown in fig. 9, a text input window for inputting text to the first query component input area 811 may be displayed as shown in fig. 10. The text entry window may include a text box that displays entered text. In addition, referring to fig. 10, when a drag-and-drop event for an object 822 corresponding to an image is input to the second query component input area 812, a user interface for inputting a query component of which query type is an image to the second query component input area 812 may be displayed as shown in fig. 11. In this case, drag and drop can be variously replaced with various input schemes. For example, according to one or more exemplary embodiments, user 1 may determine the query type via selection of the multimodal input mode selection icon 801. In addition, the area or position of the first query component input area 811 or the second query component input area 812 may be changed or varied depending on the type of query (i.e., display items corresponding to query inputs of the query type) to be displayed in the first query component input area 811 or the second query component input area 812. Alternatively, the size of the query components displayed in the query input window 810 may be changed or varied depending on the area of the query component input region. For example, as the number of characters input to the text entry window increases, e.g., to more than a predetermined number, the font size of each character displayed in the text entry window may be decreased.

Additionally, in fig. 8-11, query input window 810 is shown as receiving a query that includes at least two query types (i.e., query inputs of at least two query types). It is understood that the number of query types included in the query received through the query input window 810 may be greater than, equal to, or less than two in the exemplary embodiment. User 1 may select at least one from among the objects included in the single query type list 820. For example, one of the objects included in the single query type list 820 may be clicked on with a mouse or a mouse cursor. Alternatively, a touch may be made to one of the objects displayed on the touch screen. However, it is to be understood that one or more other exemplary embodiments are not limited thereto.

Referring to fig. 12, an object 823 corresponding to audio may be selected by an input received from the user 1. When at least one object is selected from among the objects included in the single query type list 820, the query input device may add a third query contribution input area 813 (i.e., a third area) to the query input window 810. The query input device may add a third query contribution input region 813 to the query input window 810 without changing the overall area of the query input window 810. For example, referring to fig. 13, the query input device may display a third query contribution input area 813 in a portion of the second query contribution input area 812. However, it is to be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, the query input device may display the third query contribution input area 813 on an additional area of the query input window 810 that is enlarged as compared to the situation before the third query contribution input area 813 was displayed. When the object 823 corresponding to audio is selected as shown in fig. 12, a user interface for inputting a query component whose query type is audio to the third query component input area 813 may be displayed as shown in fig. 13.

In addition, the area of each query component input region included in the query input window 810 may be changed or varied. For example, when it is difficult to display text input to the first query component input area 811 in the entirety of the first query component input area 811, the area of the first query component input area 811 may increase. As another example, as shown in FIG. 14, the boundaries 840 between query component input regions may be moved based on user input. Here, the user input may be an input to drag boundary 840, although it is understood that one or more other exemplary embodiments are not so limited.

In addition, the user interface displayed in the query component input area may be changed to a user interface for inputting a query component corresponding to another query type. Referring to fig. 15, an object 824 corresponding to sketch information among objects included in the single query type list 820 may be received, and an event generated by the user 1 may be received as a user input selecting the second query component input area 812. Here, the user input may be a drag-and-drop event starting from the object 824 displayed in the single query type list 820 and ending in the second query component input area 812, but is not limited thereto. In this case, as shown in fig. 16, a sketch input window 814 for inputting sketch information may be displayed in the second query component input area 812 of fig. 15.

In addition, the size of the query input window 810 may be changed according to user input. The query input window 810 may be enlarged or reduced according to user input. As shown in FIG. 17, when user input is received, a query input window 810 may be displayed in an enlarged or reduced area 850. Here, the user input may be an input dragging an edge of the query input window 810, although it is understood that one or more other exemplary embodiments are not limited thereto.

In addition, the location at which the query input window 810 is displayed may be moved according to user input. As shown in FIG. 18, when user input is received, a query input window 810 may be displayed in a moved area 860. Here, the user input may be an input dragging a partial location of the query input window 810, although it is understood that one or more other exemplary embodiments are not limited thereto.

Fig. 19 and 20 are exemplary diagrams illustrating a method of receiving a query for a multi-modal search according to another exemplary embodiment. Fig. 19 is an exemplary diagram illustrating a screen obtained by performing a search based on a single search mode. The query input window may include a first query component input area 1910 for inputting a query component corresponding to a first query type. Query components are received through the first query component input area 1910 and when a search button 1930 is selected, a search may be performed based on a query that includes the received query components. Here, the search may be performed using a search engine included in the query input device. Alternatively, the query input device may send a query to the search engine server and receive a response to the query, so that a search may be performed.

The search results may be displayed in a results display area 1920 as a result of the search. Referring to fig. 19, first search results 1921 and second search results 1922 may be displayed in a results display area 1920.

Subsequently, when at least one search result is selected from among the displayed search results based on user input, a query type list 1940 may be displayed. Referring to FIG. 19, when the second search result 1922 is selected, a list of query types 1940 may be displayed adjacent to the second search result 1922. In FIG. 19, the query type list 1940 is illustrated as a set of objects configured with buttons, but in one or more other exemplary embodiments each object may be replaced in various forms, such as icons, widgets, and so forth. In this case, a multi-modal search may be performed based on the second search results 1922, and the query components may be input into the second query component input area 1920. However, the present exemplary embodiment is not limited thereto. For example, in accordance with one or more other exemplary embodiments, a multi-modal search may be performed based on the original query components input to the first query component input area 1910 and the query components input to the second query component input area 1920.

When a query type is selected from the query type list 1940 of fig. 19, the query input device may switch the search mode from the single search mode to the multi-modal search mode. Referring to FIG. 20, a second query component input area 1915 may further be displayed for inputting query components corresponding to a second query type. Subsequently, when the search button 1930 is selected, a multi-modal search may be performed based on the query components input to the first query component input area 1910 and the query components input to the second query component input area 1920. In this case, a multimodal search may be performed based on the second search results 1922 and the query components input to the second query component input area 1920, although it is understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, a multi-modal search may be performed based on the original query components input to the first query component input area 1910 and the query components input to the second query component input area 1920.

Search results

1923 and 1924 may be displayed in the results display area as results of a multi-modal search.

Fig. 21 and 22 are exemplary diagrams illustrating a method of generating a query input window according to an exemplary embodiment. According to an exemplary embodiment, a search mode is selected based on the search mode selection object 2100, and then, as shown in fig. 21, an arbitrary region 2110 may be selected or generated based on a user input received from the user 1. Here, the user input may be a drag of any location, although it is understood that one or more other exemplary embodiments are not so limited. When the arbitrary area 2110 is selected, as shown in fig. 22, a query input window may be displayed in the arbitrary area 2110 according to the selected search mode. For example, when the selected search mode is a single search mode, the query input window may be a single query input window, and when the selected search mode is a multi-modal search mode, the query input window may be a multi-modal query input window. In addition, a query type list 2220 and a search button 2230 may be displayed.

Fig. 23 is an exemplary diagram illustrating a method of receiving a query for a multi-modal search according to another exemplary embodiment. In the exemplary embodiment of fig. 23, the query input device may include an input device (e.g., an input unit) that simultaneously receives input for a plurality of points. For example, the query input device may include a touch screen. In displaying the single query type list 2320, a plurality of objects included in the single query type list 2320 may be selected. Referring to fig. 23, a first object 2321 (where the query type is text) and a second object 2322 (where the query type is an image) may be selected. For example, two fingers of the user may touch

objects

2321 and 2322 displayed on the touch screen, respectively. A plurality of objects are selected, and when the selection position of each object is moved to a point, a query input window 2310 for receiving query components having a plurality of query types corresponding to the plurality of objects may be displayed. For example, referring to fig. 23, the following inputs may be received: here, two

fingers touching objects

2321 and 2322 displayed on the touch screen move to a point 2340 in a state of touching the touch screen and then disengage from the touch screen. That is, input may be received to drag and drop multiple objects to one point 2340 at the same time. A query input window 2310 including a text input window 2311 for inputting text and an image upload window 2312 for uploading images may be displayed. Here, one point 2340 may not exactly or exactly correspond to the same point. For example, when two points are within an arbitrary distance of each other (e.g., within a predetermined distance), a plurality of objects may be considered to be located on one point.

FIG. 24 is an exemplary diagram illustrating a multimodal query type list 2410 according to another exemplary embodiment. According to the present exemplary embodiment, the query input device may display a single query type list when the search mode is the single search mode, and may display a multi-modal query type list when the search mode is the multi-modal search mode. The objects included in the list of multimodal query types can represent a combination of multiple query types. Referring to fig. 24, when a search mode is set to a multimodal search mode using the search mode selection object 2400, a multimodal query type list 2410 representing a list in which a combination type of two or more query types is combined may be displayed.

Fig. 25A to 25E are exemplary diagrams illustrating a method of receiving a query for a multi-modal search according to another exemplary embodiment.

According to the present exemplary embodiment, a portion of the search results may be utilized to receive a query for a multi-modal search. Referring to fig. 25A, the query input device 2500 may display an image 2520 as a search result obtained by performing a search based on a query input to the query input window 2510.

Referring to fig. 25B, a partial region 2501 of the image 2520 can be selected. Partial region 2501 may be selected manually based on user input or may be selected automatically based on analysis of search results, such as image recognition. For example, when the user 1 draws an annular curve by dragging a mouse cursor or using a touch input, the query input device 2500 may determine the inside of the annular curve as the partial region 2501 selected from the image 2520.

When partial region 2501 is selected from the search results, a list of query types may be displayed. According to the present exemplary embodiment, referring to fig. 25C, a query type list may be displayed in the form of a floating icon or a floating widget near the selected partial region 2501. A floating icon or floating widget may be displayed on the content and be a location-movable object.

For the displayed list of query types 2520, user input for selecting a query type may be received from user 1. When a query type is selected, a query input window corresponding to the displayed query type may be displayed. For example, a partial region 2501 including a bag displayed in the image 2520 shown in fig. 25C may be selected. Subsequently, when an icon 2511 corresponding to a voice search among a plurality of query types is selected from among a plurality of displayed floating icons 2521, a query input window 2510 for inputting a query component (in which the query type is voice) may be displayed. A speech-type query component may be referred to as an audio query.

According to an exemplary embodiment, the query input window 2510 for inputting a query component (where the query type is speech) may include a speech (i.e., audio) recording icon. When the user 1 selects the voice recording icon, the query input device 2500 may operate a microphone and execute an application for acquiring voice information. Referring to fig. 25D, the user 1 may input voice into the query input device 2500 while an application for acquiring voice information is being executed.

When a query component is input through the query input window 2510, the query input device 2500 may receive a portion of the selected search result (e.g., corresponding to the partial region 2501 selected from the image 2520) and information input through the query input window 2510 as the query component. Referring to fig. 25E, a query input window 2510-1 including an image input region 2511-1 in which an image in which a bag is displayed and a voice input region 2512-1 in which at least one of a waveform of input voice information, a predetermined waveform, and text converted from voice information is displayed may be displayed as display items corresponding to a query. Here, the waveform of the input voice information may be a waveform corresponding to at least one of a predetermined portion of the voice information (e.g., the beginning of the voice information), a predetermined length of the voice information, a predetermined audio characteristic or characteristic of the voice information, and the like. As shown in fig. 25D, when the voice information "price" is input, the query input device 2500 may perform a multi-modal search based on a query including an image containing a bag and the voice "price". As shown in fig. 25E, the query input device 2500 may display information about the price of a bag included in the image as a search result obtained by performing a multi-modal search in the search result display area 2520-1.

Fig. 26 is a flowchart illustrating a process of displaying a query input window according to an exemplary embodiment. Fig. 27 and 28 are exemplary diagrams illustrating a method of displaying a query input window according to an exemplary embodiment. According to an exemplary embodiment, query components may be entered in a state where a query type is not selected prior to entering a query.

In operation S2610, a query input device may receive a query component through a query input window. The method of receiving a query contribution may be implemented in various ways. Referring to fig. 27, the query input device may display an indicator 2701 indicating the set search mode, a query input window 2710, and a search button 2720. The query input window 2710 of fig. 27 includes a user interface 2711 for inputting a query component whose query type is text. Here, the query input device may receive an input from the user 1 through which the file icon 2730 (included in the file search window 2740) is dragged and dropped to the query input window in order to receive the query component. However, it is to be understood that one or more other exemplary embodiments are not limited to a drag-and-drop input method. For example, according to another exemplary embodiment, a file may be selected by a clicking operation (e.g., double-clicking). In addition, the file search window 2740 is not limited to the form shown in fig. 27. For example, according to another exemplary embodiment, the file search window 2740 of fig. 27 may be displayed by dividing the screen of the query input device or may be displayed by changing the screen of the query input device to the file search screen.

In operation S2620, the query input device may detect a query type of the received query component. The method of detecting a query type may be implemented in various ways. For example, when the received query component is a file, the query input device may detect the query type of the query component from the extension of the file. In this case, the query input device may determine the query type of the query component as an image when the extension of the file is jpg, gif, or bmp, and may determine the query type of the query component as a video when the extension of the file is avi, mp4, or wmv. Alternatively, when an application is used to receive a query component, the query input device may detect the query type of the query component based on the type of the application. For example, when receiving information acquired with a camera application as a query component, the query input device may determine a query type of the query component as an image, and when receiving the query component with a voice recording application, the query input device may determine the query type of the received query component as voice information.

In operation S2630, the query input device may change the query input window to include a user interface through which a query component corresponding to the detected query type is received (and through which display items corresponding to the previously received query component are displayed). In addition, display items corresponding to the received query components may be displayed. Referring to fig. 28, when receiving an image file like in fig. 27, the query input device may change a query input window 2710 configured with a text input box as a user interface 2711 for inputting text to another query input window 2710 configured with an image upload tool for inputting an image 2712.

FIG. 29 is a flowchart illustrating a process of receiving a query according to an exemplary embodiment. Fig. 30 to 32 are exemplary diagrams illustrating a method of receiving a query according to an exemplary embodiment. According to an exemplary embodiment, query components may be entered without first selecting a query type.

In operation S2910, the query input device may receive a query component through the query input window. The method of receiving a query contribution may be implemented in various ways. Referring to fig. 30, the query input device may display an indicator 2702 indicating the set search mode, a query input window 2710, and a search button 2720. In fig. 30, a query input window 2710 is shown as including a user interface 2711 for entering a query component whose query type is text. Here, the query input device may receive an input from the user 1 through which the file icon 2730 (included in the file search window 2740) is dragged and dropped to the query input window in order to receive the query component. However, it is to be understood that one or more other exemplary embodiments are not limited to a drag-and-drop input method. For example, according to another exemplary embodiment, a file may be selected by a clicking operation (e.g., double-clicking). In addition, the file search window 2740 is not limited to the form shown in fig. 30. For example, according to another exemplary embodiment, the file search window 2740 of fig. 30 may be displayed by dividing the screen of the query input device or may be displayed by changing the screen of the query input device to the file search screen.

In operation S2920, the query input device may detect a query type of the received query component. The method of detecting a query type may be implemented in various ways. For example, when the received query component is a file, the query input device may detect the query type of the query component from the extension of the file. Alternatively, when an application is used to receive a query component, the query input device may detect the query type of the query component based on the type of the application.

In operation S2930, the query input device may add a query type to be used for performing the multi-modal search according to the detected query type. Thus, the query input device can add an area to the query input window that receives a query component corresponding to the added query type (and displays display items corresponding to previously received query components). Referring to fig. 31, as a query type is added, an area where a query input window 2710 is displayed may be enlarged. In addition, a user interface 2712 for inputting an image may be displayed in an area in which the query input window is enlarged. Information 2750 about the received query components, such as a preview, icon, or filename, may be displayed in the added area.

In accordance with one or more exemplary embodiments, the query input window 2710 can be enlarged to include a user interface 2712 corresponding to the type of query added. However, the present exemplary embodiment is not limited thereto. In addition, according to one or more exemplary embodiments, the form of the query input window 2710 displayed by the query input device may not be changed. In addition, the inner region of the query input window 2710 may not be divided. In this case, a plurality of query components respectively corresponding to different query types input through the query input window 2710 may be displayed together in the query input window 2710.

According to another exemplary embodiment, the query input device may display a query input window. Here, when the search mode is the multi-modal input mode, the query input window may include an area displaying the received query component (i.e., displaying the display item corresponding to the previously received query component) and an area for receiving the query component. According to an exemplary embodiment, the query input device may perform an operation of receiving the query component when the region for receiving the query component is selected. For example, in order to receive a query component whose query type is voice information, the query input device may stand by in a state in which a microphone is operated. The voice information may be information included in voice data. In operation S2910, the query input device may receive a plurality of query components corresponding to a plurality of query types sequentially or simultaneously through a region of a query input window for receiving the query components. For example, when a user writes out a search in red while tracing a bicycle, the query input device may receive, as query components, sketch information indicating the bicycle traced by the user and voice information including the keyword "red".

In operation S2920, the query input device may detect a query type of the received query component. For example, when the received query component is a file, the query input device may detect the query type of the query component from the extension of the file. Alternatively, when an application is used to receive a query component, the query input device may detect the query type of the query component based on the type of the application. As another example, the query input device may detect that the query type of the query component is an image when taking a picture with a camera. Alternatively, text may be detected as a query type when Optical Character Recognition (OCR) is used to identify characters included in a picture. When a query type of the received query component is detected, the query input device may display the received query component (or a display item corresponding to the received query component) in an area where the received query component is displayed based on the detected query type. For example, when the query type detected from the first query component is a video, the query input device may display a preview of the video in the area where the first query component is displayed, and when the query type detected from the second query component is text, the query input device may display keywords in the area where the second query component is displayed. Alternatively, when the query type detected from the query component is voice information, the query input device may display at least one of a voice waveform included in the voice information or corresponding to a predetermined waveform and text converted from the voice information in an area where the query component is displayed. The query input device may repeatedly perform operations S2910 and S2920 to receive a plurality of query components, and may display the received query components to enable a user to view the query components.

When a query type is detected in operation S2920, the query input device may add a query type to be used for the query in operation S2930. When the query input device receives an input corresponding to a search command, the query input device may generate a query that includes the added query type. The query input device may perform a search based on a query that includes the received query components and the detected query type. The query input device may display one or more search results as a result of the search.

When the query type is changed, the method of receiving the query components may be changed. In addition, various methods of receiving query components may be provided for one query type. Thus, the user interface provided to the user for entering query components may be changed or varied depending on the type of query. FIG. 33 is a flowchart illustrating a process of receiving a query according to an exemplary embodiment.

Referring to fig. 33, in operation S3310, the query input device may display a query input tool in a query input window. Here, the query input tool represents a user interface provided to a user for inputting query components. In operation S3310, the query input device may display a query input tool according to a query type included in a query input through the query input window. Examples of query input tools displayed according to query type will be described in detail below with reference to fig. 34-40.

In operation S3320, the query input device may select at least one from among a plurality of query input tools displayed in a query input window. Specifically, in operation S3320, the query input device may receive a user input and select a query input tool according to the received user input. The type or form of user input may vary. For example, the query input tool may be selected according to an operation in which a portion of a human body, a stylus, or the like touches the query input tool displayed on the touch screen or clicks a mouse cursor of the displayed query input tool.

In operation S3330, the query input device may determine whether to execute an application for receiving the query component based on the selected query input tool. For example, when the selected query input tool is a text box, the query input device may determine that the query components may be received directly through the text box without executing a separate application. That is, when the separate application is not executed, the query input device may receive the query component through the query input window in operation S3340.

When a separate application for receiving a query component is to be executed, the query input device may execute an application corresponding to the query input tool in operation S3335. The application corresponding to the query input tool may be predetermined or may be selected by the user from a list of applications. Accordingly, the query input device may receive the query component using the executed application in operation S3345.

Fig. 34 to 40 are exemplary diagrams illustrating a method of receiving a query component using a query input tool displayed in a query input window based on a query type included in a query. Fig. 34 to 40 are merely examples for describing one or more exemplary embodiments, and it is to be understood that one or more other exemplary embodiments are not limited thereto.

Fig. 34 is an exemplary diagram illustrating a method of receiving a query component in which the query type is text according to an exemplary embodiment.

When a text mode is included in the query, the query input window may include a text box 3410, as shown in fig. 34. When the text box 3410 is selected, a cursor 3411 indicating a position where text is to be input may be displayed in the text box 3410. Text may be entered via a keyboard, keypad, writing tablet that receives handwritten text, and the like. In addition, the query input device may receive copied text from the keyboard or clipboard through text box 3410. Keywords extracted from the entered text may be used as query components in the search.

According to another exemplary embodiment, the query input device may utilize OCR operations to acquire text from an image (e.g., an image acquired by operating a camera). Additionally, while the query input tool for receiving text input was provided above as text box 3410 including cursor 3411, it is to be understood that one or more other exemplary embodiments are not so limited. For example, according to another exemplary embodiment, a query input tool for receiving text input may include a writing pad to receive handwritten text that is included in a query and subsequently converted, e.g., by a search engine via an OCR operation, or by a query input device via an OCR operation into text.

Fig. 35 to 38 are exemplary diagrams illustrating a method of receiving a query component in which a query type is an image according to an exemplary embodiment. In fig. 35-38, query input device 3500 is described as a handheld device including a camera, although it is understood that one or more other exemplary embodiments are not limited thereto.

When the query received through the query input window includes an image, the query input device 3500 may display a query input window 3510 including one or more tools for receiving the image. Each of the one or more tools for receiving images may include at least one of an image upload icon 3511, a photography icon 3512, and an image address input box 3513.

When user 1 selects image upload icon 3511, query input device 3500 may operate to select an image file. For example, referring to fig. 37, query input device 3500 may execute a gallery application that displays a preview list of image files. When the user 1 selects at least one image 3520 from the displayed preview list using the gallery application, the query input device 3500 may receive the selected image 3520 as a query component.

Referring to fig. 37, when selecting an image 3520 using a gallery application, the query input device 3500 may end, close, or hide the gallery application and display the selected image 3520 in the query input window 3510. Alternatively, according to another exemplary embodiment, query input device 3500 may segment its display area. In this case, the query input device 3500 may display an execution screen of the gallery application in one of the divided areas, thereby enabling the user 1 to select an image.

When user 1 selects the photography icon 3512, the query input device 3500 may execute an application 3530 that operates a camera to take pictures, as shown in fig. 38. When an image is generated using a camera, the query input device 3500 may receive the generated image as a query component, and as shown in fig. 37, the image may be displayed in the query input window 3510.

According to another exemplary embodiment, the user 1 may input an address where an image is located using the image address input window 3513. The image address may be an address indicating a location of the image, such as a URL address, although it is understood that one or more other exemplary embodiments are not limited thereto.

In FIG. 37, images received as query components are illustrated as being displayed in the query input window 3510, although it is understood that the location at which images received as query components are displayed may vary in one or more other exemplary embodiments.

In addition, the method of receiving a query (where the query type of the query component is a video) may be implemented similarly to the method of receiving an image as a query component as described above.

When an image or video is received as a query constituent, keywords may be retrieved from the image or video using image recognition or OCR operations. The search may be performed using the acquired keyword. Alternatively, the query input device may compare the images themselves to an index to search for similar images.

Fig. 39 is an exemplary diagram illustrating a method of receiving a query component in which a query type is sketch information according to an exemplary embodiment.

When the query type included in the query is sketch information, the query input window 3910 may include a sketch input tool 3911 for inputting sketch information. The sketch input tool 3911 according to an exemplary embodiment may include one or more icons for selecting at least one of a pen mode, a brush mode, a pen mode, a color or thickness of a line, and the like. The user 1 may set a sketch input mode using the sketch input tool 3911 and input sketch information into an area that receives a query component whose query type is sketch information using a touch input, a mouse input, a track pad input, a gesture input, or the like.

When sketch information is received, the received sketch information may be displayed in the query input window 3910. The received sketch information may be converted into keywords, and the keywords obtained from the sketch information may be used for searching. Alternatively, the search may be performed in the form of sketch information itself.

Fig. 40 is an exemplary diagram illustrating a method of receiving a query component in which a query type is audio information according to an exemplary embodiment.

When the query type included in the query is audio information, the query input device may display a query input window 4010 including a tool for receiving the audio information. The tool for receiving audio information may include, for example, at least one of a sound file upload button 4011 and an audio recording icon 4012.

When the sound file upload button 4011 is selected, the query input device may display a file selection window 4021 for selecting a sound file. The user can select a voice file (i.e., an audio file) to be input as a query component using the file selection window 4021.

Alternatively, when the audio recording icon 4012 is selected, the query input device may operate a microphone (an image of which is included in the query input window 4010) and execute the application 4022 that records audio (i.e., voice information). The query input device may receive the acquired voice information as a query component using the executed application 4022.

When the voice information is input, the query input device may display a waveform, a voice spectrum, or a file name of the voice information input to the query input window 4010.

The voice information received as the query component may be used for music search by comparing the waveform itself of the voice with the index, or a keyword obtained by conversion using voice recognition may be used for search.

FIG. 41 is an exemplary diagram illustrating a method of receiving a query including a plurality of query components having a plurality of query types and displaying search results according to an exemplary embodiment.

When the query type included in the query is text and an image, a query input window displayed in the query input device 4100 may include a first region 4111 for inputting text and a second region 4112 for inputting an image. As shown in fig. 41 (a), when the text "flower" and an image including a dog shape are received as a query component, the text "flower" may be displayed as a first display item in an area 4111, and the received image may be displayed as a second display item in an area 4112.

Here, the user may manually select the first region 4111 to place the first region 4111 in a ready state to receive input of text, and may manually select the second region 4112 to place the second region 4112 in a ready state to receive input of an image. However, it is to be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, after or in response to a user entering a first query component (e.g., text) in the first region, the second region 4112 may automatically enter a ready state to receive input of a second query component (e.g., image). Here, the controller of the query input device 4100 may determine to automatically switch from a first ready state (i.e., a first state) in which the first region 4111 may receive the first query component to a second ready state (i.e., a second state) in which the second region 4112 may receive the second query component. For example, if the second query component corresponds to an audio or voice information query type, the controller may perform controls to automatically switch the query input device 4100 to a second ready state in which the microphone is automatically activated or operated to receive the second query component. In addition, if the second query component corresponds to an image query type, the controller may perform control to automatically switch the query input device 4100 to a second ready state in which the camera is automatically activated or operated to receive the second query component or in which an interface to select an image is displayed to receive the second query component.

When the ready state is automatically switched as described above, the controller may control to output an indicator of the automatic switching and/or the second ready state. For example, the indicator may include at least one of: an audio indicator or output (e.g., a predetermined notification sound), a visual indicator or output (e.g., a predetermined icon, a predetermined symbol, a predetermined image, etc.), an auxiliary device output (e.g., a flashing LED or a predetermined color LED on the query input device 4100), a vibration output, and so forth. In this case, the visual indicator may be displayed in the query input window or outside the query input window in various exemplary embodiments.

In addition, the controller may determine to automatically switch from the first ready state to the second ready state based on determining completion of the input of the first query component. For example, if the first query component is an image, the controller may determine to automatically switch from the first ready state to the second ready state in response to an image captured by a camera of the query input device 4100 or in response to a user selection of an image. Additionally, if the first query component is text or a sketch, the controller may determine to automatically switch from the first ready state to the second ready state in response to a predetermined period of time (e.g., two seconds after the last text character is entered into the first region 4111) elapsing from the last user input to the first region 4111.

The controller may also control changing the display of the query input window, the display of the second region 4112, or the display of the graphical user interface when switching to the second ready state. Here, the change in display may be based on the query type corresponding to the second query component or second region 4112. According to another exemplary embodiment, the controller may control the second region 4112 or the user interface to be displayed to receive an input of a second query component in response to the switching to the second ready state.

When the user 1 selects the search button 4120 in a state in which the received text and the received image are displayed as illustrated in (b) of fig. 41, the query input device 4100 may display search results obtained by performing the multi-modal search. In fig. 41, an image including both flowers and dogs is displayed as a search result.

Fig. 42 is a flowchart illustrating a process of performing a search according to an exemplary embodiment.

The query input apparatus according to an exemplary embodiment may receive a query in operation S4210, and determine a priority of a query component included in the received query in operation S4220. In operation S4220, the query input device may determine a priority of the query component based on a query type of the query component. For example, when text, images, and voice information are included in a query, the priority of the query components may be determined in the order of text, voice information, and images.

The priority of the query components may be determined in various ways. For example, a user may set priorities for multiple query types. According to an exemplary embodiment, when the priority of the query type is set, the priority may be determined based on the query type of the received query component. According to another exemplary embodiment, the priority of the query components may be determined based on the order in which the query components are received through the query input window. However, this is merely an example used to describe an exemplary embodiment and it is to be understood that one or more other exemplary embodiments are not limited thereto.

In operation S4230, a search may be performed based on the determined priority. For example, a first search may be performed based on text included in the query, and then a second search may be performed on the results of the first search performed based on the text, using speech information that is a query component having a lower priority than the text.

The priority of the query components may be determined by the query input device. Alternatively, when a search is performed using a search engine server that includes a search engine, the priority of the query components may be determined by the search engine server.

Fig. 43 is an exemplary diagram illustrating results obtained by performing a multi-modal search according to an exemplary embodiment.

According to an exemplary embodiment, the query input device may display a query input window 4201. The query input window 4201 may include at least one query component input area displaying a received query component (i.e., a display item corresponding to the received query component). According to an exemplary embodiment, the query input device may display the received query components based on a priority of the query type of the received query components. Referring to fig. 43, when the priority of the image query type is higher than the priority of the text query type, an image input area 4202 may be displayed above a text input area 4203.

A query component is input using the query input window 4201, and then, when the user selects the search button 4204, the results of a search performed using the query component may be displayed in the search results display area 4205. Here, among the plurality of query components, the query component having the highest priority may be determined as the main query component, and the query component having the lowest priority may be determined as the sub-query component. Among the search results based on the primary query component, information selected by the secondary query component may be displayed in a search result area. For example, referring to fig. 43, among pieces of information on a smartphone included in an image, advertisement information including product information of the smartphone, a blog, and an image may be displayed in a search result area 4205.

Fig. 44 is a flowchart illustrating a process of displaying search results according to another exemplary embodiment.

In operation S4310, a plurality of search results may be obtained (i.e., determined or obtained) based on the query received through the query input window. Here, the acquired plurality of search results may be prioritized in operation S4320. For example, the obtained plurality of search results may be prioritized based on the degree to which the query is matched. As another example, the priority of the obtained plurality of search results may be determined based on the time at which information including the respective search results is generated.

In operation S4330, the prioritized search results may be displayed in the query input window based on their priorities. Here, the search results may be displayed in another device than the query input device. For example, the search results may be displayed in another device included in a home network connected to the query input device.

Fig. 45 is an exemplary diagram illustrating priorities of search results according to another exemplary embodiment. According to the present exemplary embodiment, the search results may be prioritized based on the type of search result. The priority of the search results may be set by the user or may be determined based on the user's feedback or usage history. For example, the user may directly set the advertisement search result type to a low priority when the user has no interest in the advertisement, or the query input device may set the advertisement search result type to a relatively low priority when the number of times the advertisement information is selected is relatively small (or below a predetermined value). For example, referring to fig. 45, the priority based on the result type may be determined in the order of images, blogs, news, web documents, and advertisements. Here, prioritization of search results by a query input device is described. However, it is to be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, when a search is performed using an external search engine server, the priority of the search results may be determined by the external search engine server.

FIG. 46 is an exemplary diagram illustrating results obtained by performing a multi-modal search according to another exemplary embodiment. In the exemplary embodiment of fig. 46, the same query as in the exemplary embodiment of fig. 43 is input. A query input window 4201 including an image input area 4202 and a text input area 4203 and a search button 4204 may be displayed.

In addition, the exemplary embodiment of fig. 46 shows a case where the priorities of the search results are set as shown in fig. 45. Even when the same query is input, the search results may be displayed differently depending on the priorities of the search results. That is, the search results may be displayed in the order of images, blogs, and news in the result display area 4205-1 based on the priorities of the search results.

Fig. 47 is an exemplary diagram illustrating a method of receiving a query using a plurality of devices according to an exemplary embodiment. According to the present exemplary embodiment, a query component included in a query may be input using a plurality of devices. In fig. 47, the query input device 4400 is a connected TV, although it is understood that in one or more other exemplary embodiments the query input device 4400 is not limited to the form shown in fig. 47, but may be implemented as a PC, smart phone, smart device, laptop, portable multimedia player, tablet, netbook, workstation, and so forth.

When the query types included in the query are sketch information, voice information, and images, the query input device 4400 may display a query input window 4410 for inputting sketch information, voice information, and images.

The query input device 4400 may request voice information 4411 from the smart watch 4401 capable of communicating with the query input window 4401. The smart watch 4401 may represent an embedded system watch equipped with various operations in addition to those of a general clock. For example, the smart watch 4401 may perform computing operations, translating operations, recording operations, communicating operations, and the like. The smart watch 4401 receiving a request for voice information 4411 from the query input device 4400 may operate a microphone included in the smart watch 4401 to generate voice information 4411, and transmit the generated voice information 4411 to the query input device 4400. The query input device 4400 may receive the voice information 4411 transmitted from the smart watch 4401 as a query component. The smart watch 4401 may communicate with the query input device 4400 using wired communications or wireless communications such as bluetooth, Wi-Fi direct, Near Field Communication (NFC), infrared data association (IrDA), Radio Frequency (RF) communications, wireless Local Area Network (LAN), etc.

In addition, the query input device 4400 can request images 4412 from the smart glasses 4402 capable of communicating with the query input device 4400. The smart glasses 4402 represent wearable devices equipped with a head-mounted display (HMD). The smart glasses 4402 may perform computing operations, translation operations, recording operations, communication operations, and the like. The smart glasses 4402 receiving a request for an image 4412 from the query input device 4400 can generate the image 4412 captured by a camera included in the smart glasses 4402. The smart glasses 4402 may send the generated image 4412 to the query input device 4400. The inquiry input device 4400 may receive the image 4412 transmitted from the smart glasses 4402 as an inquiry component. The smart glasses 4402 may communicate with the inquiry input device 4400 using wired communications or wireless communications such as bluetooth, Wi-Fi direct, NFC, IrDA, RF communications, wireless LAN, etc.

The smart glasses 4402 may include a camera for tracking the eyes of the user. When the user views a certain portion of the entire screen of the TV, the smart glasses 4402 may utilize a camera for tracking the user's eyes to determine the area that the user is currently viewing and transmit an image of the area to the query input device 4400.

Fig. 47 also illustrates an exemplary diagram of a method in which a smart lens (e.g., smart glasses) acquires an image of an area viewed by a user from the entire screen of a TV according to an exemplary embodiment. Referring to fig. 47, the smart glasses 4402 may acquire an image at a location viewed by a user. Fig. 47 shows an exemplary embodiment when the user watches the bicycle displayed on the TV screen.

The pupil tracking camera may track the pupil direction of the user to determine the direction in which the user's eyes are looking. A processor included in the lens (e.g., glasses) may receive information from the pupil tracking camera indicating a viewing direction of the user and adjust a direction of the front camera based on the received information. Thus, the front camera may be synchronized with the viewing direction of the user. That is, the direction of the front camera and the direction in which the user's eyes look are the same. When the direction of the front camera is adjusted, the front camera may output an image of the adjusted direction to the processor.

The front camera may acquire images located in the viewing direction tracked by the pupil tracking camera. In other words, the front camera may capture an image in the same direction as the viewing direction of the user. In fig. 47, a case where the front camera is directed to a bicycle in the TV screen is illustrated.

The smart glasses 4402 can take a bicycle image on the TV screen and send the bicycle image to the query input device 4400. The query input device 4400 may receive the bicycle images transmitted from the smart glasses 4402 and receive a query including the bicycle images, thereby performing a search.

In addition, the query input device 4400 may receive sketch information directly using a query input tool output to the query input device 4400. The query input device 4400 that receives the voice information, the image, and the sketch information may perform a search based on a query that includes the voice information, the image, and the sketch information.

The smart watch 4401 and the smart glasses 4402 of fig. 47 may be replaced with other devices capable of communicating with the query input device 4400 in one or more other exemplary embodiments.

Fig. 48 is an exemplary diagram illustrating a method of receiving a query component using a plurality of devices according to another exemplary embodiment.

According to the present exemplary embodiment, the query input device 4800 may receive a query component to be added or included in a query using a device connected to the query input device 4800 through a network. Referring to fig. 48, the query input device 4800 may search a plurality of devices 4801 to 4803 connected to the query input device 4800 through a gateway 4810 to select one or more devices from which a query component is to be provided. Here, the query input device 4800 may select a device from which to provide the query component based on the query type of the query component. For example, when the query type of the query component to be received is voice information, the query input device 4800 may select a device including a voice input operation from among the devices 4801 to 4803.

According to an exemplary embodiment, the query input device 4800 may broadcast a request for a query component. The query input device 4800 may receive device 4801-4803 corresponding device profiles from the devices 4801-4803 responding to the request. Here, each device profile may include information regarding one or more operations provided by the corresponding device. The query input device 4800 can select a device that provides the query component based on the received device profile.

Alternatively, when the gateway 4810 manages device profiles of devices connected to the gateway 4810, the gateway 4810 may select a device providing the query component in response to the request. When the gateway 4810 selects a device that provides the query component, the gateway 4810 may send a request for the query component to the selected device.

According to another exemplary embodiment, the query input device 4800 may broadcast a request including information about the type of query. The devices 4801 to 4803 that have received the request including the information on the query type may determine whether it is possible to provide a query component having the query type included in the request. A device that provides a query component having a query type included in a request, among the devices 4801 to 4803, may transmit a response to the request to the gateway 4810 or to the query input device 4800.

When two or more devices are selected by the query input device 4800 or the gateway 4810, the query input device 4800 can display a list of the selected devices. The user may select a device from the displayed list of devices from which to enter the query components.

Fig. 49 is a flowchart illustrating a process of performing a search according to an exemplary embodiment.

Referring to fig. 49, when a query is received in operation S4510, the query input apparatus according to the present exemplary embodiment may determine whether a search mode at the time of receiving the query is a multi-modal input mode in operation S4520. Here, operation S4510 denotes that the query input device receives a query input to the query input window according to the search execution command. When it is determined in operation S4520 that the search mode is a single-mode search, the query input apparatus may perform a search based on a query type included in the query in operation S4530.

When the search mode is determined to be the multi-modal input mode in operation S4520, the query input apparatus may generate a combined query based on the plurality of query components in operation S4525. Here, a combined query means that query components having a plurality of query types are combined. According to one or more exemplary embodiments, the query components may be combined in various ways. For example, the query components may simply be combined. In detail, for example, when the text "bag" and the voice "price" are input as query components, the query may be constituted by the keyword "bag price". As another example, when a user draws two wheels on a query input window with a stylus and speaks a bicycle to input a voice signal (i.e., an audio signal), the query may be composed of or include the keyword "bicycle with two wheels". As another example, when a user draws an apple on a query input window with a stylus and speaks red to input a voice signal (i.e., an audio signal), the query may be composed of or include the keyword "red apple". As another example, when a user photographs a bag using a camera and inputs an image of the photographed bag on a query input window, the query may be composed of or include a keyword "3000 korean bag" or "$ 3000 bag". As another example, the combined query may include keywords or main features (e.g., features included in the image) added to the query components. Additionally, as another example, the combined query may include expanded keywords generated from the query components. Additionally, as another example, a combined query may be characterized in that query components are prioritized based on a priority of the query type. In operation S4525, the query input device may extract relationships between a plurality of query types included in the query, and generate a combined query based on the extracted relationships. In operation S4535, the query input device may perform a search based on the combined query generated in operation S4525.

Here, operations S4525, S4535, and S4530 may be performed by an external server instead of the query input device.

Fig. 50 is a flowchart illustrating a process of performing a search according to an exemplary embodiment.

In operation S4610, the query input device 400 according to the present exemplary embodiment may receive a query through the displayed query input window.

In operation S4620, the query input device 400 may transmit the received query to the search engine server 420. Here, when the search mode is a multi-modal search, the query transmitted to the search engine server 420 may be a combined query in which query components having a plurality of query types are combined. According to one or more exemplary embodiments, the query components may be combined in various ways. For example, the query components may simply be combined. In detail, for example, when the text "bag" and the voice "price" are input as query components, the query may be constituted by the keyword "bag price". As another example, the combined query may include keywords or main features (e.g., features included in the image) added to the query components. Additionally, as another example, the combined query may include expanded keywords generated from the query components. Additionally, as another example, a combined query may be characterized in that query components are prioritized based on a priority of the query type. According to an exemplary embodiment, the query sent to the search engine server 420 may include information indicating a search mode.

In operation S4630, the search engine server 420 may perform a single search or a multi-modal search according to the search mode in order to process the received query. In operation S4640, the search engine server 420 may transmit the search result selected in operation S4630 to the query input device 400.

Fig. 51 is a block diagram illustrating a structure of the query input device 4700 according to an exemplary embodiment.

The query input device 4700 according to an exemplary embodiment may include a display 4710, a controller 4720, and an input device 4730 (e.g., an input unit).

The display 4710 may display a query input window. The display 4710 may display various information in addition to query input. The query input device 4700 may include two or more displays 4710 depending on the type of implementation. The display 4710 may include a display device such as a Liquid Crystal Display (LCD), a light-emitting diode (LED) display, a Cathode Ray Tube (CRT) display, a Plasma Display Panel (PDP), an Organic LED (OLED) display, an active-matrix OLED (AMOLED) display, a thin-film-transistor (TFT) display, and the like. In addition, the display 4710 may include a touch sensor and a touch screen having a layered structure depending on the type of implementation. When the display 4710 performs a display operation and an input operation like a touch screen, the display 4710 may perform an operation of the input device 4730 and/or the input device 4730 may be implemented as the display 4710. Additionally, the input device 4730 may be implemented to include a display 4710.

According to an exemplary embodiment, the display 4710 may display a query input window when the search mode is a multi-modal search mode. The query input window represents an interface through which a query including a plurality of query types is received. Alternatively, when the search mode is a single search mode, the query input device 4700 may display a single query input window. A single query input window represents a query input window through which a query including only one query type is received.

The input device 4730 may receive a query including a plurality of query components corresponding to a plurality of query types through a query input window displayed by the display 4710 based on user input. The input device 4730 may utilize a keyboard, keypad, virtual keypad, track pad, writing pad, or the like to receive query components such as text or sketch information. Alternatively, the query input device 4700 may receive a query component such as an image, voice information, or video to obtain or generate a query from user input.

The controller 4720 may control elements of the query input device 4700. The controller 4720 may include a Central Processing Unit (CPU), a read-only memory (ROM) storing a control program, and a Random Access Memory (RAM) storing signals or data input from the outside of the inquiry input device 4700 or used as a memory area in a work performed by the inquiry input device 4700. The CPU may include one or more processors, such as single, dual, triple, or quad cores. The CPU, ROM and RAM may be connected to each other through an internal bus.

The controller 4720 may obtain at least one search result for the received query. When the controller 4720 comprises a search engine, the controller 4720 may directly select at least one search result for the query. When the controller 4720 does not include a search engine, the controller 4720 may send a query to a search engine server that includes a search engine and retrieve at least one search result from the search engine server. The controller 4720 may control the display 4710 to display the acquired at least one search result. The display 4710 may display the acquired at least one search result according to the control of the controller 4720.

In addition, the controller 4720 may select a plurality of query types that may be selected or used for a query. The query input window displayed by the display 4710 may receive query input according to the selected query type. In addition, as in the above-described exemplary embodiment, the query input window displayed by the display 4710 may be differently displayed according to the selected query type.

In addition, according to an exemplary embodiment, the display 4710 may display a search mode selection object for selecting a search mode. The input device 4730 may receive a user input for a search mode selection object. Here, when the user input is an input to switch the search mode to the multi-modal input mode, the controller 4720 may switch the search mode to the multi-modal input mode. When the search mode is switched to the multi-modal input mode, the display 4710 may change the query input window to a query input window including a plurality of input modes. According to an exemplary embodiment, the query input window may include regions respectively corresponding to a plurality of query types.

Additionally, the display 4710 may display a list of query types. According to an exemplary embodiment, the display 4710 may display a list of single query types when the search mode is the single mode, and the display 4710 may display a list of multi-modal query types when the search mode is the multi-modal input mode. The controller 4720 may determine at least one query type included in the query from a list of query types displayed by the display 4710. Here, to determine at least one query type, the input device 4730 may receive an input dragging and dropping a target from a list of query types to an area where a query input window is or will be displayed.

In addition, according to an exemplary embodiment, the controller 4720 may detect a query type of the received query component. The query input window displayed by the display 4710 may include an area in which the received query component (i.e., the display item corresponding to the received query component) is displayed and an area in which the query component is received. The display 4710 may display the received query component in an area corresponding to the received query component based on the detected query type.

In addition, the query input window displayed by the display 4710 may include at least one query input tool for inputting a query component corresponding to each query type.

In addition, the controller 4720 may convert text included in the search result into voice, or convert voice (i.e., audio) into text.

According to an exemplary embodiment, after or in response to a user inputting a first query input to the query input window, the controller 4720 may determine to automatically switch from a first ready state (i.e., a first state) in which a first query component may be received to a second ready state (i.e., a second state) in which a second query component may be received. For example, if the second query component corresponds to an audio or voice information query type, the controller 4720 may control automatically switching the query input device 4700 to a second ready state in which a microphone is automatically activated or operated to receive the second query component. In addition, if the second query component corresponds to an image query type, the controller 4720 may control automatically switching the query input device 4700 to a second ready state in which the camera is automatically activated or operated to receive the second query component or in which an interface to select an image is displayed to receive the second query component.

When the ready state is automatically switched as described above, the controller 4720 may control output of an indicator of the automatic switching and/or the second ready state. For example, the indicator may include at least one of: an audio indicator or output (e.g., a predetermined notification sound), a visual indicator or output (e.g., a predetermined icon, a predetermined symbol, a predetermined image, etc.), an auxiliary device output (e.g., a flashing LED or a predetermined color LED on the query input device 4700), a vibration output, and so forth.

In addition, the controller 4720 may determine to automatically switch from the first ready state to the second ready state based on determining completion of the input of the first query component. For example, if the first query component is an image, the controller 4720 may determine to automatically switch from the first ready state to the second ready state in response to an image captured by a camera of the query input device 4700 or in response to a user selection of an image. In addition, if the first query component is text or a sketch, the controller 4720 may determine to automatically switch from the first ready state to the second ready state in response to a predetermined period of time elapsing from a last user input to a region of the query input window corresponding to the first query component or the first query type.

The controller 4720 can also control changing the display of the query input window, the display of one or more regions of the query input window, or the display of a graphical user interface when switching to the second ready state. Here, the change in display may be based on a query type corresponding to the second query component. According to another exemplary embodiment, the controller 4720 may control display of a user interface to receive input of a second query component in response to switching to the second ready state.

It is to be understood that the query input window may vary and is not limited to those described above.

Fig. 52 to 57 are exemplary diagrams illustrating a method of performing a multi-modal search according to another exemplary embodiment.

Referring to fig. 52, the query input device 5200 may receive, generate, or obtain the query "bicycle," and display search results 5221 including encyclopedia information, web page information, and images searched based on the query. Here, the user 1 may scroll through the screen displayed by the display to view search results that are not currently displayed by the display.

Referring to fig. 53, the screen is scrolled so that a plurality of search results 5222, which are not previously displayed by the display, can be displayed by the display. In this case, since the screen is scrolled, the query input window 5210 may not be displayed on the screen.

The user 1 can select the image 5231 included in the search result 5222. For example, referring to fig. 54, the user 1 may select a bicycle image 5231 from among the search results 5222. Here, the user 1 may select the image 5231 using a touch screen or a mouse, although it is to be understood that one or more other exemplary embodiments are not limited thereto. When the image 5231 is selected, the query input device 5200 may display a list of query types near the selected image 5231. Referring to fig. 54, the query input device 5200 may display a floating icon 5230 near the selected image 5231. The floating icon 5230 can correspond to at least one query type.

Referring to fig. 55, the user 1 may select a floating icon 5230 displayed. Referring to fig. 56, the floating icon 5230 is selected, and thus, the query input device 5200 may display a query input area 5231 for inputting a query type corresponding to the floating icon 5230. The query input device 5200 may receive the query components through the query input area 5231.

Referring to fig. 57, the query input device 5200 can perform a search based on the selected search results and a query 5210-1 including a query component received through the query input area 5231. As shown in fig. 57, based on the bicycle image and the text "manufacturer", the query input device 5200 may display a bicycle manufacturer home page included in the image and information about other products manufactured by the bicycle manufacturer in the result display area 5223. That is, the query input device 5200 receives the search result selected from the search result display area and receives each query component using the query input area 5231, and thus, in the exemplary embodiment of fig. 56, it can be described that the query component is received using the search result display area and the query input area 5231 as a query input window.

Fig. 58 is an exemplary diagram illustrating a method of displaying a query type list according to another exemplary embodiment.

According to the present exemplary embodiment, the query input device may display a single mode selection object 5801, a multi-modal input mode selection object 5802, a query input window 5810, a search button 5830, and a single query type list 5820-1. As shown in fig. 58, the single query type list 5820-1 may include a plurality of icons (each icon corresponding to a query type) that respectively correspond to a plurality of query types.

The query input device may determine the query type through which the query input window 5810 is received using icons included in the single query type list 5820-1. For example, when the text icon 5821-1 is selected, the query input device may display a user interface for inputting text into the query input window 5810. Alternatively, when the image icon 5821-2 is selected, the query input device may display a user interface for inputting an image into the query input window 5810. In addition, when the document icon 5821-3 is selected, the query input device may display a user interface for inputting a document into the query input window 5810. In addition, when the sketch icon 5821-4 is selected, the query input device may display a user interface for inputting sketch information into the query input window 5810. In addition, when the camera icon 5821-5 is selected, the query input device may execute a camera application and display an image acquired with the camera application in the query input window 5810. In addition, when the music icon 5821-6 is selected, the query input device may perform a music search based on voice information acquired using a microphone. Alternatively, when the record icon 5821-7 is selected, the query input device may operate a microphone and utilize the microphone to obtain voice information.

Fig. 59 is an exemplary diagram illustrating a method of displaying a query type list according to another exemplary embodiment.

When the multi-modal input mode selection object 5802 is selected, the query input device may set the search mode to the multi-modal input mode. According to the present exemplary embodiment, when the search mode is the multi-modal input mode, the query input device may display a multi-modal query type list 5820-2. In this case, the multimodal query type list 5820-2 may include a combined icon in which a plurality of icons are combined. The combined icon can utilize an icon included in the combined icon to indicate a multimodal query type. In addition, according to an exemplary embodiment, the query input device may indicate a priority of the query type based on the display of the icon. For example, a combined icon 5822-1 of an image and text shown in fig. 59 indicates that the priority of the image is higher than that of the text. In addition, a combined text and speech icon 5822-2 indicates that text has a higher priority than speech. In addition, a combined camera, voice, and text icon 5822-3 indicates that the priority of the query type is in order of camera, voice, and text.

Fig. 60 is an exemplary diagram illustrating a method of generating a list of multimodal query types according to an exemplary embodiment.

According to an exemplary embodiment, the query input device may generate a plurality of combined icons 6020 corresponding to a plurality of query types using a plurality of icons 6010 (each icon corresponding to one query type) corresponding to a plurality of query types, respectively. For example, when the user selects the text icon 6011 and the image icon 6012 from among the icons 6010, the query input apparatus may generate a combined icon 6021 in which the text icon 6011 is combined with the image icon 6012. In this case, the user may also set the priority of the query type (e.g., according to the order in which the icons are selected or by modifying the combination icon 6021) and the combination icon 6021 may reflect the set priority. Alternatively, the priority may be preset or predetermined.

According to an exemplary embodiment, when the search mode is a multi-modal search mode, the query input device may display a multi-modal query type list including the user-set combination icon 6020.

Fig. 61 to 65 are exemplary diagrams illustrating a method of performing a multi-modal search according to another exemplary embodiment.

According to the present exemplary embodiment, the query input device 6100 may display an indicator 6102 indicating that the search mode is a multi-modal input mode. In addition, the query input device 6100 may display a query input window 6110 and a search button 6130.

Further, according to the present exemplary embodiment, the query input device 6100 may display a set 6120 of query input tools for entering a query contribution when the query input window 6110 is selected. Alternatively, according to another exemplary embodiment, the query input device 6100 may display the set 6120 of query input tools for entering a query contribution when a menu button included in the query input device 6100 is selected. These query input tools can be included in the query input window 6110.

As shown in fig. 61, when the user 1 selects a file open icon 6121 from the set 6120 of query input tools, the query input device 6100 may display a user interface for selecting a file. Fig. 62 is an exemplary diagram illustrating a user interface for selecting a file.

As shown in fig. 62, when user 1 selects file 6122, query input device 6100 may receive the selected file 6122 as a query component. Additionally, the query input device 6100 can detect a query type corresponding to the selected file 6122. For example, as shown in FIG. 62, when the extension of the selected file 6122 is jpg, the query input device 6100 may determine that the query type corresponding to the selected file 6122 is an image.

The query input device 6100 may display the received query component (i.e., the display items corresponding to the received query component) according to the detected query type. For example, referring to fig. 63, the query input device 6100 may display a received image 6111 in an area 6103 where the query components are displayed. In FIG. 63, the area 6103 displaying the query contribution is illustrated as being combined with a query input window 6110 that includes an area for receiving the query contribution, although it is understood that the form of the area 6103 displaying the query contribution may vary in one or more other exemplary embodiments. For example, the area 6103 may be included in the query input window 6110, or may be arranged in a separate space. When multiple query components are received sequentially or simultaneously through the query input window 6110, the received multiple query components may be accumulated or displayed together in area 6103, or displayed in separate areas.

In addition, the query input device 6100 may receive additional query components through the region included in the query input window 6110 that receives the query components. For example, referring to fig. 64, user 1 may enter the text "obama" into query input window 6110.

When user 1 selects search button 6130, query input device 6100 may perform a search based on the accumulated query contribution(s) and the detected query type(s). For example, referring to fig. 64 and 65, the query input device 6100 may display a video 6140 of an olyma delivering a lecture as a search result based on an image 6102 of the lecture scene and the text 6113 "olyma".

FIG. 66 is a block diagram illustrating a framework for performing multi-modal searches based on queries having multiple query types, according to an exemplary embodiment. In detail, FIG. 66 is a block diagram illustrating a framework for query-adaptive multimodal search.

The query interface 6600 can receive a combined query 6610 that combines multiple query components. The plurality of query components may include at least one of keywords 6611, images 6612, videos 6613, speech 6614, sketch information 6615, contextual information 6616, and so on. Here, the context information 6616 represents information that makes the query clearer, such as the current state or personal history of the user and preference information. For example, context information 6616 may include a priority of the query type.

Additionally, the query interface 6600 may include a unit or device for receiving the combined query 6610. For example, the query interface 6600 may include at least one of a keyboard to receive keywords 6611, a camera to obtain images 6612 or videos 6613, a microphone to obtain speech 6614, a touch screen to obtain sketch information 6615, a sensor to obtain context information 6616, and so on.

The search method(s) 6620 represent algorithm(s) for matching queries to databases to select documents depending on their appropriateness. For example, in a video search system, thumbnail images of videos may be matched with visual content by a single search method while a split search method is processing a query text keyword and matching the query text keyword with speech recognition information. The combined query 6610 may be processed by multiple search methods 6620 to obtain search results.

The database of matching queries may include a collection of documents 6630(s). The database includes a plurality of pieces of information to be searched. The documents included in the database may have different modalities. Each document represents one unit of information included in the database. For example, each document may include a page on the Web, a screen in a video corpus, or an image of a collection of photographs.

The query adaptation module 6640 may adjust the processing order of the search method(s) 6620 that process the query. For example, a text search method may be weighted when it is desired to search for photos of celebrities in a news video, while an example-based image search method may be weighted when it is desired to search for sports scenes.

The search method merge module 6650 may merge search results obtained by multiple search methods 6620. The merged search results may be output by search result output module 6660.

At least one of the search method 6620, the database, the query adaptation module 6640, the search method merge module 6650, and the search result output module 6660 may be applied to the external device. The external device may be a cloud computer or a server.

The external device (e.g., a cloud computer) may store the search method 6620 and the database, and may include a query adaptation module 6640, a search method consolidation module 6650, and a search result output module 6660. The external device may perform a search using a query received from the query input device and output search results to the query input device.

FIG. 67 is a block diagram schematically illustrating a system and/or a processor according to one or more exemplary embodiments. Referring to FIG. 67, when the search mode is a multi-modal search mode, the query input device may perform a search based on the query type of the received query components. For example, when the query input device receives a first query component 6705 of a text query type and a second query component 6707 of an image query type, the query input device may perform a search based on keywords corresponding to the first query component 6705 (i.e., text) and image quanta (i.e., images) of the second query component 6706. To process the second query component 6707, the query input device may utilize the second query component understanding component 6712 to identify characteristics of the second query component 6707. The second query contribution understanding component 6712 may be referred to as a second query contribution processing component. Alternatively, when the query type of the second query component 6707 is an image, the second query component understanding component 6712 may be referred to as an image processing component. Features extracted from the second query component 6707 by the second query component understanding component 6712 may be assigned descriptor keywords by the second query component features component 6722. For example, when the query type of the second query component 6707 is an image query type, the second query component understanding component 6712 can identify features of the second query component 6707 (i.e., image). In this case, the features of the second query component 6707 may include at least one of image text features and image visual features. Additionally, the second query component feature component 6722 can assign a descriptive keyword to at least one of the identified image textual features and image visual features of the second query component 6707. An example in which the query type of the second query component 6707 is an image query type will be described in detail with reference to fig. 68 to 72.

The metadata analysis component 6714 can analyze metadata associated with the second query component 6707. When the second query component 6707 comprises a file, the metadata may be built into the file by an Operating System (OS), such as a title or a comment stored in the file, or may include information stored with the file. The metadata may include text in the URL path that is entered to identify the query components to be used for the search, or may include related text, such as text located in a web page or text-based document or located near corresponding information for information (e.g., images, etc.) embedded therein. The second query contribution feature component 6722 may identify keyword features based on the output of the metadata analysis component 6714.

The second query component feature component 6722 may identify the first query component 6705 and any additional features, and the resulting query may then optionally be altered or expanded by the component 6732. Query alteration or expansion may be performed by the metadata analysis component 6714 and the second query constituent features component 6722 based on features extracted from the metadata. Alternatively, query changes or extensions may be performed based on feedback received with UI interaction component 6762. Additionally, the feedback may include query suggestions 442 based on response results to current or previous queries in addition to additional query inputs received by the user. Additionally, the optionally expanded or altered query may be used to generate (6752) a response result. In FIG. 67, the result generation operation (6752) may be identified using one or more query response documents in a database 6675, the database 6775 including features and quanta of a plurality of pieces of data (e.g., documents). Database 6775 may include an inverted index or another any type of storage format for identifying responsive results based on a query.

According to one or more exemplary embodiments, the result generation operation 6762 may generate one or more types of results. Depending on the circumstances, the most promising results may be identified with a high priority result response or a few high priority result responses. Promising results may be provided as response 6744. Alternatively, a prioritized list of response results may be used. The list may be provided by prioritizing the plurality of combined results 6746. Interactions with the user (including operations to display results and operations to receive query components) may be performed by the UI interaction component 6762.

In FIG. 68, a method of processing an image (i.e., a query component having an image query type) 6800 with an operator algorithm for identifying a plurality of points of interest (POIs) 6802 is illustrated. The operator algorithm may comprise any available algorithm for identifying POIs 6802 in the image 6800. In one or more exemplary embodiments, the operator algorithm may be different from a gaussian algorithm or a laplacian algorithm. In one or more exemplary embodiments, the operator algorithm can analyze the image 6800 two-dimensionally. Alternatively, when the image 6800 is a color image, the image 6800 may be changed to a gray scale.

In addition to any point of the image 6800 as shown in FIG. 68, the POI 6802 may also include a region 6902, a region, a group of pixels, and a feature in the image 6800 as shown in FIG. 69. Hereinafter, POI 6802 and the plurality of segments 6902 may be referred to as POI 6802, but POI 6802 may include all POIs 6802 and segments 6902. In one or more exemplary embodiments, the POI 6802 can include a special or identifiable feature of the image 6800 that is located in a stable region of the image 6800. For example, the POI 6802 may be located in a region of the image 6800 having sharp features that represent high contrast between the features illustrated in 6802a and 6902 a. On the other hand, the POI may not be located in a region of a particular color or gray level, designated 6804 (i.e., an area without special features or contrast).

The operator algorithm may identify any number of POIs 6802, e.g., thousands of POIs, in the image 6800. The POI 6802 may be a combination of a point 6802 and a section 6902 in the image 6800, and the number of POIs may be changed or varied depending on the size of the image 6800. The second query contribution processing component 6712 may compute a metric for each POI 6802 and prioritize the POIs 6802 according to the computed metrics. The metric may include signal-to-noise ratio (SNR) of the image 6800 in the POI 6802. The second query component processing component 6712 may select a subset of POIs 6802 for additional processing based on the priority. According to one or more exemplary embodiments, hundreds of POIs 6802 with the highest SNR may be selected. The number of POIs 6802 selected may be changed or varied. According to one or more exemplary embodiments, no subset may be selected and all POIs may be targeted for additional processing.

As shown in fig. 70, a set 7000 of tiles 7002 corresponding to the selected POI 6802 may be identified. Each tile 7002 may correspond to a single POI 6802 selected. Tile 7002 may include an area of image 6800 that includes POI 6802. The size of the tile 7002 to be acquired from the image 6800 may be determined for each selected POI 6802 based on the output from the operator algorithm. Tiles 7002 may be of different sizes. In addition, regions of the image 6800 included in the respective tiles 7002 may overlap with each other. The shape of tile 7002 can be any shape, including a square shape, a rectangular shape, a triangular shape, a circular shape, an elliptical shape, and so forth. The shape of each tile 7002 may be square in shape or may vary from tile to tile.

As shown in fig. 70, tiles 7002 may be normalized. According to one or more exemplary embodiments, tiles 7002 may be normalized to have the same size. Tile 7002 may be normalized by increasing or decreasing at least one of its size and resolution. However, the present exemplary embodiment is not limited thereto. For example, tile 7002 may be normalized by one or more other operations, such as applying a contrast increase operation, applying a blob removal operation, applying a sharpening operation, and applying a grayscale.

According to one or more exemplary embodiments, a descriptor may be determined for each normalized tile. The descriptor may be a description of a tile that may be added as a feature for image searching. The descriptors can be determined by computing statistics for the pixels in each tile 7002. According to one or more exemplary embodiments, the descriptors may be determined based on statistics of the gray level slopes of the pixels in each tile 7002. The descriptors can be visually expressed as histograms for each tile 7002, as can the plurality of descriptors 7102 shown in fig. 71 (tile 7002 shown in fig. 70 can correspond to similarly arranged descriptors 7102 in fig. 71). However, the present exemplary embodiment is not limited thereto. For example, a descriptor may be described as a multi-dimensional vector representing pixel gray level statistics for pixels in a tile. The T2S 236 dimensional vector may be an example of a vector representing a pixel gray scale statistic.

As shown in fig. 72, a quantization table 7200 can be used to associate a descriptor 7202 with each descriptor 7102. Quantization table 7200 can include any table, index, graph, or other data structure for mapping descriptor 7102 to descriptor key 7202. Quantization table 7200 has various forms well known to those of ordinary skill in the art. According to one or more exemplary embodiments, the quantization table 7200 may be generated by processing a number of images to identify descriptors 7102 of the respective images. Subsequently, the identified descriptors 7102 may be statistically analyzed to identify clusters or groups of similar descriptors 7102 or descriptors 7102 having statistically similar values. For example, the values of the variables in the T2S2 vector are similar. The representative descriptor 7204 for each cluster may be selected and may be assigned a corresponding descriptor key 7202 in addition to its position in the quantization table 7200. The plurality of descriptor keys 7202 may include any desired indicator for identifying the representative descriptor 7204 corresponding thereto. For example, descriptor keyword 7202 may include an integer value, alphabetic character, numeric value, symbol, text, or a combination thereof as shown in FIG. 72. According to one or more exemplary embodiments, descriptor keyword 7202 may include a sequence of characters used to identify descriptor keywords related to non-text based search patterns. For example, all descriptor keys are the first four characters of a key and may include a series of three integers followed by an underlined character. In addition, such an initial sequence may be used to identify descriptor keywords associated with the image.

The representative descriptor that most closely matches each descriptor 7102 may be identified in quantization table 7200. For example, descriptor 7102a shown in fig. 71 may be most similar to representative descriptor 7202a corresponding to quantization table 7200 shown in fig. 72. For example, descriptor 7102a shown in fig. 71 may be most similar to representative descriptor 7204a corresponding to quantization table 7200 shown in fig. 72. Thus, the descriptor key 7202 of each descriptor 7102 may be associated with an image 6800. For example, descriptor 7102a may correspond to descriptor identifier 7202a of "1". The descriptor keywords 7202 associated with the image 6800 can be different, and one or more of the descriptor keywords 7202 can be related to the image 6800 multiple times. For example, the image 6800 can include the descriptor keyword "1, 2, 3, 4" or "1, 2, 3". According to one or more exemplary embodiments, in order to account for features such as variations in an image, the descriptor 7102 may be mapped to two or more descriptor identifiers 7202 by identifying two or more representative descriptors 7204 that most closely match the descriptor 7102 and identifying a descriptor key 7202 for each of the two or more representative descriptors 7204. Based on this, the content of the image 6800 with a set of identified POIs 6802 may be expressed by a set of descriptor keywords 7202.

According to one or more exemplary embodiments, additional image-based search methods may be integrated into the search scheme. For example, a face recognition method may provide an image search based on another method. As described above, a face recognition method may be utilized to determine the identity of a person in an image in addition to or in place of identifying a descriptor keyword as described above. The identity of the person in the image may be used to supplement the search query. Alternatively, when metadata for various people is included in the repository, the stored metadata may be utilized to supplement the query.

The above description may provide a description for adaptively modifying a search scheme of an image query type to another search scheme such as text. Similar adaptation modifications may be performed for search methods of different query types (e.g., audio query types). According to one or more exemplary embodiments, any type of audio-based search method may be used. A search using query components having an audio query type may use features of one or more types for identifying audio files having similar characteristics. As described above, the audio features may be associated with descriptor keywords. The descriptor keyword may have a format that indicates the keyword associated with the audio search, as would be the case if the last four characters of the keyword were made to correspond to the four digits following the hyphen.

In the above-described exemplary embodiments, it is described that the query input window displays display items corresponding to query components (i.e., query inputs). Here, it is to be understood that the display items may be the query components themselves or representations of the query components (e.g., waveforms, thumbnail images, preview images, etc.). In addition, according to one or more exemplary embodiments, a first display item corresponding to a first query type and a second display item corresponding to a second query type may be displayed such that the first query type and the second query type are distinguishable from each other.

One or more of the illustrative embodiments may be implemented in the form of storage media including computer-executable instructions, such as program modules, executed by a computer. Computer readable media can be any available media that can be accessed by the computer and includes both volatile media, such as RAM, nonvolatile media, such as ROM, and removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile, nonvolatile, and removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media may be computer readable instructions and other data in a modulated data signal, such as a data structure, or program module, or other transport mechanism, and includes any information delivery media. Examples of computer storage media include ROM, RAM, flash memory, CD, DVD, diskette, or tape. It is also to be understood that one or more of the above components and elements of the above apparatus and devices may comprise hardware, circuitry, one or more processors, and so forth.

It should be understood that the exemplary embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each exemplary embodiment should generally be considered as available for other similar features or aspects in other embodiments.

While one or more exemplary embodiments have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the inventive concept as defined by the following claims.

Claims

1. A method of an electronic device, comprising:

presenting an image including at least one product on a display of an electronic device;

receiving a voice query from a user of the electronic device related to the at least one product in the image;

generating a search query for information related to the at least one product based on a voice query including the image of the at least one product and related to the at least one product;

obtaining search results including information related to the at least one product using the search query; and

presenting the obtained search results including information related to the at least one product on a display of the electronic device.

2. The method of claim 1, wherein the obtaining the search results comprises:

sending a search query to a server for information related to the at least one product; and

receiving, from the server, search results including information related to the at least one product, wherein the information related to the at least one product is found by the server based on the search query.

3. The method of claim 1, wherein the method further comprises selecting a partial region of the image based on at least one of an analysis performed on the image and a voice query, and

wherein the partial area comprises one product among the at least one product.

4. The method of claim 3, wherein generating the search query comprises: generating a search query related to the product among the at least one product based on the selected partial region of the image and the voice query.

5. The method of claim 4, wherein obtaining the search results comprises: obtaining search results related to the product among the at least one product by using a search query related to the selected partial region.

6. The method of claim 1, presenting search results comprising presenting a GUI and an image together on a display of an electronic device, the GUI comprising information related to the at least one product.

7. The method of claim 3, presenting search results comprising presenting a GUI on a display of the electronic device with the selected partial region of the image, the GUI comprising information related to the at least one product.

8. The method of claim 1, the information related to the at least one product comprising at least one price of the at least one product.

9. The method of claim 1, wherein the information related to the at least one product comprises an image and a price of a search result product related to the at least one product.

10. The method of claim 1, wherein the search query comprises an image query based on an image including the at least one product and a voice query from a user.

11. An electronic device, comprising:

a display;

a memory storing instructions; and

at least one hardware processor configured to execute the instructions to at least:

12. The electronic device of claim 11, wherein the at least one hardware processor is configured to execute instructions to:

receiving search results including information related to the at least one product from a server, an

Wherein the information related to the at least one product is found by the server based on the search query.

13. The method of claim 11, wherein the at least one hardware processor is configured to execute instructions to:

selecting a partial region of the image based on at least one of an analysis performed on the image and a voice query, an

Wherein the partial area comprises one product among the at least one product.

14. The method of claim 13, wherein the at least one hardware processor is configured to execute instructions to:

generating a search query related to the product among the at least one product based on the selected partial region of the image and the voice query.

15. A computer-readable recording medium having instructions recorded thereon, the instructions being executable by at least one hardware processor of an electronic device with a display to cause the electronic device to perform operations comprising: