TW202004524A - Search method, electronic device and non-transitory computer-readable recording medium - Google Patents

Abstract

Provided are a method and an apparatus for searching for and acquiring information under a computing environment. The apparatus includes: at least one input device configured to receive a first query input of a first query type and a second query input of a second query type; and a controller configured to output a query input window including a first display item corresponding to the first query input and a second display item corresponding to the second query input, to automatically switch, in response to receiving the first query input, the apparatus from a first state of receiving the first query input of the first query type to a second state of receiving the second query input of the second query type, and to obtain a search result according to a query based on the first query input and the second query input.

Description

Search method, electronic device and non-transitory computer-readable recording medium

This application claims Korean Patent Application No. 10-2014-0062568 filed on May 23, 2014 and Korean Patent Application No. 10-2014-0167818 filed on November 27, 2014 in the Korean Intellectual Property Office and Korean Patent Application No. 10-2015-0025918 filed on February 24, 2015, and the priority of the US Non-Provisional Patent Application No. 14/588,275 filed on December 31, 2014 in the US Patent and Trademark Office The part of the number is continued from the application, and the entire disclosure content is incorporated herein by reference.

Devices and methods consistent with the exemplary embodiments are related to searching and acquiring information in a computing environment, and more specifically, performing searches based on various needs of users.

Various methods of searching and obtaining information have been developed. Generally, a text-based search is performed to search for information in a computing environment. Text-based searches use search queries that contain one or more text components such as words or phrases. The text components match each other or are compared with an index or data to identify files such as web pages that contain text content, metadata, file names, or textual representations similar to text components.

As technology improves, the information to be searched is further diversified, and the amount of this information increases. Therefore, in addition to the text component, different modal components can be used to perform the search.

Aspects of one or more exemplary embodiments provide a method and apparatus that receives a single-modal query or a multi-modal query and performs a search by using the received query.

Additional aspects will be set forth in the following description and will be partially apparent from the description, or may be learned by practicing exemplary embodiments.

According to an aspect of the exemplary embodiment, there is provided an apparatus including: at least one input device configured to receive a first query input of a first query type and a second query input of a second query type; and a controller, Configured to output a query input window including a first display item corresponding to the first query input and a second display item corresponding to the second query input, in response to receiving the first query input, the device automatically receives the first query The first state of the first query input of the type is automatically switched to the second state of receiving the second query input of the second query type, and the search result according to the query is obtained based on the first query input and the second query input.

The second query type may be an audio query type; and in response to receiving the first query input, the controller may be further configured to automatically activate the microphone configured to receive the second query input.

The second query type may be an image query type; and in response to receiving the first query input, the controller may be further configured to automatically start the camera configured to receive the second query input.

In response to receiving the mode switching input, the controller may be further configured to switch the search mode from the multi-modal input mode receiving the first query input and the second query input via the query input window to generating a query to receiving A query type of input to produce a single input mode of query.

At least one input device may include a first input device configured to receive a first query input and a second input device different from the first input device and configured to receive a second query input.

According to an aspect of another exemplary embodiment, there is provided an apparatus including: a display configured to display a query input window; at least one input device configured to receive a first query input and a first query type of a first query type A second query input of two query types; and a controller configured to obtain search results based on the query based on the first query input and the second query input, wherein the display is further configured to simultaneously display corresponding to on the query input window The first zone of the first query type and the second zone corresponding to the second query type.

The controller may be further configured to determine the first query type of the first query input and the second query type of the second query input; and the display may be further configured to display the first area according to the determined first query type And the second zone according to the determined second query type.

The display may be further configured to display a query input window, wherein the first display item corresponding to the first query input and the second display item corresponding to the second query input are displayed at the same time, so that the first query type and the first query type can be distinguished from each other Second query type.

According to an aspect of another exemplary embodiment, an apparatus is provided, including: a display; a microphone, configured to obtain voice information; a camera, configured to obtain image data; a memory, configured to store text data , Image data and audio data; and the controller is configured to display the display items for selecting the query type, display the query input window corresponding to the query type selected by the display item, obtained based on the query input received through the query input window Search results, and control the display to display the search results, where the query input includes image data obtained by the camera, text data stored in the memory, image data stored in the memory, and audio data stored in the memory At least one.

The query type can come from multiple query types including text query, image query and audio query; and when the selected query type is audio query, the controller can be further configured to control the display to display on the query input window for receiving The voice information obtained through the microphone is at least one of a display item for query input and a display item for receiving audio data stored in the memory as a query input.

The query type can come from multiple query types including text query, image query, and audio query; and when the selected query type is image query, the controller can be further configured to control the display to display the image obtained through the camera for receiving images At least one of the display item of the data and the display item stored in the memory for receiving image data is used as a query input on the query input window.

The device may further include: a handwriting input unit configured to receive handwritten images, where the query type may come from multiple query types including text query, image query, audio query, and handwritten query, and where the selected query type is a handwritten query At this time, the controller can be further configured to control the display to display a display item for receiving handwritten images on the query input window.

When selecting multiple query types via display items, the controller may be further configured to control the display to display display items for receiving multiple query inputs on the query input window.

According to an aspect of another exemplary embodiment, a method is provided, including: receiving a first query input of a first query type and a second query input of a second query type; including, by means of the device output, corresponding to the first query input And the query input window of the second area corresponding to the second query input; in response to receiving the first query input, the device automatically switches the first state from receiving the first query input of the first query type to receiving the first The second state of the second query input of the second query type; and obtaining search results according to the query based on the first query input and the second query input.

The method may further include displaying a second display item corresponding to the second query input on the second area and a first display item corresponding to the first query input on the first area in response to receiving the second query input.

The second query type may be an audio query type; and the automatic switching may include automatically activating a microphone for receiving the second query input in response to receiving the first query input.

The second query type may be an image query type; and automatic switching may include automatically starting a camera for receiving the second query input in response to receiving the first query input.

According to an aspect of another exemplary embodiment, a method for obtaining search results by a device is provided, the method including: displaying a display item for selecting a query type; receiving user input based on the displayed display item; and based on the received The user input of selects at least one query type; displays a query input window corresponding to the selected at least one query type; and obtains search results based on the query input received via the displayed query input window.

The query type can come from multiple query types including text query, image query and audio query; and the method can further include displaying on the query input window for receiving via the device included or connected when the selected query type is audio query The voice data obtained to the microphone of the device and the audio data stored in the device or the memory connected to the device are used as the display items for query input.

According to an aspect of another exemplary embodiment, a method is provided, including: displaying a query input window; receiving text data and handwritten images through the displayed query input window; and based on the received text data and received handwritten images To obtain search results.

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the drawings, wherein the same graphical element symbols refer to the same elements throughout. In this regard, the exemplary embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Therefore, the following describes exemplary embodiments of the present invention only by referring to the drawings to describe exemplary embodiments. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. When before the list of components, an expression such as "at least one of" modifies the entire list of components, and does not modify individual components of the list. The expression "at least one of" modifies the list of all components, and does not modify every one of the components included in the list.

In the following invention, when a part (or element, device, etc.) is referred to as "connected" to another part (or element, device, etc.), it should be understood that the former can be "disconnected" from the latter, or via an intervening part (Or components, devices, etc.) "electrically connected" to the latter. In addition, when a part (or element, device, etc.) is described as including (or containing or having) some elements, it should be understood that it may include (or contain or have) other elements only, or in the absence of specific restrictions It may include (or contain or have) other elements than theirs.

In this specification, query means a command for performing a search. The query may contain information obtained as search results. The query may include at least one query component (ie, query input). The query component represents a unit of information constituting a query. Also, query component input to a device (eg, query input device) may be referred to as query input. For example, the query component may include at least one of the following: keywords, image data, sketch information, video data, and audio data included in the text input by the user to the query input device. The audio data may include voice information. The query type can represent the modality of the query component. In this article, modal is defined as a source of information about the search database used for searching in addition to the human-perceptible appearance. For example, the query type may indicate to which type of text, image data, sketch information, video data, and audio data the query component corresponds. A query that includes multiple query types indicates that multiple query components included in the query correspond to multiple query types. For example, when the query includes a first query component in which the query type is text and a second query component in which the query type is image, the query includes multiple query types. That is, the query type may be at least one of text query, image query, audio type, and handwriting query.

The audio data may include at least one of sound, voice, audio, and music. In this specification, the search mode may mean the operation mode of the method of performing the search. The search mode may include a single input mode (that is, a single input mode) and a multi-modal input mode (that is, a multi-modal input mode). The single input mode may represent an operation mode in which a search is performed based on a query including one or more query components having one query type. The multimodal input mode may represent an operation mode in which a search is performed based on a query including multiple query components having multiple query types.

Hereinafter, exemplary embodiments will be described in detail.

FIG. 1 is a flowchart illustrating a procedure for performing a search according to an exemplary embodiment.

Referring to FIG. 1, in operation S110, the computing device may display a query input window that enables a user to input a query. The query input window represents a user interface, and the computing device receives the user's query input through the user interface. The query input window may include a query input tool for inputting query components. For example, the query input window may include at least one of the following: a text box, a uniform resource locator (URL) input window, a file upload button, a sketch input window, an application execution button, and the like. The text box represents a box displaying text input by using an input device (for example, keyboard, keypad, virtual keypad, etc.). The URL input window represents a user interface through which a URL indicating the location of information input as a query is input. The file upload button represents a button for performing an operation in which the computing device selects a file when the user selects it and uploads the selected file. The sketch input window represents a user interface that displays information input by a user by using an input device (eg, tablet computer, touch panel, track pad, mouse, stylus, etc.). The application execution button represents a button that enables execution of the application corresponding to the selected button. For example, when an application execution button corresponding to photography is selected, an application for photography can be executed. As another example, when an application execution button corresponding to recording of sound is selected, an application for recording sound may be executed. As another example, the query input device can capture information such as Quick Response (QR) codes from images acquired by operating the camera, and receive the query component by using QR codes.

According to an exemplary embodiment, when the search mode corresponds to a multi-modal search (ie, a multi-modal input mode), the query input device may display multiple query components for receiving multiple query types (ie, a query Input) query input window. Alternatively, when the search mode corresponds to a single search (ie, a single input mode), the query input device may display a single query input window. A single query input window indicates that a query input window that receives only one query type query.

Subsequently, in operation S120, the query input device may receive a query via the displayed query input window. Here, the query input device may receive a query including multiple query types, that is, a query including multiple query components including multiple query types. That is, a query including multiple query components of multiple query types can be received.

Subsequently, the query input device may select at least one search result based on the received query. Here, at least one search result can be selected by using a search engine. A search engine represents hardware, software, or a combination of search information based on a query. The search engine may be included in the query input device, or may be included in a separate device (eg, web server, media server, web server, etc.). When the search engine is included in the separate device, the query input device may transmit the received query to the separate device. The query input device may obtain selected search results from a separate device in response to the transmitted query.

In operation S130, the query input device may display the selected search result. Here, the query input device may prioritize the search results. The query input device may display the search results based on the priority of the search results.

According to an exemplary embodiment, the query input device may convert (or communicate with a server or another device to convert) the text included in the search results into speech, and output the converted speech. Alternatively, the query input device may convert the speech included in the search results into text, and output the converted text. Therefore, the query input device enables the blind or hearing impaired to check the search results.

FIG. 2 is a conceptual diagram illustrating the configuration of a user interface according to an exemplary embodiment. The query input device may display the query input window 210 and the result display area 220 illustrated in FIG. 2, but it should be understood that one or more other exemplary embodiments are not limited thereto. The query input device can display the query input window 210 and the result display area 220 separately.

The query input window 210 may receive a first query component 211 (ie, first query input) corresponding to the first query type and a second query component 212 (ie, second query input) corresponding to the second query type. The query input window 210 may be displayed differently based on the first query type and the second query type. In addition, the query input window 210 may receive a single query or a query containing two or more queries of different types.

The result display area 220 may include a list of response results 221 and 222 (ie, search results). The first response result 221 may include summary information about the information identified in response to the search. For example, the first response result 221 may include a thumbnail of an image file, some text included in the file, a link to the searched file, an icon, and the like.

FIG. 3 is a conceptual diagram illustrating a structure for constructing a query input device 300 according to an exemplary embodiment. FIG. 3 illustrates an exemplary environment for implementing an exemplary embodiment. The query input device 300 may be implemented as a computing device. For example, the query input device 300 may be implemented as a type such as: a personal computer (PC), a smart phone, a connected television (TV), a notebook (laptop) computer, Personal digital assistant (PDA), tablet PC, portable multimedia player, mini-notebook computer, workstation, etc.

Exemplary embodiments may be described that are associated with machine-available instructions or computer program code that include program modules such as program modules executed by devices such as computers, personal portable terminals, or handheld devices Computer executable instructions. Generally, program modules containing routines, programs, objects, components, and data structures represent code that performs specific tasks or abstracts data types. The exemplary embodiments can be implemented in various systems including handheld devices, consumer electronic devices, general-purpose computers, and special computing devices. Also, the exemplary embodiments may be implemented in a distributed computing environment.

The query input device 300 may include a memory 320, at least one processor 330, at least one output device 340, at least one input/output (input/output; I/O) port 350, at least one I/O component 360, a power supply 370, and Connecting element bus 310. The bus 310 may include one or more types of buses, such as an address bus, a data bus, or a combination thereof. For simplicity, the functional blocks of FIG. 3 are illustrated as lines. However, the query input device 300 may be substantially configured with various components. For example, an output device 340 such as a display device may be regarded as an I/O component 360. In addition, the processor 330 may include a memory 320.

The query input device 300 may include various computer-readable media. The computer-readable media may be any available media that can be accessed by the query input device 300, and may include volatile media, non-volatile media, removable media, and non-removable media. Computer storage media may include volatile media, non-volatile media, removable media, or non-removable media implemented in any method or technology for storing computer-readable instructions, data structures, program modules, or information such as data. Computer storage media can include RAM, ROM, EEPROM, flash memory, memory technology, CD-ROM, DVD, holographic memory, cassette tape, magnetic tape, magnetic disk or other magnetic storage devices, which can be used to encode the desired information , Or may include any media that can be accessed by the query input device 300, but is not limited thereto. In an exemplary embodiment, the computer storage medium may be selected from various types of computer storage media. In another embodiment, the computer storage medium may be selected from non-transitory computer storage media.

The memory 320 may include volatile and/or non-volatile memory type computer storage media. The memory 320 may be removable memory, non-removable memory, or a combination thereof. For example, the memory 320 may include a semiconductor memory, a hard disk drive, an optical disk drive, and the like. The query input device 300 may include one or more processors 330 that read data from various entities such as the memory 320 or the I/O component 360. The output device 340 provides the data command to the user or another device. For example, the output device 340 may include at least one of a display device, a speaker, a printing component, a vibration motor, a communication device, and the like.

The I/O port 350 allows logically connecting the query input device 300 to other devices including the I/O component 360. For example, the I/O component 360 may include at least one of the following: microphone, joystick, game board, satellite antenna, scanner, printer, wireless device, keyboard, track pad, touch screen , Rotatable dial, camera and handwriting input unit. The handwriting input unit can be a touch screen and can be used to input handwritten images. The touch screen may include an electrode magnetic reaction (EMR) pad that senses touch by an active stylus (hereinafter referred to as a pen), but one or more other exemplary embodiments are not limited thereto .

The pen may contain a coil, and the magnetic field may be generated by the coil at a certain point of the EMR pad. The EMR pad can detect the position of the generated magnetic field to detect the position where the pen touches the EMR pad. The EMR pad can detect the position of the magnetic field generated by the pen to detect the pen touch position on the touch screen. The controller can detect the pen touch position and thus receive the handwritten image.

FIG. 4 is a conceptual diagram illustrating a network environment for performing an exemplary embodiment. The network environment is only an example of an environment that can be applied to the exemplary embodiments, and may include any number of components in terms of a generalized configuration.

The network environment may include a network 410, a query input device 400, and a search engine server 420. The network 410 may include any computer network such as the Internet, an intranet, non-public and public area networks, non-public and public wide area networks, wireless data or telephone networks. The query input device 400 is a device that provides a query. According to an exemplary embodiment, the query input device 400 may output search results and query input.

Search engine server 420 may include any computing device such as query input device 400. The search engine server 420 may provide at least some operations that provide search services.

5 to 7 are exemplary diagrams illustrating a method of receiving a single searched query according to an exemplary embodiment.

When the search mode is a single search mode (that is, a single input mode), the query input device according to the exemplary embodiment may display a single query input window 510 through which the query input including one query type is input. The query input device may display at least one search mode selection object for selecting a search mode. Referring to FIG. 5, the query input device can display a single input mode selection icon 501 (that is, a single input mode selection icon) and a multimodal input mode selection icon 502 (that is, a multimodal input mode selection icon) . When receiving user input for selecting the single input mode selection icon 501 from the user 1, the query input device may display a single query input window 510. Here, the user input may vary depending on the type of query input device or I/O device. For example, when the query input device is a PC that outputs a graphical user interface (GUI), the event of selecting icon 501 by clicking or selecting a single input mode using a mouse cursor can be selected for selecting a single input mode User input of icon 501. As another example, when the query input device is a handheld device including a touch screen, the touch of a part of the human body or the electronic pen on the position of the single input mode selection icon 501 displayed on the touch screen may be a single selection User input of the input mode selection icon 501.

In addition, the query input device may display a single query type list 520. The single query type list 520 may be a display item for selecting a query type. The display item may be a UI element displayed on the screen. A single query type list 520 may contain multiple objects. Each object included in the single query type list 520 may correspond to a query type. That is, the single query type list 520 may include a query of at least one of text type, image type, handwriting type, video type, and audio type. In FIG. 5, the single query type list 520 is illustrated as a set of button-type objects, but in one or more other exemplary embodiments, each of the objects may be in various forms (eg, icon, interface tool set) Etc.) Replace.

The query input device may determine the query type of the query component received by the single query input window 510 based on the selection of the single query type list 520. Referring to FIG. 6, the user 1 can determine what is included in the single query input window 510 by using an input that drags and drops one of the objects included in the single query type list 520 to the area displaying the single query input window 510 Query type. Drag and drop represents an input scheme for selecting an object, moving a position, and ending mouse click or touch input by using mouse click or touch input in the GUI. For example, when the drag-and-drop event of the object 521 corresponding to the sketch information is received by the single query input window 510 (as illustrated in FIG. 7), the sketch input for inputting the sketch information into the single query input window 510 may be displayed window. Subsequently, when an object corresponding to another query type is selected from objects included in the single query type list 520, the sketch input window may be changed to an input window for inputting query components corresponding to other query types. Here, drag and drop can be replaced with various input schemes differently. For example, according to another exemplary embodiment, the user 1 may determine the query type by using inputs such as those described above with reference to the selection of the single input mode selection icon 501. According to an exemplary embodiment, the area or position of the single query input window 510 may change or vary depending on the type of query to be displayed (ie, the display items of the query input corresponding to the query type) in the single query input window 510. Alternatively, the size of the query component (ie, the display item) displayed in the query input window 810 may change or vary depending on the area of the query component input area. For example, as the number of character inputs to the text input window increases (eg, exceeds a predetermined number), the font size of each of the characters displayed in the text input window may be reduced. For example, according to one or more exemplary embodiments, the user 1 may select the query type through the selection on the single input mode selection icon 501. According to one or more exemplary embodiments, the area or position of the single query input window 510 may be changed based on the query type of the query to be displayed on the query input window (ie, the display items of the query input corresponding to the query type) . On the other hand, the size of the query component (ie, the display item) of the query to be displayed on the single query input window 510 may be changed based on the query component input area. For example, when inputting characters, if a predetermined number or more characters are input, the text input window may be enlarged, or the characters in the text input window may be changed to a smaller size.

The controller is operable to display a query input window 510 corresponding to the query type selected via the display item 520, obtain search results based on the query input via the query input window 520, and display the search results. For example, when the query type is text, the controller is operable to display a display item that enables text input.

When the selected query type is audio, the controller is operable to display a display item on the query input window 510, the display item being used for at least one of the voice information acquired through the microphone and the audio data stored in the memory Enter as a query. For example, the icon for selecting the microphone enabling operation and the icon of the audio data stored in the memory may be displayed on the query input window 510. The microphone can receive voice input to output electrical signals, and can obtain voice information from the electrical signals. Audio data can be stored as a file in memory.

When the selected query type is an image, the controller is operable to display a display item on the query input window 510. The display item is used to select at least one of the image data acquired through the camera and the image data stored in the memory One is used as a query input. For example, icons used to select icons that enable the camera to operate and image data stored in memory can be displayed on the query input window 510. The camera can obtain image data. Image data can be stored as a file in memory.

In addition, the query input device may include a search button 530 for inputting a command that allows the search to be performed based on the query input to the single query input window 510. According to one or more other exemplary embodiments, the search button 530 may not be displayed, and may be changed to another form.

8 to 18 are exemplary diagrams illustrating a method of receiving a query of multi-modal search according to an exemplary embodiment.

When the search mode is a multi-modal search mode (ie, a multi-modal input mode), the query input device according to the exemplary embodiment may display a query input window 810 through which a query including a query type is input . When multiple query types are selected via the display item 820, the controller is operable to display display items 811 and 812 for receiving multiple queries on the query input window 810. For example, referring to FIG. 8, the query input window 810 may include a first query component input area 811 (ie, the first area) and a second query component input area 812 (ie, the second area).

The query input device may display at least one search mode selection object for selecting a search mode. Referring to FIG. 8, the query input device can display a single input mode selection icon 801 (that is, a single input mode selection icon) and a multimodal input mode selection icon 802 (that is, a multimodal input mode selection icon) . When receiving user input for selecting the multi-modal input mode selection icon 801 from the user 1, the query input device may display the query input window 810. A query input window that can receive multiple query components (ie, query input) with multiple query types may be referred to as a multi-modal query input window. Here, the user input may vary depending on the type of query input device or I/O device. For example, when the query input device is a PC that outputs a GUI, the event of clicking the multi-modal input mode selection icon 801 by using a mouse cursor to click or selecting the multi-modal input mode selection icon 801 can be a user who selects the multi-modal input mode selection icon 801 enter. As another example, when the query input device is a handheld device including a touch screen, a part of the human body or an electronic pen (for example, a stylus) selects the multi-modal input mode selection icon 801 on the touch screen The touch of the displayed position may be user input selecting the multi-modal input mode selection icon 801.

In addition, the query input device may display a single query type list 820. A single query type list 820 may contain multiple objects. Each object included in the single query type list 820 may correspond to a query type. In FIG. 8, the single query type list 820 is illustrated as a simple button type, but in one or more other exemplary embodiments, each of the objects can be replaced with various forms such as icons, interface toolsets, and the like.

The query input device may determine the query type included in the multi-modal query input window 810 based on the single query type list 820. Referring to FIG. 9, the user 1 can use one of the objects included in the single query type list 820 to drag and drop an input decision to the area where the multimodal query input window 810 is displayed to be included in the query input window 810 Query type. Drag and drop represents an input scheme for selecting an object, moving a position, and ending mouse click or touch input by using mouse click or touch input in the GUI. For example, when the drag-and-drop event of the object 821 corresponding to the text is input to the first query component input area 811 of the query input window 810 (as illustrated in FIG. 9 ), the text input to the first The text input window of the query component input area 811 is as illustrated in FIG. The text input window may contain a text box displaying the input text. Also, referring to FIG. 10, when a drag-and-drop event of an object 822 corresponding to an image is input to the second query component input area 812, a query component for inputting a query type of image to the second query component input area 812 may be displayed User interface, as illustrated in Figure 11. In this situation, drag and drop can be replaced with various input schemes differently. For example, according to one or more exemplary embodiments, the user 1 may determine the query type through the selection of the multi-modal input mode selection icon 801. In addition, the area or position of the first query component input area 811 or the second query component input area 812 may be displayed in the first query component input area 811 or along with the items to be displayed (ie, display items corresponding to the query input of the query type) The query type in the second query component input area 812 changes or changes. Alternatively, the size of the query component displayed in the query input window 810 may change or vary with the area of the query component input area. For example, as the number of character inputs to the text input window increases (eg, exceeds a predetermined number), the font size of each of the characters displayed in the text input window may be reduced.

In addition, in FIGS. 8 to 11, the query input window 810 is illustrated as receiving a query including at least two query types (ie, query input of at least two query types). It should be understood that in the exemplary embodiment, the number of query types included in the query received via the query input window 810 may be greater than, equal to, or less than two. User 1 can select at least one of the objects included in the single query type list 820. For example, one of the objects included in the single query type list 820 can be clicked by using a mouse or a mouse cursor. Alternatively, one of the objects displayed on the touch screen can be touched. However, it should be understood that one or more other exemplary embodiments are not limited thereto.

Referring to FIG. 12, the object 823 corresponding to the audio can be selected by input received from the user 1. When at least one object is selected from the objects included in the single query type list 820, the query input device may add the third query component input area 813 (ie, the third area) to the query input window 810. The query input device may increase the third query component input area 813 to the query input window 810 without changing the total area of the query input window 810. For example, referring to FIG. 13, the query input device may display the third query component input area 813 in a part of the second query component input area 812. However, it should be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, the query input device may display the third query component input area 813 on the extra area, where the query input window 810 is enlarged compared to the state before the third query component input area 813 is displayed . When selecting an object 823 corresponding to audio (as illustrated in FIG. 12), a user interface for inputting a query component of the query type of audio to the third query component input area 813 may be displayed, as illustrated in FIG. 13 .

In addition, the area of each query component input area included in the query input window 810 may be changed or may be changed. For example, when it is difficult to display all text input to the first query component input area 811 in the first query component input area 811, the area of the first query component input area 811 may increase. As another example, as illustrated in FIG. 14, the boundary 840 between the input regions of the query component may be moved based on user input. Here, the user input may be an input to drag the boundary 840, but it should be understood that one or more other exemplary embodiments are not limited thereto.

In addition, the user interface displayed in the input area of the query component may be changed to a user interface for inputting a query component corresponding to another query type. Referring to FIG. 15, an object 824 corresponding to sketch information among objects included in the single query type list 820 may be received, and an event generated by the user 1 may be received as a user input selecting the second query component input area 812. Here, the user input may be a drag and drop event that starts from the object 824 displayed in the single query type list 820 and ends in the second query component input area 812, but is not limited thereto. In this case, as illustrated in FIG. 16, a sketch input window 814 for inputting sketch information may be displayed in the second query component input area 812 of FIG. 15.

In addition, the size of the query input window 810 may be changed according to user input. The query input window 810 can be enlarged or reduced according to user input. As illustrated in FIG. 17, when receiving user input, the query input window 810 may be displayed in the zoom-in or zoom-out area 850. Here, the user input may be an input to drag the edge of the query input window 810, but it should be understood that one or more other exemplary embodiments are not limited thereto.

In addition, the position of the query input window 810 may be displayed according to user input. As illustrated in FIG. 18, when receiving user input, the query input window 810 may be displayed in the moved area 860. Here, the user input may be an input to drag a partial position of the query input window 810, but it should be understood that one or more other exemplary embodiments are not limited thereto.

19 and 20 are exemplary diagrams illustrating a method of receiving a query of multi-modal search according to another exemplary embodiment. FIG. 19 is an exemplary diagram illustrating a screen obtained by performing a search based on a single search mode. The query input window may include a first query component input area 1910 for inputting a query component corresponding to the first query type. The query component is received via the first query component input area 1910, and then when the search button 1930 is selected, a search can be performed based on the query containing the received query component. Here, the search can be performed by using the search engine included in the query input device. Alternatively, the query input device may transmit the query to the search engine server, and receive a response to the query, and thus may perform the search.

The search result may be displayed in the result display area 1920 due to the search. Referring to FIG. 19, the first search result 1921 and the second search result 1922 can be displayed in the result display area 1920.

Subsequently, when at least one search result is selected from the displayed search results based on the user input, the query type list 1940 may be displayed. Referring to FIG. 19, when the second search result 1922 is selected, a list of query types 1940 may be displayed near the second search result 1922. In FIG. 19, the query type list 1940 is illustrated as a set of objects configured with buttons, but in one or more other exemplary embodiments, each of the objects can use, for example, icons, interface toolsets, etc. Replace in various forms. In this case, a multimodal search may be performed based on the second search result 1922, and the query component may be input to the second query component input area 1920. However, the exemplary embodiments of the present invention are not limited thereto. For example, according to one or more other exemplary embodiments, a multimodal search may be performed based on the original query component input to the first query component input area 1910 and the query component input to the second query component input area 1920.

When the query type is selected from the query type list 1940 of FIG. 19, the query input device may switch the search mode from a single search mode to a multi-modal search mode. Referring to FIG. 20, a second query component input area 1915 for inputting query components corresponding to the second query type may be further displayed. Subsequently, when the search button 1930 is selected, a multimodal search may be performed based on the query component input to the first query component input area 1910 and the query component input to the second query component input area 1920. In this case, the multimodal search may be performed based on the second search result 1922 and the query component input to the second query component input area 1920, but it should be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, a multimodal search may be performed based on the original query component input to the first query component input area 1910 and the query component input to the second query component input area 1920. The search results 1923 and 1924 can be displayed in the result display area as the results of the multimodal search.

21 and 22 are exemplary diagrams illustrating a method of generating a query input window according to an exemplary embodiment. According to an exemplary embodiment, the object 2100 is selected based on the search mode to select the search mode, and then as illustrated in FIG. 21, an arbitrary area 2110 can be selected or generated based on user input received from the user 1. Here, the user input may be dragging at any position, but it should be understood that one or more other exemplary embodiments are not limited thereto. When selecting any area 2110, as illustrated in FIG. 22, the query input window may be displayed in any area 2110 according to the selected search mode. For example, when the selected search mode is a single search mode, the query input window may be a single query input window, and when the selected search mode is a multi-modal search mode, the query input window may be a multi-modal query input window. In addition, a list 2220 of search types and a search button 2230 may be displayed.

FIG. 23 is an exemplary diagram illustrating a method of receiving a query of a multimodal search according to another exemplary embodiment. In the exemplary embodiment of FIG. 23, the query input device may include an input device (for example, an input unit) that simultaneously receives input of multiple points. For example, the query input device may include a touch screen. Although a single query type list 2320 is displayed, multiple objects included in the single query type list 2320 may be selected. Referring to FIG. 23, a first object 2321 (where the query type is text) and a second object 2322 (where the query type is image) can be selected. For example, two fingers of the user can touch objects 2321 and 2322 displayed on the touch screen, respectively. Multiple objects are selected, and when the selected position of each object moves to a point, a query input window 2310 for receiving a query component having multiple query types corresponding to the multiple objects may be displayed. For example, referring to FIG. 23, two fingers touching objects 2321 and 2322 displayed on the touch screen can be moved to a point 2340 in the state of touching the touch screen and then separated from the touch screen input of. That is, an input of dragging and dropping multiple objects to a point 2340 at the same time can be received. A query input window 2310 including a text input window 2311 for inputting text and an image upload window 2312 for uploading images may be displayed. Here, one point 2340 may not correspond exactly or exactly to the same point. For example, when two points are arranged within an arbitrary distance relative to each other (for example, within a predetermined distance), it can be considered that multiple objects are located at one point.

FIG. 24 is an exemplary diagram illustrating a multimodal query type list 2410 according to another exemplary embodiment. According to an exemplary embodiment of the present invention, when the search mode is a single search mode, the query input device may display a single query type list, and when the search mode is a multimodal search mode, the query input device may display a multimodal query type list . The objects included in the multimodal query type list can represent a combination of multiple query types. Referring to FIG. 24, when the search mode is set to the multimodal search mode by using the search mode selection object 2400, a list of multimodal query types representing a combination type in which two or more query types are combined can be displayed 2410.

25A to 25E are exemplary diagrams illustrating a method of receiving a query of a multi-modal search according to another exemplary embodiment.

According to an exemplary embodiment of the present invention, a multi-modal search query can be received by using a part of the search results. Referring to FIG. 25A, the query input device 2500 may display the image 2520 as a search result obtained by performing a search based on the query input to the query input window 2510.

Referring to FIG. 25B, a partial region 2501 of the image 2520 can be selected. The partial area 2501 may be manually selected based on user input, or may be automatically selected based on analysis of search results such as image recognition. For example, when the user 1 draws a circular curve by dragging the mouse cursor or using touch input, the query input device 2500 may determine the interior of the circular curve as selected from the partial region 2501 of the image 2520.

When the partial area 2501 is selected from search results, a list of query types may be displayed. According to an exemplary embodiment of the present invention, referring to FIG. 25C, a list of query types may be displayed near the selected partial area 2501 in the form of floating icons or in the form of a floating interface tool set. A floating icon or floating interface tool set can be displayed on the content, and is a movable object.

For the displayed query type list 2520, user input for selecting the query type can be received from user 1. When selecting a query type, a query input window corresponding to the displayed query type may be displayed. For example, the partial area 2501 of the handbag included in the image 2520 illustrated in FIG. 25C may be selected. Subsequently, when an icon 2511 corresponding to a voice search among a plurality of query types is selected from among a plurality of displayed floating icons 2521, a query input window 2510 for inputting a query component (where the query type is voice) may be displayed. The voice-type query component may be called an audio query.

According to an exemplary embodiment, the query input window 2510 for inputting a query component (where the query type is voice) may include a voice (ie, audio) record icon. When the user 1 selects a voice recording icon, the query input device 2500 can operate the microphone and execute an application program for acquiring voice information. Referring to FIG. 25D, when the application program for acquiring voice information is executed, the user 1 can input voice to the query input device 2500.

When a query component is input through the query input window 2510, the query input device 2500 may receive a part of the selected search result (for example, corresponding to the partial region 2501 selected from the image 2520) and the information input through the query input window 2510 as the query component. Referring to FIG. 25E, the image input area 2511-1 (which displays the image in which the handbag is displayed) and the voice input area 2512-1 (which displays the waveform of the input voice information, the predetermined waveform, and the text converted from the voice information can be displayed At least one of the query input window 2510-1 as a display item corresponding to the query. Here, the waveform of the input voice information may be a waveform corresponding to at least one of a predetermined portion of the voice information (eg, the beginning of the voice information), a predetermined length of the voice information, a predetermined audio characteristic or characteristic of the voice information, and so on. As illustrated in FIG. 25D, when inputting the voice information "price", the query input device 2500 may perform a multimodal search based on the query including the image (which includes the handbag) and the voice "price". As illustrated in FIG. 25E, the query input device 2500 may display information about the price of the handbag included in the image in the search result display area 2520-1 as a search result obtained by performing a multimodal search.

FIG. 26 is a flowchart illustrating a procedure of displaying a query input window according to an exemplary embodiment. 27 and 28 are exemplary diagrams illustrating a method of displaying a query input window according to an exemplary embodiment. According to an exemplary embodiment, the query component may be entered in a state where the query type is not selected before entering the query.

In operation S2610, the query input device may receive the query component via the query input window. The method of receiving the query component can be implemented differently. Referring to FIG. 27, the query input device may display an indicator 2701 indicating that a search mode is set, a query input window 2710, and a search button 2720. The query input window 2710 of FIG. 27 includes a user interface 2711 for inputting a query component in which the query type is text. Here, the query input device may receive input from the user 1 to receive the query component, and drag and drop the file icon 2730 (included in the file search window 2740) to the query input window via the input. However, it should be understood that one or more other exemplary embodiments are not limited to the drag-and-drop input method. For example, the file can be selected by a click operation (for example, double-clicking) according to another exemplary embodiment. In addition, the archive search window 2740 is not limited to the form illustrated in FIG. 27. For example, according to another exemplary embodiment, the file search window 2740 of FIG. 27 may be displayed by dividing the screen of the query input device, or may be displayed by changing the screen of the query input device to the file search screen.

In operation S2620, the query input device may detect the query type of the received query component. The method of detecting the query type can be implemented differently. For example, when the received query component is a file, the query input device can detect the query type of the query component according to the file extension. In this case, when the file extension is jpg, gif, or bmp, the query input device can determine the query type of the query component as an image, and when the file extension is avi, mp4, or wmv, the query input device can Determine the query type of the query component as video. Alternatively, when the application is used to receive the query component, the query input device may detect the query type of the query component according to the type of application. For example, when receiving the information acquired by using the camera application as the query component, the query input device can determine the query type of the query component as an image, and when receiving the query component by using the voice recording application, query The input device can determine the query type of the received query component as voice information.

In operation S2630, the query input device may change the query input window to include a user interface, receive the query component corresponding to the detected query type via the user interface (and display the corresponding to the previously received query component via the user interface Query the display items of the component). In addition, display items corresponding to the received query component may be displayed. Referring to FIG. 28, when receiving the image file as shown in FIG. 27, the query input device may change the query input window 2710 configured with the text input box of the user interface 2711 through which text is input to the configured Another query input window 2710 of the image upload tool that inputs the image 2712.

FIG. 29 is a flowchart illustrating a procedure of receiving a query according to an exemplary embodiment. 30 to 32 are exemplary diagrams illustrating a method of receiving a query according to an exemplary embodiment. According to an exemplary embodiment, the query component can be entered without first selecting the query type.

In operation S2910, the query input device may receive the query component via the query input window. The method of receiving the query component can be implemented differently. Referring to FIG. 30, the query input device may display an indicator 2702 indicating that a search mode is set, a query input window 2710, and a search button 2720. In FIG. 30, the query input window 2710 is illustrated as including a user interface 2711 for inputting a query component in which the query type is text. Here, the query input device may receive input from the user 1 to receive the query component, and drag and drop the file icon 2730 (included in the file search window 2740) to the query input window via the input. However, it should be understood that one or more other exemplary embodiments are not limited to the drag-and-drop input method. For example, the file can be selected by a click operation (for example, double-clicking) according to another exemplary embodiment. In addition, the archive search window 2740 is not limited to the form illustrated in FIG. 30. For example, according to another exemplary embodiment, the file search window 2740 of FIG. 30 may be displayed by dividing the screen of the query input device, or may be displayed by changing the screen of the query input device to the file search screen.

In operation S2920, the query input device may detect the query type of the received query component. The method of detecting the query type can be implemented differently. For example, when the received query component is a file, the query input device can detect the query type of the query component according to the file extension. Alternatively, when the application is used to receive the query component, the query input device may detect the query type of the query component according to the type of application.

In operation S2930, the query input device may increase a query type according to the detected query type, and the query type is used to perform a multimodal search. Therefore, the query input device may add a region to the query input window that receives the query component corresponding to the added query type (and displays the display item corresponding to the previously received query component). Referring to FIG. 31, since the query type is added, the area of the query input window 2710 can be enlarged and displayed. Also, the user interface 2712 for inputting images can be displayed in the area where the query input window is enlarged. Information 2750 (such as preview, icon, or file name) about the received query component can be displayed in the added area.

According to one or more exemplary embodiments, the query input window 2710 may be enlarged to include a user interface 2712 corresponding to the added query type. However, the exemplary embodiments of the present invention are not limited thereto. Also, according to one or more exemplary embodiments, the form of the query input window 2710 displayed by the query input device may not be changed. Also, the internal area of the query input window 2710 may not be divided. In this case, a plurality of query components respectively corresponding to different query types input via the query input window 2710 may be displayed together in the query input window 2710.

According to another exemplary embodiment, the query input device may display a query input window. Here, when the search mode is the multi-modal input mode, the query input window may include an area that displays the received query component (that is, displays a display item corresponding to the previously received query component) and an area for receiving the query component Area. According to an exemplary embodiment, when an area for receiving a query component is selected, the query input device may perform an operation of receiving the query component. For example, in order to receive a query component in which the query type is voice information, the query input device may be in a state of operating the microphone. The voice information may be information contained in the voice data. In operation S2910, the query input device may continuously or simultaneously receive a plurality of query components corresponding to a plurality of query types via the area of the query input window used to receive the query component. For example, when the user writes a red search when drawing a bicycle, the query input device may receive sketch information indicating the bicycle drawn by the user and voice information including the keyword "red" as a query component.

In operation S2920, the query input device may detect the query type of the received query component. For example, when the received query component is a file, the query input device can detect the query type of the query component according to the file extension. Alternatively, when the application is used to receive the query component, the query input device may detect the query type of the query component according to the type of application. As another example, when an image is captured by using a camera, the query input device may detect that the query type of the query component is an image. Or, when recognizing characters included in a picture by using optical character recognition (OCR), the text can be detected as a query type. When detecting the query type of the received query component, the query input device may display the received query component (or corresponding to the received query component) in the area displaying the received query component based on the detected query type Display item). For example, when the query type detected from the first query component is video, the query input device may display a preview of the video in the area where the first query component is displayed, and the query type detected from the second query component is In the case of text, the query input device may display keywords in the area where the second query component is displayed. Or, when the query type detected by the self-query component is voice information, the query input device may display the voice waveform included in the voice information or corresponding to the predetermined waveform and the text converted from the voice information in the area displaying the query component At least one. The query input device may repeatedly perform operations S2910 and S2920 to receive multiple query components, and may display the received query components to enable the user to check the query components.

When the query type is detected in operation S2920, the query input device may add a query type for query in operation S2930. When the query input device receives input corresponding to the search command, the query input device may generate a query including the added query type. The query input device may perform a search based on the query including the received query component and the detected query type. The query input device may display one or more search results as the search results.

When changing the query type, the method of receiving the query component can be changed. In addition, various methods for receiving query components can be provided for one query type. Therefore, the user interface provided to the user for inputting the query component may change or vary depending on the query type. 33 is a flowchart illustrating a procedure of receiving a query according to an exemplary embodiment.

Referring to FIG. 33, in operation S3310, the query input device may display a query input tool in the query input window. Here, the query input tool means a user interface provided to the user for inputting the query component. In operation S3310, the query input device may display the query input tool according to the query type included in the query input through the query input window. Examples of the query input tool displayed according to the query type will be described in detail with reference to FIGS. 34 to 40 below.

In operation S3320, the query input device may select at least one of a plurality of query input tools displayed in the query input window. Specifically, in operation S3320, the query input device may receive user input, and select a query input tool according to the received user input. The type or form of user input can vary. For example, the query input tool can be selected according to a query input tool displayed on the touch screen by a part of the human body, a stylus, or a mouse cursor, and clicking the displayed query input tool.

In operation S3330, the query input device may determine whether to execute the application program for receiving the query component based on the selected query input tool. For example, when the query input tool is selected as a text box, the query input device may determine that the query component can be directly received through the text box without executing a separate application. That is, when the separate application program is not executed, the query input device may receive the query component via the query input window in operation S3340.

When executing the separate application program for receiving the query component, the query input device may execute the application program corresponding to the query input tool in operation S3335. The application corresponding to the query input tool may be predetermined, or may be selected by the user from the application list. Therefore, the query input device may receive the query component by using the executed application in operation S3345.

34 to 40 are exemplary diagrams illustrating a method of receiving a query component based on the query type included in the query by using the query input tool displayed in the query input window. 34 to 40 are only examples for describing one or more exemplary embodiments, and it should be understood that one or more other exemplary embodiments are not limited thereto.

34 is an exemplary diagram illustrating a method of receiving a query component in which the query type is text according to an exemplary embodiment.

When the text mode is included in the query, the query input window may include a text box 3410 as illustrated in FIG. 34. When the text box 3410 is selected, a cursor 3411 indicating the position of the input text may be displayed in the text box 3410. Text can be entered via a keyboard, keypad, writing pad to receive handwritten text, and so on. In addition, the query input device may receive the copied text from the keyboard or clipboard via the text box 3410. The keywords retrieved from the input text can be used as query components in the search.

According to another exemplary embodiment, the query input device may acquire text from an image by using an OCR operation (for example, an image acquired by operating a camera). Furthermore, although a query input tool for receiving text input is provided as a text box 3410 containing a cursor 3411 above, it should be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, a query input tool for receiving text input may include a writing pad to receive handwritten text included in the query, and then be converted by, for example, OCR operation of a search engine, or via OCR The operation is converted into text by the query input device.

35 to 38 are exemplary diagrams illustrating a method of receiving an inquiry component in which an image of an inquiry type is according to an exemplary embodiment. In FIGS. 35 to 38, the query input device 3500 is described as a handheld device including a camera, but it should be understood that one or more other exemplary embodiments are not limited thereto.

When the query received through the query input window includes an image, the query input device 3500 may display a query input window 3510 that includes one or more tools for receiving the image. The one or more tools for receiving images may each include at least one of an image upload icon 3511, a photography icon 3512, and an image address input box 3513.

When the user 1 selects the image upload icon 3511, the query input device 3500 is operable to select the image file. For example, referring to FIG. 37, the query input device 3500 can execute a gallery application that displays a preview list of image files. When the user 1 selects at least one image 3520 from the displayed preview list by using the gallery application, the query input device 3500 can receive the selected image 3520 as a query component.

Referring to FIG. 37, when an image 3520 is selected by using the gallery application, the query input device 3500 can end, close, or hide the gallery application, and the selected image 3520 is displayed in the query input window 3510. Alternatively, according to another exemplary embodiment, the query input device 2500 may divide its display area. In this case, the query input device 3500 can display the execution screen of the gallery application in one of the multiple partitions, thereby enabling the user 1 to select an image.

When the user 1 selects the photography icon 3512, the query input device 3500 can execute an application 3530 that operates a camera for capturing an image, as illustrated in FIG. 38. When generating an image by using a camera, the query input device 3500 can receive the generated image as a query component, and as illustrated in FIG. 37, the image can be displayed in the query input window 3510.

According to another exemplary embodiment, the user 1 may input the address where the image is located by using the image address input window 3513. The image address may be an address indicating the location of the image (similar to a URL address), but it should be understood that one or more other exemplary embodiments are not limited thereto.

In FIG. 37, it illustrates that the image received as the query component is displayed in the query input window 3510, but it should be understood that the position where the image received as the query component is displayed may vary in one or more other exemplary embodiments.

In addition, the method of receiving the query (where the query type of the query component is video) can be implemented similarly to the method of receiving the image as the query component as described above.

When receiving images or videos as a query component, keywords can be obtained from images or videos by using image recognition or OCR operations. The search can be performed by using the acquired keywords. Alternatively, the query input device may compare its own image with the index to search for similar images.

39 is an exemplary diagram illustrating a method of receiving a query component in which the query type is sketch information according to an exemplary embodiment.

When the query type included in the query is sketch information, the query input window 3910 may include a sketch input tool 3911 for inputting sketch information. The sketch input tool 3911 according to the exemplary embodiment may include one or more icons for selecting at least one of a pen mode, a brush mode, a pen mode, a color or thickness of a line, and the like. User 1 can set the sketch input mode by using the sketch input tool 3911, and input sketch information to receive queries where the query type is sketch information by using touch input, mouse input, track pad input, gesture input, etc. Component area.

When receiving the sketch information, the received sketch information may be displayed in the query input window 3910. The received sketch information can be converted into keywords, and the keywords obtained from the sketch information can be used for searching. Alternatively, the search can be performed by using the sketch information itself.

FIG. 40 is an exemplary diagram illustrating a method of receiving a query component in which the query type is audio information according to an exemplary embodiment.

When the query type included in the query is audio information, the query input device may display a query input window 4010 including a tool for receiving audio information. The tool for receiving audio information may include, for example, at least one of an audio file upload button 4011 and an audio recording icon 4012.

When selecting the sound file upload button 4011, the query input device may display a file selection window 4021 for selecting a sound file. The user can select the voice file (ie, audio file) input as the query component by using the file selection window 4021.

Alternatively, when selecting the audio recording icon 4012, the query input device may operate the microphone (the image of which may be included in the query input window 4010), and execute an application 4022 that records audio (ie, voice information). The query input device can use the acquired voice information received by the executed application 4022 as a query component.

When inputting voice information, the query input device may display the waveform, voice spectrum, or file name of the voice information input to the query input window 4010.

The voice information received as a query component can be used for music search by comparing the voice's own waveform and index, or the keyword obtained through conversion using voice recognition can be used for search.

41 is an exemplary diagram illustrating a method of receiving a query including multiple query components having multiple query types and displaying search results according to an exemplary embodiment.

When the query type included in the query includes text and images, the query input window displayed in the query input device 4100 may include a first area 4111 for inputting text and a second area 4112 for inputting images. As illustrated in FIG. 41(a), when receiving the text "flower" and the image containing the shape of a dog as a query component, the text "flower" can be displayed as the first display item in the area 4111, and the received image can be Displayed as the second display item in area 4112.

Here, the user can manually select the first area 4111 to place the first area 4111 in a ready state for receiving text input, and can manually select the second area 4112 to place the second area 4112 in receiving image input Ready state. However, it should be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, after the user enters the first query component (eg, text) in the first area or in response to the user entering the first query component (eg, text) in the first area ), the second area 4112 may automatically enter a ready state to receive input of a second query component (eg, image). In this regard, the controller of the query input device 4100 may determine that the first ready state (ie, the first state) of the first query component that can receive the first query component from the first area 4111 is automatically switched to the second region 4112 that can receive the second query component The second ready state (ie, the second state). For example, if the second query component corresponds to an audio or voice information query type, the controller may perform control to automatically switch the query input device 4100 to automatically activate or operate a microphone to receive the second ready state of the second query component. In addition, if the second query component corresponds to the image query type, the controller can perform control to automatically switch the query input device 4100 to automatically start or operate the camera to receive the second query component or display an interface for selecting images to receive the first 2. Query the second ready state of the component.

When the ready state is automatically switched as described above, the controller may control to output an indicator of the automatic switch and/or the second ready state. For example, the indicator may include an audio indicator or output (for example, a predetermined notification sound), a visual indicator or output (for example, a predetermined icon, a predetermined symbol, a predetermined image, etc.), an auxiliary device output (for example, a query input device 4100) at least one of blinking LEDs, predetermined color LEDs, vibration output, etc. In this case, in various exemplary embodiments, the visual indicator may be displayed in the query input window or outside the query input window.

In addition, the controller may determine to automatically switch from the first ready state to the second ready state based on determining that the input of the first query component is completed. For example, if the first query component is an image, the controller may determine to automatically switch from the first ready state to the second ready state in response to the image captured by the camera of the query input device 4100 or in response to the image . In addition, if the first query component is text or a sketch, the controller may determine to respond to a predetermined time period elapsed from the last user input to the first area 4111 (for example, when inputting the last text character to the first Two seconds after zone 4111) Automatically switch from the first ready state to the second ready state.

When switching to the second ready state, the controller may also control to change the display of the query input window, the display of the second area 4112, or the display of the graphical user interface. In this regard, the change in the display may be based on the query type corresponding to the second query component or the second area 4112. According to another exemplary embodiment, the controller may control to display the second area 4112 or the user interface in response to switching to the second ready state to receive input from the second query component.

When the user 1 selects the search button 4120 in the state where the received text and the received image are displayed, as illustrated in FIG. 41(b), the query input device 4100 can display the search obtained by performing the multimodal search result. In FIG. 41, images including both flowers and dogs are displayed as search results.

FIG. 42 is a flowchart illustrating a procedure for performing search according to an exemplary embodiment.

The query input device according to the exemplary embodiment may receive a query in operation S4210, and determine the priority of the query component included in the received query in operation S4220. In operation S4220, the query input device may determine the priority of the query component based on the query type of the query component. For example, when text, image, and voice information are included in the query, the priority of the query component can be determined in the order of text, voice information, and image.

The priority of the query component can be determined differently. For example, the user can set the priority of multiple query types. According to an exemplary embodiment, in setting the priority of the query type, the priority may be determined based on the query type of the received query component. According to another exemplary embodiment, the priority of the query components may be determined based on the order in which the query components are received via the query input window. However, this situation is merely an example for describing an exemplary embodiment, and it should be understood that one or more other exemplary embodiments are not limited thereto.

In operation S4230, the search may be performed based on the determined priority. For example, the first search can be performed based on the text included in the query, and then by using the voice information of the query component having a lower priority than the text, the results of the first search based on the text can be performed Perform a second search.

The priority of the query component can be determined by the query input device. Alternatively, when performing a search by using a search engine server including a search engine, the search engine server may determine the priority of the query component.

FIG. 43 is an exemplary diagram illustrating a result obtained by performing a multi-modal search according to an exemplary embodiment.

According to an exemplary embodiment, the query input device may display a query input window 4201. The query input window 4201 may include at least one query component input area that displays the received query component (that is, the display item corresponding to the received query component). According to an exemplary embodiment, the query input device may display the received query component based on the priority of the query type of the received query component. Referring to FIG. 43, when the priority of the image query type is higher than the priority of the text query type, the image input area 4202 may be displayed above the text input area 4203.

By inputting a query component using the query input window 4201, and then when the user selects the search button 4204, the results of the search performed by using the query component can be displayed in the search result display area 4205. Here, among the plurality of query components, the query component with the highest priority may be determined as the primary query component, and the query component with the lowest priority may be determined as the secondary query component. Among the search results based on the primary query component, the information selected by the secondary query component can be displayed in the search result area. For example, referring to FIG. 43, among the pieces of information about the smart phone included in the image, advertisement information including product information of the smart phone, blog, and image may be displayed in the search result area 4205.

44 is a flowchart illustrating a procedure of displaying search results according to another exemplary embodiment.

In operation S4310, a plurality of search results may be acquired (ie, determined or obtained) based on the query received via the query input window. Here, the obtained multiple search results may be prioritized in operation S4320. For example, the obtained multiple search results can be prioritized based on the degree of matching query. As another example, the priority of the acquired multiple search results may be determined based on the time when the information containing the corresponding search results is generated.

In operation S4330, the prioritized search results may be displayed in the query input window based on their priority. Here, the search result may be displayed in another device instead of the query input device. For example, the search result may be displayed on another device included in the home network connected to the query input device.

FIG. 45 is an exemplary diagram illustrating the priority of search results according to another exemplary embodiment. According to an exemplary embodiment of the present invention, search results may be prioritized based on their type. The priority of the search results can be set by the user, or can be determined based on the user's feedback or usage history. For example, when the user does not pay attention to the advertisement, the user can directly set the search result type of the advertisement to a low priority, or when the number of times of selecting advertisement information is relatively small (or lower than a predetermined value), the query input device can Set the ad search result type to a relatively low priority. For example, referring to FIG. 45, the priority based on the result type can be determined in the order of images, blogs, news, web files, and advertisements. Herein, it is described that the query input device determines the priority of the search results. However, it should be understood that one or more other exemplary embodiments are not limited thereto. For example, according to another exemplary embodiment, when a search is performed by using an external search engine server, the priority of the search results may be determined by the external search engine server.

46 is an exemplary diagram illustrating a result obtained by performing a multi-modal search according to another exemplary embodiment. In the exemplary embodiment of FIG. 46, the same query as that of the exemplary embodiment of FIG. 43 is input. A query input window 4201 and a search button 4204 including an image input area 4202 and a text input area 4203 may be displayed.

In addition, the exemplary embodiment of FIG. 46 shows a state where the priority of search results is set as explained in FIG. 45. Even when the same query is input, the search results may be displayed differently depending on the priority of the search results. That is, the search results can be displayed in the result display area 4205-1 in the order of images, blogs, and news based on the priority of the search results.

47A is an exemplary diagram illustrating a method of receiving a query by using multiple devices according to an exemplary embodiment. According to an exemplary embodiment of the present invention, a query component included in a query can be input by using multiple devices. In FIG. 47A, it is illustrated that the query input device 4400 is a networked TV, but it should be understood that in one or more other exemplary embodiments, the query input device 4400 is not limited to the form illustrated in FIG. 47A and may be changed to Implemented as a PC, smart phone, smart device, laptop, portable multimedia player, tablet computer device, mini-notebook computer, workstation, etc.

When the query types included in the query are sketch information, voice information, and image, the query input device 4400 may display a query input window 4410 for entering sketch information, voice information, and image.

The query input device 4400 may request voice information 4411 from a smart watch 4401 that can communicate with the query input window 4401. The smart watch 4401 may represent an embedded system watch equipped with various operations in addition to the operation of a universal clock. For example, the smart watch 4401 can perform calculation operations, translation operations, recording operations, communication operations, and the like. The smart watch 4401 receiving the request for the voice information 4411 from the query input device 4400 may operate the microphone included in the smart watch 4401 to generate voice information 4411, and transmit the generated voice information 4411 to the query input device 4400. The query input device 4400 can receive the voice information 4411 transmitted from the smart watch 4401 as a query component. Smart watch 4401 can be used by wired communication or wireless such as Bluetooth, Wi-Fi direct connection, near field communication (NFC), infrared data association (IrDA), radio frequency (RF) communication, wireless local area network (LAN), etc. Communicate with the query input device 4400.

In addition, the query input device 4400 may request an image 4412 from the smart glasses 4402 that can communicate with the query input device 4400. The smart glasses 4402 represent a wearable device equipped with a head-mounted display (HMD). The smart glasses 4402 can perform calculation operations, translation operations, recording operations, communication operations, and the like. The smart glasses 4402 receiving the request for the image 4412 from the query input device 4400 may generate the image 4412 captured by the camera included in the smart glasses 4402. The smart glasses 4402 can transmit the generated image 4412 to the query input device 4400. The query input device 4400 can receive the image 4412 transmitted from the smart glasses 4402 as a query component. The smart glasses 4402 can communicate with the query input device 4400 by using wired communication or wireless communication such as Bluetooth, Wi-Fi Direct, NFC, IrDA, RF communication, wireless LAN, and so on.

The smart glasses 4402 may include a camera for tracking the user's eyes. When the user watches a certain part of the entire screen of the TV, the smart glasses 4402 can determine the area currently viewed by the user by using the camera for tracking the user's eyes, and can transmit the image of the area to the inquiry input device 4400.

47B is an exemplary diagram illustrating a method in which smart glasses (for example, smart glasses) according to an exemplary embodiment acquire an image of an area viewed by a user from the entire screen of a TV. Referring to FIG. 47B, the smart glasses 4402 can acquire images at a position viewed by the user. FIG. 47B shows an exemplary embodiment when a user views a bicycle displayed on a TV screen.

The pupil tracking camera 4405 can track the direction of the user's pupil to determine the direction the user's eyes are looking at. The processor included in the glasses (eg, glasses) can receive information indicating the viewing direction of the user from the pupil tracking camera 4405, and adjust the direction of the front camera 4403 based on the received information. Therefore, the front camera 4403 can be synchronized with the user's viewing direction. That is, the direction of the front camera 4403 is the same as the direction seen by the user's eyes. When adjusting the direction of the front camera 4403, the front camera 4403 can output the image with the adjusted direction to the processor.

The front camera 4403 can acquire images located in the viewing direction tracked by the pupil tracking camera 4405. In other words, the front camera 4403 can capture images in the same direction as the user's viewing direction. In FIG. 47B, the situation where the front camera 4403 points to the bicycle on the TV screen will be described.

The smart glasses 4402 can acquire the bicycle image on the TV screen, and transmit the bicycle image to the query input device 4400. The query input device 4400 can receive the bicycle image transmitted from the smart glasses 4402, and receive the query including the bicycle image, thereby performing the search.

In addition, the query input device 4400 can directly receive sketch information by using the query input tool output to the query input device 4400. The query input device 4400 with the received voice information, image, and sketch information may perform a search based on the query including voice information, image, and sketch information.

In one or more other exemplary embodiments, the smart watch 4401 and smart glasses 4402 of FIGS. 47A and 47B may be used to replace other devices that communicate with the query input device 4400.

FIG. 48 is an exemplary diagram illustrating a method of receiving a query component by using multiple devices according to another exemplary embodiment.

According to an exemplary embodiment of the present invention, the query input device 4800 may receive a query component added to or included in a query by using a device connected to the query input device 4800 via a network. Referring to FIG. 48, the query input device 4800 may search for a plurality of devices 4801 to 4803 connected to the query input device 4800 via the gateway 4810 in order to select one or more devices that provide query components. Here, the query input device 4800 may select a device that provides the query component based on the query type of the query component. For example, when the query type of the received query component is voice information, the query input device 4800 may select a device including voice input operations from among the devices 4801 to 4803.

According to an exemplary embodiment, the query input device 4800 may broadcast a request for a query component. The query input device 4800 may respond to the request from the respective device profiles of the devices 4801 to 4803 and receive the devices 4801 to 4803. Here, each of the device profiles may include information about one or more operations provided by the corresponding device. The query input device 4800 may select a device that provides a query component based on the received device profile.

Alternatively, when the gateway 4810 manages the device profile of the device connected to the gateway 4810, the gateway 4810 may respond to the request to select the device that provides the query component. When the gateway 4810 selects the device that provides the query component, the gateway 4810 may transmit the request for the query component to the selected device.

According to another exemplary embodiment, the query input device 4800 may broadcast a request containing information about the type of query. The devices 4801 to 4803 that have received the request containing information about the query type may determine whether it is possible to provide a query component with the query type included in the request. Among the devices 4801 to 4803, a device that provides a query component having the query type included in the request may transmit a response to the request to the gateway 4810 or to the query input device 4800.

When two or more devices are selected by the query input device 4800 or the gateway 4810, the query input device 4800 may display a list of selected devices. The user can select the device to input the query component from the displayed list of the device.

FIG. 49 is a flowchart illustrating a procedure for performing search according to an exemplary embodiment.

Referring to FIG. 49, when receiving a query in operation S4510, the query input device according to an exemplary embodiment of the present invention may determine whether the search mode when receiving the query is a multi-modal input mode in operation S4520. Here, operation S4510 indicates that the query input device receives the query input to the query input window according to the search performance command. When the search mode is determined to be a single-mode search in operation S4520, the query input device may perform a search based on the query type included in the query in operation S4530.

When the search mode is determined to be the multi-modal input mode in operation S4520, the query input device may generate a combined query based on a plurality of query components in operation S4525. Here, combined query means combining query components with multiple query types. According to one or more exemplary embodiments, query components may be combined differently. For example, query components can be simply combined. In detail, for example, when inputting the text "handbag" and the voice "price" as query components, the query can be composed of the keyword "handbag price". As another example, when a user draws two wheels on a query input window with a touch pen and speaks a bicycle to input a voice signal (that is, an audio signal), the query may consist of the keyword "bicycle with two wheels" or Contains the keyword "bicycle with two wheels". As another example, when a user draws an apple on the query input window with a touch pen and speaks red to input a voice signal (that is, an audio signal), the query may consist of the keyword "red apple" or include the keyword "red apple" ". As another example, when the user photographs the handbag with the camera and enters the image of the photographed handbag on the query input window, the query may consist of or include the keywords "3000 won handbag" or "3000 dollar handbag" The keyword "3000 won handbag" or "3000 dollar handbag". As another example, the combined query may include keywords or main features (eg, features included in the image) added to the query component. In addition, as another example, the combined query may include the extension keyword generated from the query component. In addition, as another example, a combined query may be characterized by prioritizing query components based on the priority of the query type. In operation S4525, the query input device may retrieve relationships between multiple query types included in the query, and generate a combined query based on the retrieved relationships. In operation S4535, the query input device may perform a search based on the combined query generated in operation S4525.

Here, operations S4525, S4535, and S4530 can be performed by an external server instead of an inquiry input device.

FIG. 50 is a flowchart illustrating a procedure for performing search according to an exemplary embodiment.

In operation S4610, the query input device 400 according to an exemplary embodiment of the present invention may receive a query via the displayed query input window.

In operation S4620, the query input device 400 may transmit the received query to the search engine server 420. Here, when the search mode is a multi-modal search, the query transmitted to the search engine server 420 may be a combined query in which query components having multiple query types are combined. According to one or more exemplary embodiments, query components may be combined differently. For example, query components can be simply combined. In detail, for example, when inputting the text "handbag" and the voice "price" as query components, the query can be composed of the keyword "handbag price". As another example, the combined query may include keywords or main features (eg, features included in the image) added to the query component. In addition, as another example, the combined query may include the extension keyword generated from the query component. In addition, as another example, a combined query may be characterized by prioritizing query components based on the priority of the query type. According to an exemplary embodiment, the query transmitted to the search engine server 420 may include information indicating the search mode.

In operation S4630, the search engine server 420 may perform a single search or a multi-modal search according to the search mode for processing the received query. In operation S4640, the search engine server 420 may transmit the search result selected in S4630 to the query input device 400.

FIG. 51 is a block diagram illustrating the structure of a query input device 4700 according to an exemplary embodiment.

The query input device 4700 according to the exemplary embodiment may include a display 4710, a controller 4720, and an input device 4730 (for example, an input unit).

The display 4710 may display a query input window. The display 4710 can display various pieces of information other than query input. The query input device 4700 may include two or more displays 4710 depending on the type of implementation. The display 4710 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, a cathode ray tube (CRT) display, a plasma display panel (PDP), an organic LED (OLED) display, an active matrix OLED (AMOLED) Display devices such as displays and thin-film transistor (TFT) displays. Also, the display 4710 may include a touch sensor and a touch screen with a layered structure depending on the implementation type. When the display 4710 performs a display operation such as a touch screen and an input operation, the display 4710 can perform the operation of the input device 4730, and/or the input device 4730 can be implemented as the display 4710. Also, the input device 4730 may be implemented to include the display 4710.

According to an exemplary embodiment, when the search mode is a multi-modal search mode, the display 4710 may display a query input window. The query input window represents an interface through which queries containing multiple query types are received. Alternatively, when the search mode is a single search mode, the query input device 4700 may display a single query input window. A single query input window represents a query input window via which queries containing only one query type are received.

The input device 4730 may receive a query including a plurality of query components corresponding to a plurality of query types via a query input window displayed by the display 4710 based on user input. The input device 4730 can receive query components such as text or sketch information by using a keyboard, keypad, virtual keypad, track pad, writing pad, and the like. Alternatively, the query input device 4700 may receive a query component such as images, voice information, or video to obtain or generate a query based on user input.

The controller 4720 may control elements of the query input device 4700. The controller 4720 may include a central processing unit (CPU), a read-only memory (ROM) storing control programs, and a signal or data input from outside the inquiry input device 4700 or used as a working memory performed by the inquiry input device 4700 Random Access Memory (RAM) of the area. The CPU may include one or more processors such as single core, dual core, triple core, or quad core. The CPU, ROM and RAM can be connected to each other via an internal bus.

The controller 4720 may obtain at least one search result of the received query. When the controller 4720 includes a search engine, the controller 4720 may directly select at least one search result of the query. When the controller 4720 does not include a search engine, the controller 4720 may transmit the query to the search engine server including the search engine to obtain at least one search result from the search engine server. The controller 4720 may control the display 4710 to display at least one acquired search result. The display 4710 may display the acquired at least one search result according to the control of the controller 4720.

In addition, the controller 4720 may select multiple query types to be selected or used for the query. The query input window displayed by the display 4710 can receive query input according to the selected query type. Also, as in the exemplary embodiment described above, the query input window displayed by the display 4710 may be displayed differently depending on the selected query type.

In addition, according to an exemplary embodiment, the display 4710 may display a search mode selection object for selecting a search mode. The input device 4730 can receive user input of the search mode selection object. Here, when the user inputs the input to switch the search mode to the multi-modal input mode, the controller 4720 may switch the search mode to the multi-modal input mode. When switching the search mode to the multi-modal input mode, the display 4710 may change the query input window to a query input window containing multiple input modes. According to an exemplary embodiment, the query input window may include areas corresponding to multiple query types, respectively.

Also, the display 4710 may display a list of query types. According to an exemplary embodiment, when the search mode is a single mode, the display 4710 may display a single query type list, and when the search mode is the multi-modal input mode, the display 4710 may display a multi-modal query type list. The controller 4720 may determine at least one query type included in the query from the list of query types displayed on the display 4710. Here, in order to determine at least one query type, the input device 4730 may receive input that drags and drops the target from the query type list to the area where the query input window is displayed or will be displayed.

In addition, according to an exemplary embodiment, the controller 4720 may detect the query type of the received query component. The query input window displayed by the display 4710 may include an area that displays the received query component (that is, a display item corresponding to the received query component) and an area that receives the query component. The display 4710 may display the received query component in the area corresponding to the received query component based on the detected query type.

In addition, the query input window displayed by the display 4710 may include at least one query input tool for inputting query components corresponding to each query type.

Also, the controller 4720 may convert the text included in the search results into speech, or convert speech (ie, audio) into text.

According to an exemplary embodiment, after the user inputs the first query input to the query input window or in response to the user inputting the first query input to the query input window, the controller 4720 may determine that the first query component can be received from The first ready state (ie, the first state) is automatically switched to the second ready state (ie, the second state) that can receive the second query component. For example, if the second query component corresponds to an audio or voice information query type, the controller 4720 can control to automatically switch the query input device 4700 to automatically activate or operate a microphone to receive the second ready state of the second query component. In addition, if the second query component corresponds to the image query type, the controller 4720 can control to automatically switch the query input device 4700 to automatically start or operate the camera to receive the second query component or display an interface for selecting images to receive the first 2. Query the second ready state of the component.

When the ready state is automatically switched as described above, the controller 4720 may control to output an indicator of the automatic switch and/or the second ready state. For example, the indicator may include an audio indicator or output (for example, a predetermined notification sound), a visual indicator or output (for example, a predetermined icon, a predetermined symbol, a predetermined image, etc.), an auxiliary device output (for example, a query input device At least one of the flashing LED on the 4700 or the LED of a predetermined color), vibration output, etc.

In addition, the controller 4720 may determine to automatically switch from the first ready state to the second ready state based on determining that the input of the first query component is completed. For example, if the first query component is an image, the controller 4720 may determine to automatically switch from the first ready state to the second ready in response to the image captured by the camera of the query input device 4700 or in response to the image status. In addition, if the first query component is text or a sketch, the controller 4720 may determine to respond to a predetermined time elapsed from the last user input in the area of the query input window corresponding to the first query component or the first query type The cycle is automatically switched from the first ready state to the second ready state.

When switching to the second ready state, the controller 4720 can also control to change the display of the query input window, the display of one or more areas of the query input window, or the display of the graphical user interface. In this regard, the displayed change may be based on the query type corresponding to the second query component. According to another exemplary embodiment, the controller 4720 may control to display a user interface in response to switching to the second ready state to receive input from the second query component.

It should be understood that the query input window may vary and is not limited to those described above.

52 to 57 are exemplary diagrams illustrating a method of performing multi-modal search according to another exemplary embodiment.

Referring to FIG. 52, the query input device 5200 can receive, generate, or obtain a query "bicycle", and displays search results 5221 including encyclopedia information, web page information, and images based on the query search. Here, the user 1 can scroll the screen displayed by the display to check search results not currently displayed on the display.

Referring to FIG. 53, the screen is scrolled, and therefore, a plurality of search results 5222 that were not previously displayed by the display can be displayed by the display. In this case, because the screen is scrolled, the query input window 5210 may not be displayed on the screen.

User 1 may select the image 5231 included in the search result 5222. For example, referring to FIG. 54, the user 1 can select the bicycle image 5231 from the search results 5222. Here, the user 1 can select the 5231 by using a touch screen or a mouse, but it should be understood that one or more other exemplary embodiments are not limited thereto. When selecting an image 5231, the query input device 5200 may display a list of query types near the selected image 5231. Referring to FIG. 54, the query input device 5200 can display a floating icon 5230 near the selected image 5231. The floating icon 5230 may correspond to at least one query type.

Referring to FIG. 55, the user 1 can select the displayed floating icon 5230. Referring to FIG. 56, the floating icon 5230 is selected, and therefore, the query input device 5200 may display a query input area 5231 for inputting a query type corresponding to the floating icon 5230. The query input device 5200 may receive the query component via the query input area 5231.

Referring to FIG. 57, the query input device 5200 may perform a search based on the selected search result and the query 5210-1 including the query component received via the query input area 5231. As illustrated in FIG. 57, based on the bicycle image and the text “manufacturer”, the query input device 5200 can display the bicycle manufacturer’s homepage included in the image and information about other products manufactured by the bicycle manufacturer in the result display area 5223 . That is, the query input device 5200 receives the search result selected from the search result display area, and receives each query component by using the query input area 5231, and therefore, in the exemplary embodiment of FIG. 56, it can be described as The query component is received by using the search result display area and the query input area 5231 as a query input window.

FIG. 58 is an exemplary diagram illustrating a method of displaying a query type list according to another exemplary embodiment.

According to an exemplary embodiment of the present invention, the query input device may display a single mode selection object 5801, a multi-modal input mode selection object 5802, a query input window 5810, a search button 5830, and a single query type list 5820-1. As illustrated in FIG. 58, a single query type list 5820-1 may include multiple icons corresponding to multiple query types (each icon corresponds to a query type).

The query input device may determine the query type by using the icons included in the single query type list 5820-1, through which the query input window 5810 is received. For example, when the text icon 5821-1 is selected, the query input device may display a user interface for inputting text into the query input window 5810. Alternatively, when selecting the image icon 5821-2, the query input device may display a user interface for inputting an image to the query input window 5810. In addition, when the file icon 5821-3 is selected, the query input device may display a user interface for inputting the file to the query input window 5810. In addition, when selecting the sketch icon 5821-4, the query input device may display a user interface for inputting sketch information to the query input window 5810. Also, when the camera icon 5821-5 is selected, the query input device can execute the camera application, and the image obtained by using the camera application is displayed in the query input window 5810. In addition, when selecting the music icon 5821-6, the query input device may perform a music search based on the voice information acquired by using the microphone. Alternatively, when selecting the recording icon 5821-7, the query input device can operate the microphone and obtain voice information by using the microphone.

FIG. 59 is an exemplary diagram illustrating a method of displaying a list of query types according to another exemplary embodiment.

When the multi-modal input mode selection object 5802 is selected, the query input device may set the search mode to the multi-modal input mode. According to an exemplary embodiment of the present invention, when the search mode is the multi-modal input mode, the query input device may display a multi-modal query type list 5820-2. In this case, the multi-modal query type list 5820-2 may include combined icons that combine multiple icons. The combination icon can indicate the multimodal query type by using the icons included in the combination icon. Also, according to an exemplary embodiment, the query input device may indicate the priority of the query type based on the display of the icon. For example, the combined icon 5822-1 of the video and text illustrated in FIG. 59 indicates that the priority of the video is higher than the priority of the text. Also, the combined icon of text and speech 5822-2 indicates that the priority of text is higher than that of speech. Also, the combined icon of camera, voice, and text 5822-3 indicates that the priority of the query type is in the order of camera, voice, and text.

FIG. 60 is an exemplary diagram illustrating a method of generating a multimodal query type list according to an exemplary embodiment.

According to an exemplary embodiment, the query input device may generate a plurality of combined icons corresponding to the plurality of query types by using a plurality of icons 6010 respectively corresponding to the plurality of query types (each icon corresponds to a query type) 6020. For example, when the user selects the text icon 6011 and the image icon 6012 from the icons 6010, the query input device may generate a combined icon 6021 that combines the text icon 6011 and the image icon 6012. In this case, the user can also set the priority of the query type (for example, according to the order of selecting icons or by modifying the combination icon 6021), and the combination icon 6021 can reflect the setting priority. Alternatively, the priority may be preset or predetermined.

According to an exemplary embodiment, when the search mode is the multi-modal search mode, the query input device may display a list of multi-modal query types including the combined icon 6020 set by the user.

61 to 65 are exemplary diagrams illustrating a method of performing multi-modal search according to another exemplary embodiment.

According to an exemplary embodiment of the present invention, the query input device 6100 may display an indicator 6102 indicating the search mode as a multi-modal input mode. In addition, the query input device 6100 may display a query input window 6110 and a search button 6130.

In addition, according to an exemplary embodiment of the present invention, the query input device 6100 may display a set of query input tools 6120 for inputting query components when the query input window 6110 is selected. Alternatively, according to another exemplary embodiment, the query input device 6100 may display a set of query input tools 6120 for inputting a query component when a menu button included in the query input device 6100 is selected. The query input tool may be included in the query input window 6110.

As illustrated in FIG. 61, when the user 1 selects the file opening icon 6121 from the query input tool set 6120, the query input device 6100 may display a user interface for selecting a file. FIG. 62 is an exemplary diagram illustrating a user interface for selecting files.

As illustrated in FIG. 62, when the user 1 selects the file 6122, the query input device 6100 may receive the selected file 6122 as a query component. Also, the query input device 6100 can detect the query type corresponding to the selected file 6122. For example, as illustrated in FIG. 62, when the extension of the selected file 6122 is jpg, the query input device 6100 may determine the query type corresponding to the selected file 6122 as an image.

The query input device 6100 may display the received query component (that is, the display item corresponding to the received query component) according to the detected query type. For example, referring to FIG. 63, the query input device 6100 may display the received image 6111 in the area 6103 displaying the query component. In FIG. 63, it is illustrated that the area 6103 displaying the query component is combined with the query input window 6110 including the area receiving the query component, but it should be understood that the form of the area 6103 displaying the query component may be implemented in one or more other exemplary embodiments Case changes. For example, the area 6103 may be included in the query input window 6110, or may be disposed in a separate space. When receiving multiple query components continuously or simultaneously via the query input window 6110, the received multiple query components may be accumulated or displayed together in the area 6103, or displayed in a separate area.

In addition, the query input device 6100 may receive additional query components via a region that receives the query components included in the query input window 6110. For example, referring to FIG. 64, the user 1 can input the text "Obama" into the query input window 6110.

When the user 1 selects the search button 6130, the query input device 6100 may perform a search based on the accumulated query components and the detected query type. For example, referring to FIGS. 64 and 65, the query input device 6100 may display the video 6140 of Obama giving a speech as a search result based on the image 6102 of the speaking scene and the text 6113 "Obama".

66 is a block diagram illustrating a framework for performing multimodal search based on a query having multiple query types according to an exemplary embodiment. In detail, FIG. 66 is a block diagram illustrating a framework for query adaptive multimodal search.

The query interface 6600 may receive a combined query 6610 that combines multiple query components. The plurality of query components may include at least one of keywords 6611, images 6612, video 6613, voice 6614, sketch information 6615, context information 6616, and the like. Here, the context information 6616 indicates information that clarifies the query in detail, which is similar to the user's current status or personal history and preference information. For example, the context information 6616 may include the priority of the query type.

In addition, the query interface 6600 may include a unit or device for receiving a combined query 6610. For example, the query interface 6600 may include a keyboard for receiving keywords 6611, a camera for acquiring images 6612 or video 6613, a microphone for acquiring voice 6614, a touch screen for acquiring sketch information 6615, At least one of a sensor or the like for acquiring context information 6616.

The search method 6620 represents an algorithm to match the query with the database in order to select files depending on their suitability. For example, in the video search system, although the segmentation search method is processing query text keywords and matching the query text keywords with voice recognition information, the video thumbnail images can be matched with visual content by a single search method. The combined query 6610 can be processed by multiple search methods 6620 to obtain search results.

The database matching the query may contain a collection of files 6630. The database contains pieces of information to be searched. The files contained in the database may have different modalities. Each of the documents represents the unit of information contained in the database. For example, each document may include a page on a web page, a screen in a video corpus, or an image in a photo collection.

The query adaptation module 6640 can adjust the processing order of the search method 6620 for processing queries. For example, when it is desired to search for photos of celebrities in news videos, the weight can be given to the text search method, but when it is desired to search for sports scenes, the weight can be given to the example-based image search method.

The search method merging module 6650 may merge search results obtained by multiple search methods 6620. The combined search results can be output through the search result output module 6660.

At least one of the search method 6620, the database, the query adaptation module 6640, the search method merge module 6650, and the search result output module 6660 can be applied to an external device. The external device may be a cloud computer or a server.

An external device (for example, a cloud computer) may store the search method 6620 and the database, and may include a query adaptation module 6640, a search method merge module 6650, and a search result output module 6660. The external device can perform a search by using the query received from the query input device, and output the search result to the query input device.

67 is a block diagram schematically illustrating a system and/or processor according to one or more exemplary embodiments. Referring to FIG. 67, when the search mode is a multi-modal search mode, the query input device may perform a search based on the query type of the received query component. For example, when the query input device receives the first query component 6705 of the text query type and the second query component 6707 of the image query type, the query input device may be based on the key corresponding to the first query component 6705 (ie, text) The word and the input amount (ie, image) of the second query component 6706 perform the search. In order to process the second query component 6707, the query input device can identify the characteristics of the second query component 6707 by using the second query component understanding component 6712. The second query component understanding component 6712 may be referred to as a second query component processing component. Alternatively, when the query type of the second query component 6707 is an image, the second query component understanding component 6712 may be referred to as an image processing component. The features retrieved from the second query component 6707 by the second query component understanding component 6712 can be assigned descriptor keywords by the second query component feature component 6722. For example, when the query type of the second query component 6707 is the image query type, the second query component understanding component 6712 can identify the feature (ie, image) of the second query component 6707. In this case, the features of the second query component 6707 may include at least one of image text features and image visual features. In addition, the second query component feature component 6722 may assign a description keyword to at least one of the image text feature and the image visual feature of the identified second query component 6707. An example in which the query type of the second query component 6707 is the image query type will be described in detail with reference to FIGS. 68 to 72.

The metadata analysis component 6714 can identify the metadata associated with the second query component 6707. When the second query component 6707 includes a file, the metadata may be built into the file by the operating system (OS) (similar to the name or comment stored in the file), or may include information stored along with the file. Metadata may include input for identification in a URL path or related text such as text located in a webpage or text-based document or in the vicinity of corresponding information of information built therein (eg, an image or the like) The text of the query component to be used for searching. The second query component feature component 6722 can identify keyword features based on the output of the metadata analysis component 6714.

The second query component feature component 6722 may identify the first query component 6705 and any additional features, and then the resulting query may be changed or expanded by component 6732 as the case may be. The query change or expansion can be performed by the metadata analysis component 6714 and the second query component feature component 6722 based on the features retrieved from the metadata. Alternatively, the query change or expansion may be performed based on the feedback received by using the UI interactive component 6762. Also, in addition to the additional query input received by the user, the feedback may include query suggestions 442 based on the response results of the current or previous queries. Also, queries that expand or change as appropriate can be used to generate (6752) response results. In FIG. 67, the result generation operation (6752) can be identified by using one or more query response files in a database 6675, which contains the characteristics and quantities of data fragments (eg, files). The database 6775 may contain an inverted index or another arbitrary type of storage format for identifying query-based response results.

According to one or more exemplary embodiments, the result generating operation 6762 may generate one or more types of results. Depending on the situation, the most promising results may be identified along with higher priority result responses or a few higher priority result responses. Promising results can be provided in response to 6744. As an alternative, a list of prioritized response results may be used. A list can be provided by prioritizing multiple combined results 6746. The interaction with the user (including the operation of displaying the result and the operation of receiving the query component) can be performed by the UI interactive component 6762.

68 to 72 are exemplary diagrams illustrating a procedure for capturing image features of a query component having an image query type according to one or more exemplary embodiments.

In FIG. 68, a method of processing an image (that is, a query component having an image query type) 6800 by using an operator algorithm for identifying a plurality of points of interest (POI) 6802 is described. The operator algorithm may include any available algorithm for identifying the POI 6802 in the image 6800. In one or more exemplary embodiments, the operator algorithm may be different from the Gaussians algorithm or the Laplace algorithm. In one or more exemplary embodiments, the operator algorithm may analyze the image 6800 in two dimensions. As the case may be, when the image 6800 is a color image, the image 6800 may be changed to grayscale.

In addition to any points of the image 6800 as illustrated in FIG. 68, the POI 6802 may also include sections 6902, regions, pixel groups, and features in the image 6800 as illustrated in FIG. 69. Hereinafter, POI 6802 and multiple sections 6902 may be referred to as POI 6802, but POI 6802 may include all POI 6802 and sections 6902. In one or more exemplary embodiments, POI 6802 may include special or recognizable features of image 6802 located in the stable area of image 6802. For example, POI 6802 may be located in the area of image 6800 with steep features that represent high contrast between the features illustrated in 6802a and 6902a. On the other hand, the POI may not be located in a section of a certain color or gray scale indicated by 6804 (that is, an area without special characteristics or contrast).

The operator algorithm can, for example, identify any number of POIs 6802 in the image 6800, such as thousands of POIs. POI 6802 may be a combination of point 6802 and segment 6902 in image 6800, and the number of POIs may change or vary depending on the size of image 6800. The second query component processing component 6712 may calculate the metrics for each of the POI 6802, and prioritize the POI 6802 according to the calculated metrics. The measurement may include the signal strength or signal-to-noise ratio (SNR) of image 6800 in POI 6802. The second query component processing component 6712 can select a subset of POI 6802 for additional processing based on priority. According to one or more exemplary embodiments, hundreds of POI 6802 with the highest SNR may be selected. The number of selected POI 6802 can be changed or varied. According to one or more exemplary embodiments, a subset may not be selected, and all POIs may be targets for additional processing.

As illustrated in FIG. 70, a set 7000 of patches 7002 corresponding to the selected POI 6802 can be identified. Each of the patches 7002 may correspond to the selected single POI 6802. The patch 7002 may include a region of the image 6800, which includes the POI 6802. The size of the patch 7002 acquired from the image 6800 may be determined based on the output of the operation algorithm from each of the selected POI 6802. The patch 7002 may have different sizes. Also, the regions of the image 6800 included in the respective patches 7002 can overlap each other. The shape of the patch 7002 may be any shape including a square shape, a rectangular shape, a triangular shape, a circular shape, an elliptical shape, and the like. The shape of each of the patches 7002 may be a square shape or may be different between the patches.

The patch 7002 as illustrated in FIG. 70 can be normalized. According to one or more exemplary embodiments, the patch 7002 may be normalized to have the same size. The patch 7002 can be normalized by increasing or decreasing at least one of its size and resolution. However, the exemplary embodiments of the present invention are not limited thereto. For example, the patch 7002 may be normalized via one or more other operations such as application of contrast increasing operation, application of light spot removing operation, application of sharpening operation, and application of grayscale.

According to one or more exemplary embodiments, the descriptor may be determined for each of the normalized patches. The descriptor may be a description of a patch that can be added as a feature for image search. The descriptor can be determined by calculating the statistics of the pixels in each of the tiles 7002. According to one or more exemplary embodiments, the descriptor may be based on statistical determination of the grayscale slope of the pixels in each of the tiles 7002. The descriptor may be visually expressed as a histogram for each of the patches 7002, which is similar to the multiple descriptors 7102 illustrated in FIG. 71 (the patch 7002 illustrated in FIG. 70 may correspond similarly (Descriptor 7102 in FIG. 71). However, the exemplary embodiments of the present invention are not limited thereto. For example, the descriptor may be described as a multi-dimensional vector representing pixel grayscale statistics of pixels in the tile. The T2S2 36-dimensional vector may be an example of a vector representing pixel gray statistics.

As illustrated in FIG. 72, the quantization table 7200 may be used to associate the descriptor 7202 with each of the descriptors 7102. The quantization table 7200 may include any table, index, chart, or other data structure used to map the descriptor 7102 to the descriptor key 7202. The quantization table 7200 with various forms is well known to those skilled in the art. According to one or more exemplary embodiments, the quantization table 7200 may be generated by a descriptor 7102 that processes several images to identify individual images. Subsequently, the identified descriptor 7102 may be statistically analyzed for identifying clusters or groups of similar descriptors 7102 or descriptors 7102 having statistically similar values. For example, the values of the variables in the T2S2 vector are similar. The representative descriptor 7204 of each cluster may be selected, and in addition to the position in the quantization table 7200, a corresponding descriptor key 7202 may be assigned thereto. The plurality of descriptor keys 7202 may include any desired indicators for identifying the representative descriptor 7204 corresponding thereto. For example, the descriptor key 7202 may include integer values, alphabetic characters, numeric values, symbols, text, or a combination thereof, as illustrated in FIG. 72. According to one or more exemplary embodiments, the descriptor key 7202 may include a series of characters for identifying the descriptor key related to the non-text-based search pattern. For example, all descriptor keywords are the first four characters of the keyword, and may include a series of three integers and subsequent underscore characters. Also, this initial sequence can be used to identify descriptor keywords related to the video.

The representative descriptor that most closely matches each descriptor 7102 may be identified in the quantization table 7200. For example, the descriptor 7102a illustrated in FIG. 71 may most closely correspond to the representative descriptor 7202a of the quantization table 7200 illustrated in FIG. 9. For example, the descriptor 7102a illustrated in FIG. 71 may most closely correspond to the representative descriptor 7204a of the quantization table 7200 illustrated in FIG. 72. Therefore, the descriptor key 7202 of each descriptor 7102 may be related to the video 6800. For example, the descriptor 7102a may correspond to the descriptor identifier 7202a "1". The descriptor key 7202 associated with the image 6800 may be different, and one or more of the descriptor keys 7202 may be related to the image 6800 multiple times. For example, the image 6800 may include descriptor keywords "1,2,3,4" or "1,2,2,3". According to one or more exemplary embodiments, in order to consider characteristics such as changes in the image, two or more representative descriptors 7204 and two or more than two that most closely match the descriptor 7102 may be identified by The descriptor key 7202 of each of the representative descriptors 7204 maps the descriptor 7102 to two or more descriptor identifiers 7202. Based on this, the content of the image 6800 with a set of identified POI 6802 can be expressed by a set of descriptor keywords 7202.

According to one or more exemplary embodiments, another image-based search method may be integrated into the search scheme. For example, the facial recognition method may provide image search based on another method. As described above, in addition to or instead of identifying the descriptor key as described above, the identity of the person in the image can be determined by using the facial recognition method. The identity of the person in the image can be used to supplement the search query. Or, when the metadata of many people is included in the library, the query can be supplemented by using the stored metadata.

The above description may provide the description of the adaptive search scheme for the image query type to another search scheme such as text. Similar adjustments can be performed for search methods of different query types (for example, audio query types). According to one or more exemplary embodiments, any type of audio-based search method may be used. A search using a query component with an audio query type can use one or more types of features to identify audio files with similar characteristics. As described above, audio features can be related to descriptor keywords. The descriptor keyword may have a format indicating the keyword associated with the audio search, which is similar to the situation where the last four characters of the keyword correspond to the four digits after the hyphen.

In the exemplary embodiment described above, it is described that the query input window displays a display item corresponding to the query component (ie, query input). Here, it should be understood that the display item may be the query component itself or a representation of the query component (such as waveforms, thumbnail images, preview images, etc.). In addition, according to one or more exemplary embodiments, a first display item corresponding to the first query type and a second display item corresponding to the second query type may be displayed so that the first query type and the second query type may be mutually Differentiate.

One or more exemplary embodiments may be implemented in the form of a storage medium containing computer-executable instructions (such as program modules) executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes volatile media such as RAM, non-volatile media such as ROM, and removable and non-removable media. In addition, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile media, non-volatile media, and removable and non-removable media implemented as any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The communication medium can be computer readable instructions, and other data in the modulated data signal (such as data structure or program module) or other transmission mechanisms, and includes any information transmission medium. Examples of computer storage media include ROM, RAM, flash memory, CD, DVD, magnetic disk, or tape. It is further understood that one or more of the above-described components and elements of the devices and devices described above may include hardware, circuits, one or more processors, and so on.

It should be understood that the exemplary embodiments described herein should be considered in descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each exemplary embodiment should generally be considered as available for other similar features or aspects in other embodiments.

Although one or more exemplary embodiments have been described with reference to the drawings, those of ordinary skill in the art will understand that various changes can be made in form and details without departing from the present invention as defined by the scope of the appended patent applications The spirit and scope of the inventive concept.

1‧‧‧ user 210‧‧‧Query input window 211‧‧‧First query component 212‧‧‧Second query component 220‧‧‧Result display area 221‧‧‧response result 222‧‧‧response result 300‧‧‧Query input device 310‧‧‧Bus 320‧‧‧Memory 330‧‧‧ processor 340‧‧‧Output device 350‧‧‧I/O port 360‧‧‧I/O components 370‧‧‧Power 400‧‧‧Query input device 410‧‧‧ Internet 420‧‧‧Search engine server 501‧‧‧Single input mode selection icon 502‧‧‧Multi-modal input mode selection icon 510‧‧‧Single query input window 520‧‧‧Single query type list 521‧‧‧Object 530‧‧‧Search button 801‧‧‧Single input mode selection icon 802‧‧‧Multi-modal input mode selection icon 810‧‧‧Query input window 811‧‧‧ The first query component input area 812‧‧‧Second query component input area 813‧‧‧The third query component input area 814‧‧‧ Sketch input window 820‧‧‧Single query type list 821‧‧‧Object 822‧‧‧Object 823‧‧‧Object 824‧‧‧Object 830‧‧‧Search 840‧‧‧Border 850‧‧‧Zoom in or zoom out 860‧‧‧ via mobile zone 1910‧‧‧The first query component input area 1915‧‧‧The second query component input area 1920‧‧‧The second query component input area 1921‧‧‧First search results 1922‧‧‧Second search results 1923‧‧‧ search results 1924‧‧‧ search results 1930‧‧‧search button 1940‧‧‧Query Type List 2100‧‧‧Search mode selection object 2110‧‧‧Any area 2210‧‧‧Query input window 2220‧‧‧Query Type List 2230‧‧‧Search button 2310‧‧‧Query input window 2311‧‧‧Text input window 2312‧‧‧Image upload window 2320‧‧‧Single query type list 2321‧‧‧The first object 2322‧‧‧Second Object 2330‧‧‧Search 2340‧‧‧ one point 2400‧‧‧Search mode selection object 2410‧‧‧Multi-modal query type list 2500‧‧‧Inquiry input device 2501‧‧‧Some districts 2510‧‧‧Query input window 2510-1‧‧‧Enquiry input window 2511‧‧‧Character 2511-1‧‧‧Image input area 2512-1‧‧‧Voice input area 2520‧‧‧Image 2520-1‧‧‧Search result display area 2521‧‧‧floating icon 2701‧‧‧ indicator 2702‧‧‧ Indicator 2710‧‧‧Query input window 2711‧‧‧User interface 2712‧‧‧User interface 2720‧‧‧Search button 2730‧‧‧File icon 2740‧‧‧File search window 2750‧‧‧Information 2751‧‧‧ Text 3410‧‧‧Text box 3411‧‧‧Cursor 3500‧‧‧Inquiry input device 3510‧‧‧Query input window 3511‧‧‧Video upload icon 3512‧‧‧Photograph icon 3513‧‧‧Image address input box 3520‧‧‧ Selected image 3530‧‧‧Application 3910‧‧‧Enquiry input window 3911‧‧‧ Sketch input tool 4010‧‧‧Query input window 4011‧‧‧Sound file upload button 4012‧‧‧Audio record icon 4021‧‧‧File selection window 4022‧‧‧Application 4100‧‧‧Inquiry input device 4110‧‧‧Query input window 4111‧‧‧Voice Information 4112‧‧‧Image 4120‧‧‧search button 4201‧‧‧Query input window 4202‧‧‧Image input area 4203‧‧‧Text input area 4204‧‧‧search button 4205‧‧‧Search result display area 4400‧‧‧Inquiry input device 4401‧‧‧Smart Watch 4402‧‧‧Smart glasses 4403‧‧‧Front camera 4405‧‧‧Pupillary tracking camera 4410‧‧‧Query input window 4411‧‧‧Voice Information 4412‧‧‧Image 4700‧‧‧Inquiry input device 4710‧‧‧Monitor 4720‧‧‧Controller 4730‧‧‧ input device 4800‧‧‧Inquiry input device 4801‧‧‧ device 4802‧‧‧device 4803‧‧‧device 4810‧‧‧ Gateway 5200‧‧‧Inquiry input device 5210‧‧‧Query input window 5210-1‧‧‧Query 5221‧‧‧ search results 5222‧‧‧ search results 5223‧‧‧Result display area 5230‧‧‧floating icon 5231‧‧‧Query input area 5801‧‧‧Select objects in single mode 5802‧‧‧Multi-modal input mode selection object 5810‧‧‧Query input window 5820-1‧‧‧ List of single query types 5820-2‧‧‧Multi-modal query type list 5821-1‧‧‧Text icon 5821-2‧‧‧Image icon 5821-3‧‧‧File icon 5821-4‧‧‧ sketch icon 5821-5‧‧‧Camera icon 5821-6‧‧‧ Music icon 5821-7‧‧‧Log icon 5822-1‧‧‧Combination icon 5822-2‧‧‧Combination icon 5822-3‧‧‧Combination icon 5830‧‧‧Search button 6010‧‧‧ icon 6011‧‧‧Text icon 6012‧‧‧Image icon 6020‧‧‧Combination icon 6021‧‧‧Combination icon 6100‧‧‧Inquiry input device 6102‧‧‧Image 6103‧‧‧ text 6110‧‧‧Query input window 6111‧‧‧Image 6113‧‧‧ text 6120‧‧‧ collection 6121‧‧‧File opening icon 6122‧‧‧ Selected files 6130‧‧‧Search button 6140‧‧‧Video 6600‧‧‧Query interface 6610‧‧‧Combination query 6611‧‧‧Keyword 6612‧‧‧Image 6613‧‧‧Video 6614‧‧‧Voice 6615‧‧‧ Sketch Information 6616‧‧‧ contextual information 6620‧‧‧Search method 6630‧‧‧ File Collection 6640‧‧‧Inquiry and adjustment module 6650‧‧‧Search method merge module 6660‧‧‧Search result output module 6705‧‧‧First query component 6707‧‧‧Second query component 6712‧‧‧Second query component understanding component 6714‧‧‧ metadata analysis module 6722‧‧‧Second query component feature component 6732‧‧‧Component 6742‧‧‧Query suggestions 6744‧‧‧ Response 6746‧‧‧Combination result 6752‧‧‧Result generation operation 6762‧‧‧Result operation 6775‧‧‧ Database 6800‧‧‧ video 6802‧‧‧point of concern (POI) 6802a‧‧‧point of concern (POI) 6804‧‧‧ 6902‧‧‧Section 6902a‧‧‧ 7000‧‧‧ collection 7002‧‧‧ SMD 7102‧‧‧Descriptor 7102a‧‧‧Descriptor 7200‧‧‧Quantization table 7202‧‧‧Descriptor keywords 7204‧‧‧ representative descriptor 7204a ‧‧‧ representative descriptor S110‧‧‧Operation S120‧‧‧Operation S130‧‧‧Operation S2610‧‧‧Operation S2620‧‧‧Operation S2630‧‧‧Operation S2910‧‧‧Operation S2920‧‧‧Operation S2930‧‧‧Operation S3310‧‧‧Operation S3320‧‧‧Operation S3330‧‧‧Operation S3335‧‧‧Operation S3340‧‧‧Operation S3345‧‧‧Operation S4210‧‧‧Operation S4220‧‧‧Operation S4230‧‧‧Operation S4310‧‧‧Operation S4320‧‧‧Operation S4330‧‧‧Operation S4510‧‧‧Operation S4520‧‧‧Operation S4525‧‧‧Operation S4530‧‧‧Operation S4535‧‧‧Operation S4610‧‧‧Operation S4620‧‧‧Operation S4630‧‧‧Operation S4640‧‧‧Operation S4650‧‧‧Operation

These and/or other aspects will become apparent and easier to understand from the following description of exemplary embodiments taken in conjunction with the drawings, in which: FIG. 1 is a flowchart illustrating a procedure for performing a search according to an exemplary embodiment. FIG. 2 is a conceptual diagram illustrating the configuration of a user interface according to an exemplary embodiment. FIG. 3 is a conceptual diagram illustrating a structure for constructing a query input device according to an exemplary embodiment. FIG. 4 is a conceptual diagram illustrating a network environment for performing an exemplary embodiment. 5 to 7 are exemplary diagrams illustrating a method of receiving a single searched query according to an exemplary embodiment. 8 to 18 are exemplary diagrams illustrating a method of receiving a query of multi-modal search according to an exemplary embodiment. 19 and 20 are exemplary diagrams illustrating a method of receiving a query of multimodal search according to another exemplary embodiment. 21 and 22 are exemplary diagrams illustrating a method of generating a query input window according to an exemplary embodiment. FIG. 23 is an exemplary diagram illustrating a method of receiving a multimodal search query according to another exemplary embodiment. 24 is an exemplary diagram illustrating a multi-modal query type list according to another exemplary embodiment. 25A to 25E are exemplary diagrams illustrating a method of receiving a query of a multi-modal search according to another exemplary embodiment. FIG. 26 is a flowchart illustrating a procedure of displaying a query input window according to an exemplary embodiment. 27 and 28 are exemplary diagrams illustrating a method of displaying a query input window according to an exemplary embodiment. FIG. 29 is a flowchart illustrating a procedure of receiving a query according to an exemplary embodiment. 30 to 32 are exemplary diagrams illustrating a method of receiving a query according to an exemplary embodiment. 33 is a flowchart illustrating a procedure of receiving a query according to an exemplary embodiment. 34 is an exemplary diagram illustrating a method of receiving a query component in which the query type is text according to an exemplary embodiment. 35 to 38 are exemplary diagrams illustrating a method of receiving a query component in which the query type is an image according to an exemplary embodiment. 39 is an exemplary diagram illustrating a method of receiving a query component in which the query type is sketch information according to an exemplary embodiment. FIG. 40 is an exemplary diagram illustrating a method of receiving a query component in which the query type is audio information according to an exemplary embodiment. 41 is an exemplary diagram illustrating a method of receiving a query including multiple query components having multiple query types and displaying search results according to an exemplary embodiment. FIG. 42 is a flowchart illustrating a procedure for performing search according to an exemplary embodiment. FIG. 43 is an exemplary diagram illustrating a result obtained by performing a multi-modal search according to an exemplary embodiment. 44 is a flowchart illustrating a procedure of displaying search results according to another exemplary embodiment. FIG. 45 is an exemplary diagram illustrating the priority of search results according to another exemplary embodiment. 46 is an exemplary diagram illustrating a result obtained by performing multi-modal search according to another exemplary embodiment. 47A is an exemplary diagram illustrating a method of receiving a query by using multiple devices according to an exemplary embodiment. 47B is an exemplary diagram illustrating a method in which smart glasses according to an exemplary embodiment acquire an image of an area seen by a user from the entire television (television. TV) screen. FIG. 48 is an exemplary diagram illustrating a method of receiving a query component via multiple devices according to another exemplary embodiment. FIG. 49 is a flowchart illustrating a procedure for performing search according to an exemplary embodiment. FIG. 50 is a flowchart illustrating a procedure for performing search according to an exemplary embodiment. FIG. 51 is a block diagram illustrating the structure of a query input device according to an exemplary embodiment. 52 to 57 are exemplary diagrams illustrating a method of performing multi-modal search according to another exemplary embodiment. FIG. 58 is an exemplary diagram illustrating a method of displaying a query type list according to another exemplary embodiment. FIG. 59 is an exemplary diagram illustrating a method of displaying a list of query types according to another exemplary embodiment. FIG. 60 is an exemplary diagram illustrating a method of generating a multimodal query type list according to an exemplary embodiment. 61 to 65 are exemplary diagrams illustrating a method of performing multi-modal search according to another exemplary embodiment. 66 is a block diagram illustrating a framework for performing multimodal search based on a query having multiple query types according to an exemplary embodiment. 67 illustrates a diagram of a system for implementing one or more exemplary embodiments. 68 to 72 are exemplary diagrams illustrating a procedure for capturing image features of a query component having an image query type according to one or more exemplary embodiments.

S110‧‧‧Operation

S120‧‧‧Operation

S130‧‧‧Operation

Claims (13)

A search method, including: The query component indicating the search request received from the user; Capture images including multiple objects by an image capture device; Present the image on the display; Receiving user input on the presented image to select a partial area on the presented image, wherein the partial area corresponds to at least one of the plurality of objects; Send a query towards a search server based on the selected partial area on the presented image and the received query component; Receiving search results based on the sent query from the search server; and The received search results are presented on the display, the search results providing information associated with the at least one of the plurality of objects. The search method as described in item 1 of the patent application scope, wherein the search results presented include multiple images. The search method according to item 1 of the patent application scope, wherein the presented search results include images of search result products corresponding to the products included in the selected partial area. The search method as described in item 3 of the patent application scope, wherein the search result presented further includes the price of the search result product. The search method according to item 1 of the patent application scope, wherein presenting the received search results on the display includes: The received search result is displayed in a result display area, where the received search result includes at least one of a thumbnail of an image file, some text contained in the file, a link of the searched file, or an icon. The search method as described in item 1 of the patent application scope, wherein the sent query includes an image file corresponding to the selected partial area. An electronic device, including: monitor; Memory, store instructions; At least one hardware processor configured to execute the instructions to at least achieve: The query component indicating the search request received from the user; Capture images including multiple objects by an image capture device; Presenting the image on the display; Receiving user input on the presented image to select a partial area on the presented image, wherein the partial area corresponds to at least one of the plurality of objects; Send a query towards a search server based on the selected partial area on the presented image and the received query component; Receiving search results based on the sent query from the search server; and The received search results are presented on the display, the search results providing information associated with the at least one of the plurality of objects. The electronic device as described in item 7 of the patent application scope, wherein the search results presented include multiple images. The search method as described in item 7 of the patent application scope, wherein the presented search results include images of search result products corresponding to the products included in the selected partial area. The search method as described in item 9 of the patent application scope, wherein the search result presented further includes the price of the search result product. The search method as described in item 7 of the patent application scope, wherein the at least one hardware processor is configured to execute the instruction to at least achieve: The received search result is displayed in a result display area, where the received search result includes at least one of a thumbnail of an image file, some text contained in the file, a link of the searched file, or an icon. The search method as described in item 7 of the patent application scope, wherein the at least one hardware processor is configured to execute the instruction to at least achieve: Presenting a plurality of stored images on the display; and Receiving a user selection of one of the plurality of stored images presented, The image presented therein corresponds to the one image selected by the user selection. A non-transitory computer-readable recording medium having instructions stored therein, the instructions can be executed by at least one hardware processor of an electronic device with a display, so that the electronic device performs the following operations: The query component indicating the search request received from the user; Capture images including multiple objects by an image capture device; Presenting the image on the display; Receiving user input on the presented image to select a partial area on the presented image, wherein the partial area corresponds to at least one of the plurality of objects; Send a query towards a search server based on the selected partial area on the presented image; Receiving search results based on the sent query and the received query component from the search server; and The received search results are presented on the display, the search results providing information associated with the at least one of the plurality of objects.

Images