KR101914548B1 - Mobile terminal having function of voice recognition and method for providing search results thereof - Google Patents

Abstract

A mobile terminal that selectively converts a voice query into a text query and selectively provides search results for a text query. When a search result that is not desired by a user is provided, the user can easily change the domain of the search result, The present invention provides a mobile terminal and a search result providing method that can provide a search result intended by a user by providing a user interface capable of performing search results. According to another aspect of the present invention, there is provided a method for providing search results of a mobile terminal, the method comprising: receiving a voice query from a user and converting the received voice query into a text query; Executing the text query to obtain a search result and displaying a plurality of domains included in the search result; Receiving an input associating the text query with a domain of one of the plurality of domains; And displaying a search result corresponding to the one domain.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mobile terminal having a voice recognition function and a search result providing method therefor.

The present invention relates to a mobile terminal having a voice recognition function and a method for providing a search result, and more particularly, to a mobile terminal providing a search result of a voice query and a method for providing the search result.

A terminal can be divided into a mobile or portable terminal and a stationary terminal depending on whether the terminal is movable or not. The mobile terminal can be divided into a handheld terminal and a vehicle mount terminal according to whether the user can directly carry the mobile terminal.

As the functions of such terminals are diversified, they are implemented in the form of a multimedia player having complex functions such as photographing and photographing of movies, reproduction of music and video files, games, and broadcasting reception . Further, in order to support and enhance the function of the terminal, it may be considered to improve the structural and software parts of the terminal.

In one example, the terminal may receive a voice query from a user and provide a search result corresponding to the voice query. In this case, the voice query is converted into a text query, and the text query is executed in the search engine, so that the search result corresponding to the text query in the database can be provided to the user.

However, the display size is very limited because the terminal must be portable for its use. Depending on display size constraints, there is a limit to providing multiple search results for a single text query at a time. Recently, in order to solve such a problem, an intelligent search technology has appeared that provides an optimal search result for one text query based on the user's context and usage history.

One embodiment of the present invention provides a mobile terminal that selectively provides a search result for a text query by converting a voice query into a text query, and when the user provides an undesired search result, easily changes the domain of the search result And a user interface capable of performing editing or re-recognition of a text query, thereby providing a search result intended by a user, and a method for providing the search result.

According to another aspect of the present invention, there is provided a method for providing a search result of a mobile terminal, the method comprising: receiving a voice query from a user and converting the received voice query into a text query; displaying a converted text query; Selecting and displaying a first search result corresponding to at least one of the current context of the user and the application usage history of the user among the search results; The method comprising the steps of: displaying a plurality of domains included in the search result; receiving an input associating the text query with one of the plurality of domains; and displaying a second search result corresponding to the one domain Wherein the step of displaying the plurality of domains comprises the steps of: Identifying semantic information related to each of the first search results from metadata including semantic information related to each of the plurality of semantic information, matching the identified semantic information with a previously defined query domain, Acquiring a domain of the acquired domain; and displaying the obtained plurality of domains.

In one embodiment, the input for associating the text query with one of the plurality of domains is an input for dragging and dropping the text query to the one domain.

In one embodiment, the step of displaying the plurality of domains is a step of displaying a plurality of domains included in the search result upon receipt of an input for selecting the text query.

In one embodiment of the present invention, the step of executing the text query to obtain a search result may include displaying a search result corresponding to a main domain of the text query.

In one embodiment, the plurality of domains include candidate domains of the text query.

According to another embodiment of the present invention, there is provided a method of providing search results of a mobile terminal, the method comprising: displaying an item capable of editing the text query; Receiving an input associating the text query with an item capable of editing the text query; And providing an interface through which the text query can be edited.

In one embodiment, an input for associating the text query with an item capable of editing the text query is an input for dragging and dropping the text query into an item capable of editing the text query .

According to another embodiment of the present invention, there is provided a method of providing search results of a mobile terminal, the method comprising: displaying an item capable of re-recognizing the text query; Receiving an input that associates the text query with an item that can re-recognize the text query; And re-recognizing the text query.

In one embodiment, an input for associating the text query with an item that can recognize the text query is an input for dragging and dropping the text query into an item capable of re-recognizing the text query .

According to an embodiment of the present invention, a mobile terminal includes a microphone for receiving a voice query from a user; Selecting a first search result corresponding to at least one of a current context of the user and an application usage history of the user among the obtained search results; and converting the voice query into a text query, executing the text query to obtain a search result, And displaying a plurality of domains corresponding to the first search result according to a user's selection in a state where the first search result is displayed; And a touch-sensitive display for displaying the text query and the plurality of domains, receiving an input associating the text query with a domain of one of the plurality of domains, and displaying a second search result corresponding to the one domain Wherein the control unit identifies semantic information related to each of the search results from metadata including semantic information related to each of the first search results and matches the identified semantic information to a predefined query domain And obtains the plurality of domains according to the semantic information.

According to an embodiment of the present invention, in a mobile terminal having a voice recognition function and a search function, an intuitive user interface based on a touch input is provided to change a domain of a search result or correct a voice recognition error. Accordingly, it is possible to eliminate the hassle of starting the whole process from the beginning in order to change the domain of the search result or correct the speech recognition error in the search process using the speech recognition.

1 is a block diagram illustrating a mobile terminal 100 according to the present invention.
2A and 2B are perspective views showing an appearance of a mobile terminal 100 according to the present invention.
3 is a flowchart illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to the related art.
4A to 4F are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to the related art.
5 is a flowchart illustrating a process of providing search results for a query input by a user according to an embodiment of the present invention.
6A to 6D are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating a process of providing a search result for a query input by a user according to an embodiment of the present invention.
8A to 8E are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to an embodiment of the present invention.
FIG. 9 is a flowchart illustrating a process of providing a search result for a query input by a user, according to an embodiment of the present invention.
10A to 10D are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to an embodiment of the present invention.
11 is a conceptual diagram illustrating a process in which the electronic device 700 provides a search result for a query input by a user according to an embodiment of the present invention.
12 illustrates an exemplary embodiment for processing the receipt of a voice query in accordance with speech recognizer 712.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art can easily carry out the technical idea of the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly explain the present invention, parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification.

The mobile terminal described herein may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigator, have. However, it should be understood that the configuration according to the embodiments described herein may be applied to a fixed terminal such as a digital TV, a desktop computer, and the like, unless it is applicable only to a mobile terminal. It will be easy to see.

1 is a block diagram illustrating a mobile terminal 100 according to the present invention. 1, a mobile terminal 100 includes a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, A controller 160, an interface unit 170, a controller 180, and a power supply unit 190. The components shown in Fig. 1 are not essential, so that a mobile terminal having more or fewer components can be implemented. Hereinafter, the components 110 to 190 of the mobile terminal 100 will be described in order.

The wireless communication unit 110 may include one or more modules for enabling wireless communication between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and the network in which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115.

The broadcast receiving module 111 receives broadcast signals and broadcast related information from an external broadcast management server through a broadcast channel. Here, the broadcast-related information means information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information can also be provided through a mobile communication network. In this case, the broadcast-related information may be received by the mobile communication module 112. The broadcast signal and the broadcast-related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 112 transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. Such wireless signals may include various types of data depending on a voice call signal, a video call signal, a text message, or a multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be embedded in the mobile terminal 100 or externally. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies.

The short-range communication module 114 refers to a module for short-range communication. As a short range communication technology, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like can be used.

The position information module 115 is a module for acquiring the position of the mobile terminal 100, and a representative example thereof is a Global Position System (GPS) module.

1, an A / V (Audio / Video) input unit 120 is for inputting an audio signal and a video signal, and may include a camera 121, a microphone 122, and the like. The camera 121 processes image frames such as still images and moving images obtained by the image sensor in the video communication mode or the photographing mode. The image frame processed by the camera 121 can be displayed on the display unit 151. [ The image frame may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ More than two cameras 121 may be provided depending on the use environment.

The microphone 122 processes the sound signal input from the outside in the communication mode, the recording mode, the voice selection mode, and the like as electrical voice data. The voice data processed by the microphone 122 in the communication mode can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 112 and output. The microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated when an external sound signal is input.

The user input unit 130 generates input data for controlling the operation of the mobile terminal 100 by a user. The user input unit 130 may include a key pad, a dome switch, a touch pad (static pressure and static electricity), a jog wheel, a jog switch, and the like.

The sensing unit 140 senses the current state of the mobile terminal 100 such as presence or absence of a user, the open / close state of the mobile terminal 100, position, orientation, acceleration, deceleration, And generates a sensing signal. For example, when the mobile terminal 100 is in the form of a slide phone, the sensing unit 140 may detect whether the slide phone is opened or closed. The sensing unit 140 may sense whether the power supply unit 190 is powered on, whether the interface unit 170 is connected to an external device, and the like.

The sensing unit 140 may include a proximity sensor 141. The sensing unit 140 may include a touch sensor (not shown) for sensing a touch operation with respect to the display unit 151.

The touch sensor may have the form of a touch film, a touch sheet, a touch pad, or the like. The touch sensor may be configured to convert a pressure applied to a specific portion of the display portion 151 or a change in capacitance generated at a specific portion of the display portion 151 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the touch pressure.

When the touch sensor and the display unit 151 have a mutual layer structure, the display unit 151 can be used as an input device in addition to the output device. The display unit 151 may be referred to as a 'touch screen'.

If there is a touch input via the touch screen, corresponding signals are sent to a touch controller (not shown). The touch controller processes the signals transmitted from the touch sensor, and then transmits data corresponding to the processed signals to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched.

In the case where the touch screen is electrostatic, it can be configured to detect the proximity of the sensing object by a change of the electric field along the proximity of the sensing target. Such a touch screen may be classified as proximity sensor 141. [

The proximity sensor 141 refers to a sensor that detects the presence or absence of an object to be sensed without mechanical contact using an electromagnetic force or infrared rays. The proximity sensor 141 has a longer life than the contact type sensor and its utilization is also high. Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor.

Hereinafter, for convenience of explanation, a proximity action is referred to as " proximity touch " while an object to be sensed does not touch the touch screen, and an action of touching the sensing object on the touch screen is called & touch ".

The proximity sensor 141 detects the presence or absence of a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, The information corresponding to the presence / absence of proximity touch and the proximity touch pattern can be output to the touch screen.

The output unit 150 generates an output related to visual, auditory, tactile, and the like. The output unit 150 may include a display unit 151, an audio output module 152, an alarm unit 153, and a haptic module 154.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, when the mobile terminal 100 operates in the call mode, the display unit 151 displays a UI (User Interface) or a GUI (Graphic User Interface) related to the call. When the mobile terminal 100 operates in the video communication mode or the photographing mode, the display unit 151 displays the photographed image, the received image, the UI, the GUI, and the like.

The display unit 151 may be a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic light emitting diode (OLED), a flexible display display, a 3D display, and an e-ink display.

At least one display (or display element) included in the display unit 151 may be configured to be transparent or light transmissive so that the display can see the outside through it. This can be referred to as a transparent display. A typical example of such a transparent display is TOLED (Transparent OLED). The rear structure of the display unit 151 may also be of a light transmission type. With this structure, the user can see an object located behind the terminal body through the area occupied by the display unit 151 in the terminal body.

There may be two or more display units 151 according to the embodiment of the mobile terminal 100. [ For example, in the mobile terminal 100, a plurality of display portions may be spaced apart from each other or positioned integrally with one another, and may be located on different surfaces.

The audio output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception mode, a call mode or a recording mode, a voice selection mode, a broadcast reception mode, The sound output module 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The audio output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event of the mobile terminal 100. Examples of events generated in the mobile terminal 100 include call signal reception, message reception, key signal input, touch input, and the like. The alarm unit 153 may output a signal for informing occurrence of an event in a form other than a video signal or an audio signal, for example, vibration. The display unit 151 and the audio output module 152 may be classified as a part of the alarm unit 153 because the video signal or the audio signal can also be output through the display unit 151 or the audio output module 152. [

The haptic module 154 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 154 is vibration. The intensity, pattern, and the like of the vibration generated by the hit module 154 are controllable. For example, different vibrations may be synthesized and output or sequentially output.

In addition to the vibration, the haptic module 154 may include a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or a suction force of the air through the injection port or the suction port, a touch on the skin surface, contact with an electrode, Various effects can be generated such as the effect of heat generation and the effect of reproducing the cold sensation using the heat absorbing or heatable element.

The haptic module 154 can be configured to not only transmit the tactile effect through the direct contact but also to allow the user to feel the tactile effect through the muscular sense of the finger or arm. At least two haptic modules 154 may be provided according to the configuration of the mobile terminal 100.

The memory 160 may store a program for the operation of the controller 180 and temporarily store input and output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 160 may store data related to vibrations and sounds of various patterns that are output upon touch input on the touch screen.

The memory 160 may be a flash memory, a hard disk, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM) And may include at least one storage medium. The mobile terminal 100 may operate in association with a web storage that performs storage functions of the memory 160 on the Internet.

The interface unit 170 serves as a path for communication with all external devices connected to the mobile terminal 100. The interface unit 170 receives data from an external device or supplies power to each component in the mobile terminal 100 or transmits data to the external device. For example, the interface unit 170 may include a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, Input / output ports, video I / O ports, earphone ports, and the like.

The identification module is a chip for storing various information for authenticating the usage right of the mobile terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM), a general user authentication module Universal Subscriber Identity Module (USIM)). A device equipped with an identification module (hereinafter referred to as "identification device") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the port.

The interface unit 170 may be a path through which the power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle or various command signals input from the cradle by the user, (Not shown). The various command signals or power supplied from the cradle may also act as a signal for recognizing that the mobile terminal 100 is correctly mounted in the cradle.

The controller 180 controls the overall operation of the mobile terminal 100. For example, control and processing related to voice communication, data communication, video communication, and the like. The control unit 180 may include a multimedia module 181 for multimedia playback. The multimedia module 181 may be implemented in the control unit 180 or may be implemented separately from the control unit 180. [ The control unit 180 may perform pattern selection processing for selecting handwriting input and drawing input on the touch screen as characters and images, respectively.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components.

The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be applied to various types of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays Microprocessors, microprocessors, and other electronic units for performing other functions, as will be appreciated by those skilled in the art. In some cases, the embodiments described herein may be implemented by the controller 180 itself.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code may be implemented in a software application written in a suitable programming language. Such software codes may be stored in the memory 160 and executed by the control unit 180. [

Hereinafter, a method of processing user input to the mobile terminal 100 will be described.

The user input unit 130 is operated to receive a command for controlling the operation of the mobile terminal 100, and may include a plurality of operation units. The operating units may also be referred to as manipulating portions and may be employed in any manner as long as they are operated in a tactile manner by the user's tactile sense.

Various kinds of time information can be displayed on the display unit 151. [ Such visual information can be displayed in the form of letters, numbers, symbols, graphics, icons, and the like, and can be formed as a three-dimensional stereoscopic image. At least one of a character, a number, a symbol, a graphic, and an icon may be displayed in a predetermined arrangement for inputting time information, thereby being implemented as a keypad. Such a keypad may be referred to as a so-called " soft key ".

The display unit 151 may operate as an entire area or may be divided into a plurality of areas and operated. In the latter case, the plurality of areas can be configured to operate in association with one another. For example, an output window and an input window may be displayed on the upper and lower portions of the display unit 151, respectively. The output window and the input window are areas allocated for outputting or inputting information, respectively. In the input window, a soft key with a number for inputting a telephone number may be output. When the soft key is touched, the number corresponding to the touched soft key is displayed in the output window. When the operating unit is operated, a call connection to the telephone number displayed in the output window may be attempted, or the text displayed in the output window may be entered into the application.

The display unit 151 or the touch pad may be configured to detect touch scrolling. The user can move the cursor or the pointer positioned on the displayed object, for example, the icon, on the display unit 151 by scrolling the display unit 151 or the touch pad. Further, when the finger is moved on the display unit 151 or the touch pad, the path along which the finger moves may be visually displayed on the display unit 151. [ This will be useful for editing the image displayed on the display unit 151. [

One function of the mobile terminal 100 may be executed in response to a case where the display unit 151 and the touch pad are touched together within a predetermined time range. In the case of being touched together, there may be a case where the user clamps the main body of the mobile terminal 100 using the thumb and index finger. At this time, one function of the mobile terminal 100 to be executed may be activation or deactivation for the display unit 151 or the touch pad, for example.

2A and 2B are perspective views showing an appearance of a mobile terminal 100 according to the present invention. FIG. 2A shows a front side and one side of the mobile terminal 100, and FIG. 2B shows a rear side and the other side of the mobile terminal 100. FIG.

Referring to FIG. 2A, the mobile terminal 100 includes a bar-shaped terminal body. However, the mobile terminal 100 is not limited thereto, and may be realized in various forms such as a slide type, a folder type, a swing type, and a swivel type in which two or more bodies are movably coupled to each other.

The terminal body includes a case (a casing, a housing, a cover, and the like) that forms an appearance. In the embodiment, the case may be divided into a front case 101 and a rear case 102. A variety of electronic components are embedded in the space formed between the front case 101 and the rear case 102. At least one intermediate case may be additionally disposed between the front case 101 and the rear case 102. [

The cases may be formed by injection molding of a synthetic resin or may be formed to have a metal material such as stainless steel (STS), titanium (Ti) or the like.

The display unit 151, the sound output unit 152, the camera 121, the user input unit 130 (see FIG. 1), the microphone 122, the interface 170, can do.

The display unit 151 occupies a main portion of the front case 101. [ The audio output unit 152 and the camera 121 are located in a region adjacent to one end of the display unit 151 and the first user input unit 131 and the microphone 122 are located in a region adjacent to the other end. The second user input unit 132 and the interface 170 may be located on the sides of the front case 101 and the rear case 102. [

The user input unit 130 is operated to receive a command for controlling the operation of the mobile terminal 100. [ The user input unit 130 may include a plurality of operation units 131 and 132.

The first or second operating units 131 and 132 can receive various commands. For example, the first operation unit 131 can receive commands such as start, end, scroll, and the like. The second operation unit 132 can receive commands such as adjusting the size of the sound output from the sound output unit 152, switching to the touch selection mode of the display unit 151, and the like.

Referring to FIG. 2B, a rear camera 121 'may be additionally mounted on the rear surface of the terminal body, that is, the rear case 102. The rear camera 121 'has a photographing direction opposite to that of the front camera 121 (see FIG. 2A), and may be configured to have pixels different from those of the front camera 121.

For example, the front camera 121 may be configured to have a low pixel, and the rear camera 121 'may be configured to have a high pixel. Accordingly, when the front camera 121 is used during a video call, when the user's face is photographed and the photographed image is transmitted to the other party in real time, the size of the transmitted data can be reduced. On the other hand, the rear camera 121 'can be used for the purpose of storing a high-quality image.

Meanwhile, the cameras 121 and 121 'may be installed in the terminal body so as to be rotated or popped up.

The flash 123 and the mirror 124 may be further positioned adjacent to the rear camera 121 '. The flash 123 emits light toward the subject when the user photographs the subject with the rear camera 121 '. The mirror 124 illuminates the user's face when the user photographs himself (self-photographing) using the rear camera 121 '.

A rear sound output unit 152 'may be additionally disposed on the rear surface of the terminal body. The rear sound output unit 152 'may perform a stereo function together with the front sound output unit 152 (see FIG. 2A), and may perform a speakerphone function during a call.

In addition to the antenna for communication, an antenna 116 for receiving broadcast signals may be additionally provided on the side surface of the terminal body. The antenna 116 constituting a part of the broadcast receiving module 111 (see FIG. 1) can be installed to be able to be drawn out from the terminal body.

A power supply unit 190 for supplying power to the mobile terminal 100 is mounted on the terminal body. The power supply unit 190 may be built in the main body of the terminal or may be detachable from the outside of the main body of the terminal.

The rear case 102 may further include a touch pad 135 for sensing a touch. The touch pad 135 may be of a light transmission type like the display unit 151 (see FIG. 2A). Further, a back display unit for outputting time information can be additionally mounted on the touch pad 135. [ At this time, information output from both sides of the front display unit 151 and the rear display unit can be controlled by the touch pad 135. [

The touch pad 135 operates in correlation with the display unit 151. [ The touch pad 135 may be positioned parallel to the rear of the display unit 151. The touch pad 135 may have a size equal to or smaller than that of the display unit 151.

The mobile terminal 100 may have a function of providing a search result corresponding to a query received from a user. When the mobile terminal 100 receives the text query from the user through the user input unit 130, the control unit 180 executes the text query in the built-in search engine and outputs the search result through the output unit 150 . Alternatively, the wireless communication unit 110 may transmit the text query to the server through the network to be executed in the external search engine, receive the search result from the server, and output the search result through the output unit 150. When the mobile terminal 100 receives the voice query from the user through the microphone 121, the control unit 180 converts the voice query into a text query and executes the text query in the search engine described above to provide search results to the user It is possible.

When a plurality of search results are returned from a search engine, each search result may be classified by a domain. That is, search results may be associated with each domain, which may be predefined as an index representing a general category to which any search result may be associated. However, even if the domain is not covered within a predefined range, it may be newly created according to the attribute of the search result. A domain can be obtained by identifying semantic information associated with a subject included in the search result. For example, various forms of content may include metadata that identifies the meaning embedded in such content, such as an email address, email address, or telephone number, an alias used in the messaging service, and the like. The mobile terminal 100 may include a process of reading the semantic information from the metadata and matching it to a predefined query domain.

In addition, intelligent search engines can provide optimal search results by reflecting the user's context and usage history in the query. For example, "What is today's weather?" The user can provide the weather of the area where the user is currently located using the location information of the mobile terminal 100 as a search result. Alternatively, the query "I play my favorite game" does not explicitly indicate which game to execute, but the mobile terminal 100 analyzes the execution time and / or the execution count of the game to determine whether the game played most frequently or for a long time As a result of the search.

3 is a flowchart illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to the related art. As the user perceives, the process may proceed immediately, almost instantly, progressively, or at another appropriate rate. Once the process is activated, the process may be controlled automatically by a device such as the mobile terminal 100, or may be controlled by the user, independently of the user. Although the flow of the process described below includes a number of actions that appear to occur in a particular order, this process may be performed serially or in parallel (e.g., using a parallel processor or a multi-threading environment) , ≪ / RTI > more or fewer operations.

The mobile terminal 100 monitors whether a voice query item is selected (S110). The voice query item is a menu or icon that can be selected by the user, and the mobile terminal 100 can display the voice query item according to the user's input. That is, in a normal state, the mobile terminal 100 does not display a voice query item (e.g., it is hidden on the screen) but can display a voice query item when the user issues a command to display a voice query item . Alternatively, the voice query item may be implemented as a hardware button or fixedly displayed on the screen of the mobile terminal 100, so that the voice query item can be selected by the user under any circumstances.

If the voice query item is selected in operation 110, the mobile terminal 100 enters the listening mode (S120). That is, the microphone 121 is activated, and the function of receiving the voice input is activated. In the listening mode, the mobile terminal 100 receives all the external sounds including the user as an analog signal. The input analog signal is converted into a digital signal after the noise is removed through a preprocessing process. The digital signal may be temporarily stored in the memory 160. [

At the same time, the mobile terminal 100 visually or acoustically outputs an indicator to notify the user that the mobile terminal 100 is in the listening mode (S130). This is to induce the user to give a voice command because the mobile terminal 100 is in the listening mode. Accordingly, the mobile terminal 100 can display a phrase such as " Hearing " on the screen or output a predetermined alert sound.

The mobile terminal 100 may receive a voice query from the user (S140). The mobile terminal 100 converts the voice query received from the user into a text query (S150). In this process, a speech recognition technique, for example, STT (Speech To Text), which converts speech into text recognizable from a search engine, can be used. The mobile terminal 100 may execute the converted text query in a built-in search engine or send it to a server to be executed in an external search engine (S160).

The search engine returns a search result corresponding to the text query. If there is no search result, the search engine outputs a message indicating that there is no search result. The mobile terminal 100 determines whether a search result exists (S170). If the search result exists, the mobile terminal 100 displays the search result corresponding to the text query (S180). However, if there is no search result, the mobile terminal 100 indicates that there is no search result (S190).

4A to 4F are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to the related art.

As shown in FIG. 4A, the mobile terminal 100 may display the voice query item 202 on the screen 200 either fixedly or at the user's option. When the voice query item 202 is selected, the mobile terminal 200 enters the listening mode and displays a message 204 on the screen indicating that the mobile terminal 200 is in the listening mode.

As shown in FIG. 4B, since the mobile terminal 100 is in the listening mode, it can receive a voice query from the user. Upon receiving the voice query from the user, the mobile terminal 100 may convert the voice query into a text query executable in the search engine. In this case, the mobile terminal 100 may display the converted text query 206 on the screen 200.

As shown in FIG. 4C, the mobile terminal 100 may forward the text query to the search engine, and the search engine may return a search result corresponding to the text query. Then, the mobile terminal 100 can display the returned search result 208 on the screen. At this time, the search result 208 displayed on the screen 200 can be an optimal search result selected from a plurality of search results through various criteria such as the user's context and usage history. However, if the search result displayed on the screen 200 is not a search result desired by the user, the user must edit the text query or the mobile terminal 100 must re-recognize the text query.

4D, when the voice query item 202 displayed on the screen 200 is selected again, a message 210 indicating that the mobile terminal 200 is in the listening mode is displayed again on the screen.

As shown in FIG. 4E, the user may speak a new voice query different from the voice query of FIG. 4B in order to receive a desired search result. For example, if a user wanted a search result for " Restaurant Beijing Spot ", he could say a voice query modified by simply saying " Beijing Spot " instead of " Beijing Spot ". The mobile terminal 100 may convert the newly received voice query into a text query and display the converted text query 212 on the screen 200. [

As shown in FIG. 4F, the mobile terminal 100 can deliver the text query to the search engine, and the search engine can return the search result corresponding to the text query. Then, the mobile terminal 100 can display the returned search result 214 on the screen. The user now gets the desired search results.

5 is a flowchart illustrating a process of providing search results for a query input by a user according to an embodiment of the present invention. As the user perceives, the process may proceed immediately, almost instantly, progressively, or at another appropriate rate. Once the process is activated, the process may be controlled automatically by a device such as the mobile terminal 100, or may be controlled by the user, independently of the user. Although the flow of the process described below includes a number of actions that appear to occur in a particular order, this process may be performed serially or in parallel (e.g., using a parallel processor or a multi-threading environment) , ≪ / RTI > more or fewer operations.

The description of steps 110 to 160 shown in FIG. 5 is the same as the description of steps 110 to 160 shown in FIG. 3, and a description thereof will be omitted. Subsequently, the mobile terminal 100 determines whether a search result exists (S170). If there is no search result, the mobile terminal 100 indicates that there is no search result (S190). However, if there is a search result, the mobile terminal 100 displays the search result corresponding to the main domain of the text query (S182). Here, the main domain may be the domain corresponding to the search result most suitable for the text query. As described above, when there are a plurality of search results, an optimal search result can be selected based on predetermined criteria such as the user's context and usage history. In other words, the primary domain is the domain corresponding to the optimal search result.

Then, the mobile terminal 100 monitors whether a domain display item is selected (S210). The mobile terminal 100 may display the domain display item together with the search result. The domain display item does not necessarily have to be explicitly displayed. For example, when the user touches a portion of the screen where the search result is displayed, the domain display item may be regarded as selected.

If the domain display item is selected, the mobile terminal 100 determines whether there is a search result corresponding to another domain other than the search result corresponding to the main domain (S220). If there is no search result corresponding to another domain, a message indicating that there is no more domain is displayed (S240), and the process is terminated. However, if there is a search result corresponding to another domain, the corresponding domain is displayed on the screen (S230). A domain is a selectable item, and may include information about a domain. For example, a domain may be represented by an icon that includes a name or domain for the domain or a thumbnail for the search result. If there are search results corresponding to more than a predetermined number of domains, a predetermined number of candidate domains may be selected according to the priority order, and only the selected candidate domain may be displayed on the screen. The priority is determined based on, for example, the user context or the usage history, as described above.

The mobile terminal 100 monitors whether a domain displayed on the screen is selected (S250). When a plurality of domains are displayed, any one of the domains can be selected. In one embodiment, the domain displayed on the screen may be selected according to the gesture of dragging and dropping the text query displayed on the screen into the domain. When the domain is selected, the mobile terminal 100 may display a search result corresponding to the selected domain (S260). According to an embodiment, the mobile terminal 100 acquires search results for a domain other than the main domain at a time from the search engine, acquires only search results corresponding to the main domain, and then corresponds to the selected domain when another domain is selected Can be obtained again separately.

6A to 6D are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to an embodiment of the present invention.

6A, when the user selects the voice query item 302 displayed on the screen 300, the mobile terminal 100 enters a listening mode and displays a message 304 indicating that the user has entered the listening mode, ). Then, upon receiving the voice query of the user, the mobile terminal 100 converts the voice query into a text query, and causes the text query to be executed in the search engine. The mobile terminal 100 acquires the search result 308 corresponding to the main domain of the converted text query from the search engine and displays it on the screen 300. Here, the mobile terminal 100 may display the converted text query 306 on the screen 300. In this state, the user can select a domain display item. For example, the user can touch a portion of the screen 300 where the text query 306 is displayed.

6B, when the user touches a portion where the text query 306 is displayed on the screen 300, the mobile terminal 100 searches the domains 310 to 316 corresponding to the search results 308 and other search results Is displayed on the screen 300. The domains 310 to 316 are selectable items, for example, icons or menus, and may reflect the names of domains, thumbnails of search results, and the like.

6C, a user may drag and drop a portion where the text query 306 is displayed to a domain 310, which is denoted as a domain, e.g., " restaurant " in the domains 310 to 316 . This means the selection of the domain 310 and can be an intuitive user interface that associates the text query 306 with the domain 310.

6D, the mobile terminal 100 obtains a search result 318 corresponding to a selected domain of the text query 306, and displays the search result 318 on the screen 300. If the mobile terminal 100 receives the entire search result of the text query 306 from the search engine, the mobile terminal 100 can simply select a search result corresponding to the selected domain and display the selected search result on the screen 300. If only the search results for the main domain of the text query 306 are received, the mobile terminal 100 may transmit the text query 306 and the selected domain to the search engine to obtain search results corresponding to the selected domain have.

FIG. 7 is a flowchart illustrating a process of providing a search result for a query input by a user according to an embodiment of the present invention. As the user perceives, the process may proceed immediately, almost instantly, progressively, or at another appropriate rate. Once the process is activated, the process may be controlled automatically by a device such as the mobile terminal 100, or may be controlled by the user, independently of the user. Although the flow of the process described below includes a number of actions that appear to occur in a particular order, this process may be performed serially or in parallel (e.g., using a parallel processor or a multi-threading environment) , ≪ / RTI > more or fewer operations.

The description of steps 110 through 160 shown in FIG. 7 is the same as the description of steps 110 through 160 shown in FIG. 3, so that a description thereof will be omitted. Subsequently, the mobile terminal 100 determines whether a search result exists (S170). If there is no search result, the mobile terminal 100 indicates that there is no search result (S190). However, if there is a search result, the mobile terminal 100 displays a search result corresponding to the text query (S180).

Then, the mobile terminal 100 displays the edit item on the screen and monitors whether the displayed edit item is selected (S310). In one embodiment, edited items displayed on the screen may be selected according to the gesture of dragging and dropping the text query displayed on the screen to the edit item. When the edit item is selected, the mobile terminal 100 provides an interface for editing the text query (S320). The mobile terminal 100 may provide an interface for editing a text query, for example, by displaying a virtual keyboard on the screen.

8A to 8E are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to an embodiment of the present invention.

8A, when the user selects the voice query item 402 displayed on the screen 400, the mobile terminal 100 enters a listening mode and displays a message 404 indicating that the user has entered the listening mode, ). Then, upon receiving the voice query of the user, the mobile terminal 100 converts the voice query into a text query, and causes the text query to be executed in the search engine. The mobile terminal 100 obtains a search result corresponding to the main domain of the text query converted from the search engine, and displays the search result on the screen 400. Here, the mobile terminal 100 may display the converted text query 406 on the screen 400. If there is no search result, the mobile terminal 100 may display a message 408 on the screen 400 indicating that there is no search result. In such a state, the user can display an item capable of modifying the text query. For example, the user may touch the portion of the screen 400 where the text query 406 is displayed.

Referring to FIG. 8B, when the user touches a portion where the text query 406 is displayed on the screen 400, the mobile terminal 100 displays an item 410 for editing the text query on the screen 400 . The item 410 in which the text query can be edited is a selectable item, for example, an icon or a menu.

Referring to FIG. 8C, a user may drag and drop a portion where the text query 406 is displayed to an item 410 in which a text query can be edited. This means the selection of an item 410 in which a text query can be edited and can be an intuitive user interface that associates a text query 406 with an item 410 for editing a text query.

Referring to FIG. 8D, the mobile terminal 100 may provide a user interface for editing a text query. For example, the mobile terminal 100 displays the virtual keyboard 412 on the screen 400 and locates the cursor in the text query 406. The user can then directly modify the text query 406 via the virtual keyboard 412. [

Referring to FIG. 8E, the mobile terminal 100 obtains a search result 416 corresponding to the modified text query 414, and displays the search result 416 on the screen 400.

FIG. 9 is a flowchart illustrating a process of providing a search result for a query input by a user, according to an embodiment of the present invention. As the user perceives, the process may proceed immediately, almost instantly, progressively, or at another appropriate rate. Once the process is activated, the process may be controlled automatically by a device such as the mobile terminal 100, or may be controlled by the user, independently of the user. Although the flow of the process described below includes a number of actions that appear to occur in a particular order, this process may be performed serially or in parallel (e.g., using a parallel processor or a multi-threading environment) , ≪ / RTI > more or fewer operations.

The description of steps 110 to 160 shown in FIG. 9 is the same as the description of steps 110 to 160 shown in FIG. 3, and a description thereof will be omitted. Subsequently, the mobile terminal 100 determines whether a search result exists (S170). If there is no search result, the mobile terminal 100 indicates that there is no search result (S190). However, if there is a search result, the mobile terminal 100 displays a search result corresponding to the text query (S180).

Then, the mobile terminal 100 displays the deletion item on the screen and monitors whether the deletion item displayed is selected (S410). In one embodiment, a modification item displayed on the screen may be selected according to the gesture of dragging and dropping the text query displayed on the screen to the deletion item. When the delete item is selected, the mobile terminal 100 provides an interface for re-entering the voice query (S420). The mobile terminal 100 can visually or audibly output a message indicating that the mobile terminal 100 is in the listening mode, for example, and return to the initial state of the voice query function.

10A to 10D are conceptual diagrams illustrating a process in which the mobile terminal 100 provides a search result for a query input by a user according to an embodiment of the present invention.

10A, when the user selects the voice query item 502 displayed on the screen 500, the mobile terminal 100 enters a listening mode and displays a message 504 indicating that the user has entered the listening mode on the screen 500 ). Then, upon receiving the voice query of the user, the mobile terminal 100 converts the voice query into a text query, and causes the text query to be executed in the search engine.

Here, the mobile terminal 100 can convert the voice query into a text query according to a predetermined criterion. For example, by selecting one text query having the most similar pronunciation among a plurality of text queries similar to the pronunciation of the voice query, You can convert the query to a text query.

The mobile terminal 100 acquires search results corresponding to the main domain of the text query converted from the search engine and displays the search results on the screen 500. Here, the mobile terminal 100 may display the converted text query 506 on the screen 400. If there is no search result, the mobile terminal 100 may display a message 508 on the screen 500 indicating that there is no search result. In such a state, the user can display an item that can re-recognize the text query. For example, the user may touch the portion of the screen 500 where the text query 506 is displayed.

Referring to FIG. 10B, when the user touches a portion where the text query 506 is displayed on the screen 500, an item 510 for re-recognizing the text query is displayed on the screen 500. The item 510 that can re-recognize the text query can be a selectable item, e.g., an icon or a menu.

Referring to FIG. 10C, a user may drag and drop the portion where the text query 506 is displayed to an item 510 in which the text query can be re-recognized. This means the selection of an item 510 that can re-understand the text query and can be an intuitive user interface that associates the text query 506 with the item 510 that can re-understand the text query.

Referring to FIG. 10D, the mobile terminal 100 may convert the voice query back into a new text query. For example, the mobile terminal 100 may select a text query that is similar to the currently selected text query among a plurality of queries similar in pronunciation to the voice query. The mobile terminal 100 obtains the search result 514 corresponding to the text query 512 re-recognized as described above and displays it on the screen 500.

11 is a conceptual diagram illustrating a process in which the electronic device 700 provides a search result for a query input by a user according to an embodiment of the present invention.

In one embodiment, the voice query received from the user 800 is converted into a text query before being executed in the search engine 900. For example, a voice query received via voice input 702, such as a microphone, may be recognized as text before the query is executed in at least one search engine 900, including a text search engine configured to process text queries have.

The electronic device 700 is configured to receive and process queries via a text input 706, such as a voice input 702, a keyboard, such as a microphone. The voice input unit 702 is activated to receive the voice query, and the text input unit 706 is activated to receive the text query. However, the search engine 900 on which the query is executed may include a text search engine configured to receive only text queries. Therefore, the electronic device 700 includes a query conversion unit 704, which is configured to receive a voice query and recognize it as a text query. The text query is transmitted to the query execution unit 708 for subsequent processing.

The electronic device 700 performs an additional process to improve the reception and processing of the query. For example, the electronic device 700 is set to receive a voice query, but some terms in the voice query may be difficult to discern due to variations in pronunciation. Thus, a voice query term database that includes a range of pronunciations of such terms can correlate variably pronounced query terms with common query terms that represent related technologies. The voice query term database may be used to correlate voice terms of a voice query with accepted text terms of a text query for transmission to a text search engine 900. [ The query transformer 704 can thus convert the voice query into a text query by identifying the voice query terms in the voice query term database and converting such voice query terms into identified general text query terms.

Alternatively or additionally, processing of the query may be accomplished by parsing the query according to the language model according to the domain of the query. For example, a language model may be designed to process a voice query according to a language that is generally available while the user speaks a voice query.

12 illustrates an exemplary embodiment for processing the reception of voice query 1002 in accordance with speech recognizer 712. [ The speech recognizer may include an n-gram language model 718, which is set up to identify query terms based on words of a public three-language sequence within the language model. The n-gram language model 718 can be trained to recognize the query terms of the voice query 1002 using the speech query set and the corresponding text query as a training set, Lt; / RTI > The speech recognizer 712 may include a pronunciation model 720 that includes a representation of the pronunciation of the various query terms of the speech query 1002. The speech recognizer 712 may also include an acoustic model 716 that identifies the phonemes from the acoustic input that includes the speech query 1002. Phonemes identified by the acoustic model 716 may be matched to individual query terms according to the pronunciation model 720 and may be parsed into statistically probable query terms according to the n-gram language model 718. [ The speech recognizer 712 may therefore generate a recognition result 1004 that may be provided as a result that includes a text query 1006. [

In addition, the translated or recognized query may not be accurate enough to be relied upon during the conversion of the voice query 1002 to the text query 1006. For example, the voice query 1002 translated into the text query 1006 may not be accurate enough. Thus, the electronic device 700 can be configured to verify the translated query after converting the query and before executing the translated query in the search engine 900. [ If the user confirms the transformed query, the transformed query may be executed in the search engine 900, but if the user does not acknowledge the transformed query, alternative parsing of the query may be proposed and / May be entered again. Those skilled in the art will be able to devise various improved embodiments in connection with reception and normalization in implementing the techniques disclosed herein.

A second aspect that may be changed in embodiments of the techniques relates to a search engine 900 on which queries may be executed. In a first example, various search engines 900, e.g., a web search engine configured to search web pages on the Internet or other network, a file search engine configured to search for files stored on a local or network file system, An object search engine configured to search for objects stored in a database, or a database search engine configured to search for records in a database. In a second example, the various search engines may be configured to accommodate different types of queries, such as natural language queries, keyword queries, mathematical queries, or queries written in a programming language. In a third example, the various search engines 900 may support various logical configurations such as field-based constraints, Boolean logic, or programming logic. Many types of queries can be executed in various forms of search engine 900 while implementing the techniques disclosed herein.

When the text query is executed in the search engine 900, the search engine 900 returns the search results to the query execution unit 704. The search results may be obtained from a data source such as a web page, a file, an object, or a record. Search results can be associated with each domain, which can be predefined as an index representing a general category in which any search result may be associated. However, even if the domain is not covered within a predefined range, it may be newly created according to the attribute of the search result. A domain can be obtained by identifying semantic information associated with a subject included in the search result. For example, various forms of content may include metadata that identifies the meaning embedded in such content, such as an email address, email address, or telephone number, an alias used in the messaging service, and the like. The electronic device 700 may involve reading the semantic information from the metadata and matching it to a predefined query domain.

Alternatively, the domain may be obtained by an identifiable special attribute of the search result. For example, the search results may include regular expressions, heuristics, machine-learning classifiers, and so on, where the first character is capitalized and is typically structured, such as a combination of two or more words, Or may be matched to the domain through a pattern matching technique. Alternatively, a domain can be an important factor in the process of retrieving search results. For example, a name mentioned in the title of the search result or remarkably mentioned in the associated content can be estimated as a domain. Alternatively, a natural language parser may be used to evaluate content related to the current events and to select a domain for the search results. Alternatively, a statistical classifier, such as a neural network or Bayesian classification function, may be used to determine the domain of the search results. For example, the domain may be determined by the user by identifying the domain in which the URL was clicked.

It should be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

Mobile terminal: 100 Electronic device: 700

Claims (10)

Receiving a voice query from a user, converting the voice query into a text query, and displaying the converted text query;
Selecting and displaying a first search result corresponding to at least one of a user's current context and a user's application usage history among the search results by executing the text query to obtain a search result;
Displaying a plurality of domains included in the search result according to a user's selection in a state where the first search result is displayed;
Receiving an input associating the text query with a domain of one of the plurality of domains; And
And displaying a second search result corresponding to the one domain,
Wherein the displaying the plurality of domains comprises:
Identifying semantic information associated with each of the first search results from metadata including semantic information related to each of the first search results;
Acquiring a plurality of domains corresponding to the semantic information by matching the identified semantic information with a predefined query domain; And
And displaying the acquired plurality of domains. 2. The method of claim 1, wherein the input associating the text query with a domain of one of the plurality of domains comprises:
And dragging and dropping the text query into the one domain. The method of claim 1, wherein the displaying of the plurality of domains comprises:
And displaying a plurality of domains included in the first search result when receiving an input for selecting the text query. delete The method as claimed in claim 1,
And candidate domains of the text query. The method of claim 1, further comprising: displaying an item capable of editing the text query;
Receiving an input associating the text query with an item capable of editing the text query; And
Further comprising the step of providing an interface for editing the text query. 7. The method of claim 6, wherein the input associating the text query with an item capable of editing the text query comprises:
Wherein the text query is an input for dragging and dropping the text query into an item capable of editing the text query. 2. The method of claim 1, further comprising: displaying an item capable of re-recognizing the text query;
Receiving an input that associates the text query with an item that can re-recognize the text query; And
Further comprising the step of re-recognizing the text query. 9. The method of claim 8, wherein the input associating the text query with an item that re-
Wherein the text query is an input for dragging and dropping the text query into an item capable of re-recognizing the text query. A microphone for receiving a voice query from a user;
Selecting a first search result corresponding to at least one of a current context of the user and an application usage history of the user among the obtained search results; and converting the voice query into a text query, executing the text query to obtain a search result, And displaying a plurality of domains corresponding to the first search result according to a user's selection in a state where the first search result is displayed; And
A touch-sensitive display for displaying the text query and the plurality of domains, receiving an input associating the text query with a domain of one of the plurality of domains, and displaying a second search result corresponding to the one domain ≪ / RTI &
Wherein,
Identifying semantic information related to each of the search results from metadata including semantic information related to each of the first search results, matching the identified semantic information with a previously defined query domain, And obtains a plurality of domains.

Images