KR20200126357A - Server and control method thereof, and image processing apparatus and control method thereof - Google Patents

Abstract

According to an embodiment of the present invention, a server comprises: a communication interface that communicates with one or more image processing devices; a storage for storing data; a processor that provides a result of processing a first event to an image processing device when the first event by the utterance of a user is received from the image processing device, and stores the history of the first event in the storage according to the processing of the first event, wherein the processor determines a relation between the first event and the second event when the second event by non-utterance user input is received from the image processing device, and when it is determined that there is a relation between the first event and the second event, processes a second event based on the history of the first event stored in the storage.

Description

Server and its control method, image processing device and its control method {SERVER AND CONTROL METHOD THEREOF, AND IMAGE PROCESSING APPARATUS AND CONTROL METHOD THEREOF}

The present invention relates to a server in a system that processes a video signal to be displayed as an image, a control method thereof, an image processing apparatus, and a control method thereof, and in particular, a function corresponding to the voice command by recognizing a user's voice command or The present invention relates to a server having a structure capable of executing an operation and a control method thereof, and an image processing apparatus and a control method thereof.

The image processing apparatus processes image signals/image data received from the outside according to various image processing processes. The image processing apparatus may display the processed image signal as an image on a display panel having its own, or may output the processed image signal to the corresponding display device so that it is displayed as an image on another display device including the panel. That is, if the image processing apparatus is a device capable of processing an image signal, it may include both a panel capable of displaying an image and a panel capable of displaying an image. As an example of the former case, there is a TV, and the latter case An example of this is a set-top box.

As the image processing apparatus continues to reflect the addition and expansion of various functions according to the development of technology, in accordance with this trend, the configuration of inputting a command reflecting the user's intention in the image processing apparatus to the image processing apparatus has various structures or A method is being proposed. For example, in the related art, when a key/button on a user's remote controller is pressed, the remote controller transmits a control signal to the image processing apparatus that causes the user to execute a desired operation. Various configurations have been proposed for controlling an image processing apparatus by reflecting a user's intention, such as detecting a motion or utterance caused by, and analyzing the detected content to execute a corresponding action.

A server according to an embodiment of the present invention includes a communication interface for communicating with one or more image processing apparatuses; A storage for storing data; When a first event due to a user's utterance is received from the image processing device, the processing result performed on the first event is provided to the image processing device, and the history of the first event is stored according to the processing of the first event. And a processor storing the storage in the storage, wherein the processor determines a relationship between the first event and the second event when a second event is received from the image processing apparatus according to a user input of a non-ignition method, and the If it is determined that the first event and the second event are related, the second event is processed based on a history of the first event stored in the storage.

Here, when the second event is received, the processor refers to the history of the first event stored in the storage, and whether the second event can be processed independently of the first event or the processing result of the first event It may be determined whether to process the second event or not.

In addition, when the first event is received, the processor performs a search in a preset database based on the first event, and receives the second event while a plurality of search results are derived from the first event. Then, one final result selected based on the second event from among the plurality of search results for the first event may be transmitted to the image processing apparatus.

Here, if there are a plurality of search results for the first event, the processor transmits a request command for requesting selection of one of the plurality of search results and the plurality of search results to the image processing apparatus, and , The second event corresponding to the request command may be received from the image processing apparatus.

In addition, the second event may be converted into a form of a voice command according to an utterance by the image processing apparatus so that the server can recognize it and received by the server.

In addition, a method for controlling a server according to an embodiment of the present invention includes the steps of: receiving a first event by a user's utterance from an image processing apparatus; Providing a result of processing performed on the first event to the image processing apparatus, and storing a history of the first event according to the processing of the first event; Receiving a second event from the image processing apparatus according to a user input of a non-speech method; Determining the relationship between the first event and the second event, and if the first event and the second event are determined to be related, processing the second event based on a previously stored history of the first event It characterized in that it comprises a.

Here, the step of determining the relationship between the first event and the second event includes whether or not the second event can be processed independently of the first event by referring to the previously stored history of the first event. It may include the step of determining whether to process the second event based on the processing result of the first event.

In addition, the providing of the processing result performed on the first event to the image processing apparatus includes performing a search in a preset database based on the first event, and the pre-stored first event In the processing of the second event based on the history of, when the second event is received while a plurality of search results are derived from the first event, among the plurality of search results for the first event And transmitting one final result selected based on the second event to the image processing apparatus.

Here, in the receiving of the second event, if there are a plurality of search results for the first event, the plurality of search results and a request command for requesting selection of any one of the plurality of search results are provided. Transmitting to an image processing apparatus; And receiving the second event corresponding to the request command from the image processing apparatus.

In addition, the second event may be converted into a form of a voice command according to an utterance by the image processing apparatus so that the server can recognize it and received by the server.

In addition, an image processing apparatus according to an embodiment of the present invention includes: a storage for storing data; A user interface including a speech input interface into which a user's speech is input, and a non-speech input interface through which a user input in a non-speech manner is performed; A processor that processes an operation corresponding to a first event generated by the speech input interface to be executed; The history of the occurrence and processing of the first event is stored in the storage, and when a second event is generated by the non-ignition input interface after the first event occurs, between the first event and the second event And a controller configured to control the second event to be processed based on a history of the first event stored in the storage when determining the relevance and determining that the first event and the second event are related. .

Here, when the second event occurs, the controller refers to the history of the first event stored in the storage, and whether the second event can be processed independently of the first event or the processing result of the first event It may be determined whether to process the second event or not.

In addition, when a plurality of results are searched for an operation corresponding to the first event, the controller displays the plurality of search results in a selectable manner, and the second event is displayed while the plurality of search results are displayed. When it occurs, it may be determined whether the second event indicates selection of one of the plurality of search results.

Here, the communication interface further comprises a communication interface that communicates with a server that performs a search process for a voice command corresponding to the utterance, wherein the controller transmits the first event to the server and retrieves the search result for the first event. It can be received from the server and displayed on the display.

In addition, the controller may convert the second event into a form of a voice command according to an utterance so that the server can recognize the second event and transmit it to the server.

In addition, a method of controlling an image processing apparatus according to an embodiment of the present invention includes the steps of: generating a first event by a speech input interface into which a user's speech is input; And executing an operation corresponding to the first event, and executing the operation corresponding to the first event may include: storing a history of occurrence and processing of the first event in the storage; Generating a second event by a non-speech input interface in which a user input of a non-speech method is performed; To determine the relationship between the first event and the second event, and if the first event and the second event are determined to be related, controlling the second event to be processed based on the stored history of the first event It characterized in that it comprises a step.

Here, the step of determining the relationship between the first event and the second event includes whether or not the second event can be processed independently of the first event by referring to the previously stored first event history. It may include the step of determining whether to process the second event based on the processing result of the first event.

In addition, the step of executing the operation corresponding to the first event further includes, when a plurality of results are searched for the operation corresponding to the first event, displaying the plurality of search results in a selectable manner, The determining of the relationship between the first event and the second event may include determining whether the second event indicates selection of any one of the plurality of search results.

Here, the image processing apparatus communicates with a server that performs a search process for a voice command corresponding to an utterance, and transmits the first event to the server to receive and display the search result for the first event from the server. can do.

In addition, the image processing apparatus may convert the second event into a form of a voice command according to an utterance so that the server can recognize the second event and transmit it to the server.

In addition, an image processing apparatus according to an embodiment of the present invention includes a user interface including a speech input interface through which a user's utterance is input, and a non-speech input interface through which a user input of a non-speech method is performed; A communication interface that communicates with a server for searching for an operation corresponding to a voice command by speech; A processor that receives a search result for a first event generated by the speech input interface from the server and processes the search result to be executed; When a second event instructing to select any one of the plurality of search results occurs through the non-speech input interface in a state in which a plurality of search results are derived for the first event by the server, the server A controller that converts the second event into a voice command recognizable by the server and transmits it to the server so that a final search result can be provided by selecting any one of the plurality of search results based on the second event It characterized in that it comprises a.

1 is a block diagram of an image processing apparatus included in a system according to a first embodiment of the present invention;
2 is a block diagram showing the configuration of a server included in the system according to the first embodiment of the present invention;
3 and 4 are flowcharts illustrating a method of providing a search result corresponding to an event to a display device by the interactive server of FIG. 2;
5 is a block diagram showing a configuration of a server-side processor of the interactive server of FIG. 2 divided by function;
6 and 7 are exemplary views showing a data transmission relationship between the display device and the interactive server of FIG. 2 and each component in the interactive server;
8 and 9 are flowcharts showing a method of controlling a display device according to a second embodiment of the present invention;
10 is a block diagram of a configuration of a processor of a display device according to a third embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, only configurations directly related to the spirit of the present invention will be described, and descriptions of other configurations will be omitted. However, in implementing an apparatus or system to which the spirit of the present invention is applied, it should be noted that this does not mean that a configuration omitted from the description is unnecessary.

1 is a block diagram of an image processing apparatus 100 included in a system according to a first embodiment of the present invention.

In the following embodiment, a case where the image processing apparatus 100 is implemented as a display device having a structure capable of displaying an image by itself, for example, a TV, will be described. However, the idea of the present invention can be applied even when the image processing apparatus 100 is a device having a structure capable of outputting image data or control information to other display devices without displaying an image on its own. It is not limited to the embodiments described below. That is, the idea of the present invention can be implemented in various types of devices by applying various changes.

As shown in FIG. 1, the image processing apparatus 100 or the display apparatus 100 according to the present embodiment receives an image signal from an external image supply source (not shown). The type or characteristic of the image signal that can be received by the display device 100 is not limited. For example, the display device 100 receives a broadcast signal transmitted from a transmission equipment (not shown) of a broadcasting station, and the corresponding broadcast signal You can display the broadcast video by tuning.

The display apparatus 100 includes a communication interface 110 that communicates with the outside to transmit and receive data/signals, and a processor that processes data received through the communication interface 110 according to a preset process. 120, a display 130 for displaying the image data as an image when the data processed by the processor 120 is image data, and a user interface 140 for performing an input operation by a user; , A storage 150 for storing data/information, and a controller 160 for controlling general operations of the display apparatus 100.

The communication interface 110 transmits and receives data so that the display device 100 can communicate with an external device (not shown) such as a server. The communication interface 110 connects to an external device (not shown) through a wired/wireless wide area/local area network or a local access method according to a preset communication protocol.

The communication interface 110 may be implemented by a collection of connection ports or connection modules for each device, and a protocol for connection or an external device to be connected is not limited to one type or type. Does not. The communication interface 110 may be embedded in the display device 100, and all or part of the configuration is additionally installed in the form of an add-on or dongle to the display device 100 It is also possible.

Since the communication interface 110 transmits and receives signals according to a protocol designated for each connected device, signals may be transmitted and received for each connected device based on an individual connection protocol. In the case of image data, for example, the communication interface 110 includes a radio frequency (RF) signal, a composite/component video, a super video, SCART, a high definition multimedia interface (HDMI), and a display. Signals can be transmitted/received based on various standards such as DisplayPort, UDI (unified display interface), or wireless HD.

The processor 120 performs various processes on data/signals received through the communication interface 110. When image data is received by the communication interface 110, the processor 120 performs an image processing process on the image data, and outputs the image data on which this process has been performed to the display 130, thereby Allows an image based on data to be displayed. When the signal received by the communication interface 110 is a broadcast signal, the processor 120 extracts video, audio and additional data from the broadcast signal tuned to a specific channel, adjusts the video to a preset resolution, and displays the display 130. To be displayed.

The type of the image processing process performed by the processor 120 is not limited, for example, decoding corresponding to the image format of the image data, and converting the image data of the interlace method into a progressive method. De-interlacing, scaling to adjust image data to a preset resolution, noise reduction to improve image quality, detail enhancement, frame refresh rate conversion, etc. It may include.

Since the processor 120 can perform various processes according to the type and characteristic of data, the process that the processor 120 can perform cannot be limited to an image processing process. In addition, data that can be processed by the processor 120 cannot be limited to being received by the communication interface 110. For example, when a user's utterance is input through the user interface 140, the processor 120 may process the utterance according to a preset voice processing process.

The processor 120 is an image processing board in which a system-on-chip (SOC) incorporating these various functions or individual chipsets capable of independently performing each of these processes are mounted on a printed circuit board. It is implemented as (not shown) and is built into the display device 100.

The display 130 displays the image signal/image data processed by the processor 120 as an image. The implementation method of the display 130 is not limited, and includes liquid crystal, plasma, light-emitting diode, organic light-emitting diode, and surface-conducting electron gun. Conduction electron-emitter), carbon nano-tubes, nano-crystals, etc. can be implemented in various display methods.

The display 130 may additionally include an additional component according to the implementation method. For example, in the case of a liquid crystal system, the display 130 includes a liquid crystal display panel (not shown), a backlight unit (not shown) supplying light thereto, and a panel driving substrate (not shown) for driving the panel (not shown). Poem), etc.

The user interface 140 transmits various preset control commands or information to the controller 160 according to a user's manipulation or input. The user interface 140 converts various events generated by the user according to the user's intention and transmits the information to the controller 160. Here, the event generated by the user can be in various forms, for example, the user's manipulation, speech, gesture, and the like.

The user interface 140 is provided to detect information input in a corresponding manner in response to a method in which a user inputs information. Accordingly, the user interface 140 can be classified into an interactive input interface 141 and a non-interactive input interface 142.

The interactive input interface 141 is provided to input a user's speech. That is, the interactive input interface 141 is implemented as a microphone, and senses various sounds generated in an external environment of the display apparatus 100. The interactive input interface 141 can generally detect speech by a user, but can also detect sounds generated by various environmental factors other than the user.

The non-interactive input interface 142 is provided to perform user input other than a speech method. In this case, the implementation form of the non-interactive input interface 142 is various, for example, a remote controller separated from the display device 100 or a menu key installed outside the display device 100. -key) or an input panel, a motion sensor or a camera that detects a user's gesture, etc. are possible.

Alternatively, the non-interactive input interface 142 may be implemented as a touch-screen installed on the display 130. In this case, the user may cause a preset command or information to be transmitted to the controller 160 by touching the input menu or UI image displayed on the display 130.

The storage 150 stores various data under the control of the controller 160. The storage 150 is implemented as a nonvolatile memory such as a flash-memory and a hard-disc drive so that data can be stored regardless of whether or not system power is provided. The storage 150 accesses the controller 160 to read, write, modify, delete, or update previously stored data.

The controller 160 is implemented as a CPU and controls the operation of all components of the display apparatus 100 including the processor 120 according to the occurrence of a predetermined event. For example, in the case of a user utterance event, the controller 160 controls the processor 120 to process the input utterance when a user utterance is input through the interactive input interface 141. If the user speaks the channel number, the controller 160 controls the broadcast image to be switched and displayed with the spoken channel number.

Here, the process of processing the input speech may be performed by the display apparatus 100 alone, but may be performed by sharing with an external device in consideration of various factors such as system load or processing capability. This configuration will be described below.

2 is a block diagram showing the configuration of servers 200 and 300 included in the system of the present embodiment.

As shown in FIG. 2, the display device 100 is connected to enable communication with one or more servers 200 and 300 through a network. Here, the display device 100 includes a speech-to-text (STT) server 200 that converts a user's speech into a voice command, and an action/action/result corresponding to the voice command by analyzing the voice command. Each of them is connected to the interactive server 300 for determining the. Here, the names of the servers 200 and 300 are applied only for mutual identification in consideration of each function, and the corresponding names do not limit the constitutional characteristics of each server 200 and 300.

The STT server 200 generates the contents of the voice signal as text by analyzing the waveform of the voice signal. Accordingly, when a voice signal is received from the display apparatus 100, the STT server 200 converts it into a voice command of text. Here, the utterance refers to an action performed by the user, and the voice command refers to the content of the utterance being data in text format.

The interactive server 300 searches for operations of various display devices 100 corresponding to voice commands. The interactive server 300 analyzes the voice command received from the display device 100, and according to the analysis result, a control signal for performing an operation corresponding to the voice command or search result information based on the voice command. It is transmitted to the display device 100.

The interactive server 300 includes a server-side communication interface 310 for communicating with the display device 100, and a server-side processor 320 that processes voice commands received through the server-side communication interface 310. Includes.

An operation performed when a user of the display apparatus 100 makes a speech under the system of the display apparatus 100, the STT server 200, and the interactive server 300 will be described below.

When the user's utterance is input, the display apparatus 100 transmits a voice signal according to the utterance to the STT server 200. The STT server 200 converts the voice signal received from the display device 100 into a data-formed voice command, and transmits the voice command to the display device 100. The display device 100 transmits the voice command received from the STT server 200 to the interactive server 300.

The interactive server 300 analyzes the contents of the voice command, derives a result corresponding to the voice command, and returns it to the display apparatus 100. For example, when the content of the voice command is "Who is the performer of'program ABC' this week?", the interactive server 300 may use "this week", "program ABC", and the like in an internal or external database. A search is performed with the keyword of "performer" and the result is obtained.

Here, since the database is structured for each keyword category, it is possible to search for a desired result with a specific keyword. The structured database is a matter that can be implemented in various ways depending on the design and construction of the database, and detailed descriptions thereof will be omitted.

However, when the interactive server 300 performs a search using one or more keywords derived from a voice command, a plurality of search results may appear. As an example of the previous voice command, a case in which "program ABC", a broadcast program to be broadcast this week, broadcasts at different times or different broadcast channels may be considered.

In this case, the interactive server 300 needs to select any one search result suitable for the user's intention from among a plurality of search results for the voice command. As a corresponding method for the interactive server 300 to select one search result, for example, the following two methods are possible.

In the first method, any one search result is selected according to a specific rule established by the client of the interactive server 300, that is, the display apparatus 100 itself. For example, if the voice command is a content for inquiring about the broadcast time of a specific program, and there are a plurality of programs that satisfy such a condition, the most recently broadcasted program may be selected. In the second method, when the user of the display apparatus 100 makes an utterance for selecting any one of the search results, the interactive server 300 connects the utterance with the previous utterance history to obtain a search result suitable for the user's original intention. You can choose.

However, in such a system, the user is provided with environments of various input methods, so even if the display apparatus 100 supports the voice recognition input environment, the user selects and inputs the most convenient method in each situation through various input methods. Will perform. In an environment of a system supporting such a multiple input mode, if history management is individually performed between multiple inputs of a user, there will be limitations in selecting or deriving a result that reflects the user's original intention. If the user initially performs utterance input and then changes the input method in a different way, but the original intended result is not provided, the user may feel uncomfortable with the use environment.

Therefore, in the interactive system environment by the interactive server 300 and the display device 100, there are a plurality of results initially searched by the user's utterance, and then the user inputs any one search result through the non-speech method input. When selecting, the interactive system needs to provide a search result that matches the intention of the user's initial speech.

Here, the system according to the prior art does not integrally support the history management for each of these multiple input modes. That is, in the interactive server 300 according to the prior art, input by the user's utterance of the display device 100, the input by the user's gesture, and the input by the user's remote controller (not shown) are individually input. It is managed as, and each of these inputs is not managed by interlinking.

In the system according to the prior art, a state in which the display apparatus 100 displays a plurality of search results received from the interactive server 300 according to the second method is considered. At this time, if the user does not speak, but may perform input in a non-speech method through the non-conversational input interface 142, the interactive server 300 according to the prior art inputs between the speech input method and the non-speech input method Since the history of is not referred to in conjunction with each other, results that do not fit the user's intention are produced.

For example, consider a case in which a user utters "record'program ABC'" and a plurality of contents related to'program ABC' are displayed on the display apparatus 100. In the conventional case, recording is performed by selecting one content normally when the user selects one utterance, whereas when one is selected as a separate non-speech input method other than utterance, only the content is selected. Is performed, but no recording is performed. This is because information by the non-speech input by the user is not transmitted to the interactive server 300, and thus the interactive server 300 cannot perform an operation after the speech.

In consideration of this point, the following method is proposed according to this embodiment.

An event generated by the interactive input interface 141 of the display apparatus 100 according to a user's input is referred to as a first event, and an event generated by the non-interactive input interface 142 is referred to as a second event for convenience. . When a first event occurs from the display device 100, the interactive server 300 processes the first event and stores the history of the first event, and when the second event occurs, the previously stored first event The second event is processed based on the processing history. Here, when the second event occurs, the display device 100 converts the second event into a form of a voice command so that the interactive server 300 provided to recognize the voice command can recognize the second event, It is transmitted to the type server 300.

When determining an action for a predetermined event, the interactive server 300 may refer to a history of previously occurring events. The interactive server 300 does not individually manage the events generated by each of the interactive input interface 141 and the non-interactive input interface 142 of the display device 100, but manages the newly occurring events. When determining a corresponding action, an event that has previously occurred may be referred to regardless of whether the interactive input interface 141 and the non-interactive input interface 142 are present.

For example, if there are a plurality of search results corresponding to the user's initial speech, the interactive server 300 provides the plurality of search results so that the user can select them. At this time, even if the user inputs an instruction for selecting any one of the search results in a non-ignition method, the interactive server 300 manages the event input through the integrated management of the new event even if the input method is different from the previous event. Can be correlated. Accordingly, a search result desired by the user may be derived and provided to the user.

3 and 4 are flowcharts illustrating a method for the interactive server 300 to provide a search result corresponding to an event to the display apparatus 100.

As shown in FIG. 3, in step S100, the interactive server 300 receives a first event, which is a voice command by speech, from the display device 100.

In step S110, the interactive server 300 derives a search keyword by analyzing the first event.

In step S120, the interactive server 300 performs a search using the derived search keyword, and stores the occurrence and processing history of the first event.

In step S130, the interactive server 300 determines whether there are multiple search results.

If it is determined that there are not a plurality of search results, the interactive server 300 transmits the search results to the display apparatus 100 and ends in step S140.

On the other hand, if it is determined that there are a plurality of search results, in step S150, the interactive server 300 transmits the plurality of search results to the display device 100, and the display device 100 selects one of the plurality of search results. ).

As illustrated in FIG. 4, in step S160, the interactive server 300 receives a second event including a selection instruction from the display apparatus 100. Here, the second event is generated by a non-ignition type user input in the display apparatus 100.

In step S170, the interactive server 300 determines whether the relationship between the first event and the second event, that is, whether the second event is to be processed dependently on the first event, by referring to the previously stored occurrence and processing history of the first event. Judge. For example, the interactive server 300 determines whether the second event is a selection instruction command for a plurality of search results searched by the first event.

If it is determined in step S180 that the second event is not related to the first event, the interactive server 300 analyzes and processes the second event independently from the first event in step S190.

On the other hand, if it is determined in step S180 that the second event is a selection instruction for a plurality of search results searched by the first event, the interactive server 300 selects the search result according to the second event in step S200. In this process, the interactive server 300 may perform an additional search depending on whether or not it is necessary.

In step S200, the interactive server 300 transmits a final search result according to the first event and the second event to the display apparatus 100.

Hereinafter, a specific internal configuration of the interactive server 300 will be described with reference to FIG. 5.

5 is a block diagram showing the configuration of the server-side processor 320 of the interactive server 300 divided by function.

As shown in FIG. 5, the server-side processor 320 of the interactive server 300 is a dialog engine 321 that analyzes voice commands received through the server-side communication interface 310. And, it includes a content manager 322 (content manager) that performs a search based on the search keyword received from the dialog engine 321.

The dialog engine 321 includes a dialog engine manager 323, a dialog agent 324, an action agent 325, and a database 326. The classification of these components for the server-side processor 320 or the dialogue engine 321 is applied for convenience by function, and does not mean that the server-side processor 320 or the dialogue engine 321 must have such a classification form. .

In addition, the above-described components may have a hardware configuration, but may be a software configuration, or may be applied by mixing hardware and software.

The conversation engine manager 323 delivers the event in the form of a voice command transmitted to the conversation engine 321 to the conversation agent 324. The conversation agent 324 analyzes the contents of the event, generates the analysis result as structured data, and returns it to the conversation engine manager 323.

Here, the structured data generated by the conversation agent 324 is referred to hereinafter as a dialog frame. The dialog frame includes the content of the voice command event, and is data structured by categories of the voice command so that other components such as the dialog engine manager 323, the action agent 325, and the content manager 322 can decode and refer to it. . The category and the form of structure may be variously determined according to the design method of the server-side processor 320.

The dialog frame includes information on the user's purpose in the voice command, attributes necessary to achieve the objective, and values of the attributes. A specific example of the conversation frame will be described later.

The conversation engine manager 323 delivers the conversation frame received from the conversation agent 324 to the action agent 325. The action agent 325 determines an action that fits the dialog frame received from the dialog engine manager 323 and returns the determination result to the dialog engine manager 323. That is, the action agent 325 determines an action corresponding to the event.

The action agent 325 may request information search by transmitting the search keyword extracted from the conversation frame to the content manager 322 when information search is required during the determination process. The content manager 322 performs a search on various types of search databases inside or outside the interactive server 300 with the received keyword, and returns the search result to the action agent 325.

When the dialog engine manager 323 receives an action or search result corresponding to an event from the action agent 325, the received result is transmitted to the server-side communication interface 310 to be transmitted to the display apparatus 100.

The database 326 includes all information that can be referenced by the conversation engine manager 323, the conversation agent 324, and the action agent 325 in the above process. For example, the database 326 determines context information for the conversation agent 324 to generate a conversation frame, event history information, conversation model information, and the action agent 325 to determine an action corresponding to the event. Action model information for, etc. may be included.

Under the above structure, an example of a specific operation when the interactive server 300 returns a result corresponding to a voice command received from the display device 100 to the display device 100 will be described.

6 and 7 are exemplary diagrams showing a data transmission relationship performed between the display apparatus 100 and the interactive server 300 and each component in the interactive server 300.

As shown in FIG. 6, when the user performs a speech of "Who is the performer of'program ABC' of this week?", the display apparatus 100 acquires a voice command corresponding to the speech. The voice command corresponding to this utterance is referred to as a first event. The display apparatus 100 transmits the first event to the interactive server 300 (410).

The conversation engine manager 323 transmits the first event received through the server-side communication interface 310 to the conversation agent 324 (420).

The conversation agent 324 creates a conversation frame corresponding to the first event by analyzing the first event. When the first event is the content of "Who is the performer of'program ABC' this week?", the dialogue frame may be expressed as follows.

(1) Purpose: Inquiry about performers of broadcast programs

(2) Title: program ABC

(3) Time: this week

In the above dialog frame, "Purpose", "Title", and "Time" are the condition categories of the conversation frame, and each condition category includes attribute values of "Inquiry about the performer of the broadcast program", "program ABC", and "This week". do. By analyzing the content of the first event, the conversation agent 324 may derive a category of conditions including keywords included in the first event and attribute values for each category.

Upon receiving the conversation frame thus generated from the conversation agent 324 (430), the conversation engine manager 323 transmits the conversation frame to the action agent 325 (440).

The action agent 325 creates the following keyword based on the received dialog frame and transmits it to the content manager 322 (450 ).

(1) Title: program ABC

(2) Time: 2013/06/03~2013/06/09

In the case of attribute values of the time condition category, a specific date must be specified for search. Accordingly, the action agent 325 determines a specific date and time interval of this week based on the current time, based on the attribute value of "this week" of the time category of the conversation frame.

The content manager 322 is a keyword delivered from the action agent 325 and performs a search on a search service for various contents and metadata. Search targets may include all databases, search services, and content services that the content manager 322 can access.

The content manager 322 transmits the search result obtained through the search to the conversation agent 324 (460). For example, the search result may include a plurality as follows.

(1) program ABC / 2013.06.04 17:00 / Channel 4

(2) program ABC / 2013.06.08 10:00 / Channel 20

(3) program ABC / 2013.06.09 20:00 / Channel 35

Each search result includes information on the program title, broadcast time, and broadcast channel, respectively, and further includes performer information, although not indicated above. According to the above search results, there are three cases of "program ABC" that is broadcast in "this week".

The action agent 325 transmits the search result to the dialog engine manager 323 (470). In this process, since the search results are plural, the action agent 325 adds a command for allowing the display apparatus 100 to select and display the plurality of search results.

The conversation engine manager 323 transmits the search result and command to the display apparatus 100 (480).

As shown in FIG. 7, the display apparatus 100 displays a plurality of search results so that the user can select it according to a command received from the dialog engine manager 323, and displays that one of the displayed search results is selected. The device 100 notifies the user by the system ignition.

Accordingly, the user inputs an instruction to select the first search result, for example, by a non-ignition method such as a button operation or a gesture of a remote controller (not shown). This input is referred to as a second event.

In response to the second event, the display apparatus 100 generates a fictitious voice command including contents of the second event. Here, the display apparatus 100 transmits the second event to the voice command similar to the first event so that the interactive server 300 provided to recognize the event in the form of a voice command by speech can recognize the second event. It is converted into a form and transmitted to the interactive server 300. The second event may include, for example, "No. 1".

The display apparatus 100 transmits the second event to the conversation engine manager 323 (510).

The conversation engine manager 323 transmits the second event to the conversation agent 324 (520).

Since the second event only includes the content of "No. 1", it is difficult for the conversation agent 324 to process the second event only with the content of the second event. Accordingly, the conversation agent 324 calls processing status information on the first event from the previously performed event processing history, and generates the following conversation frame based on the called information.

(1) Purpose: Inquiry about performers of broadcast programs

(2) Order: 1

In this dialog frame, the target category and its attribute values are not included in the second event, but the chat agent 324 may generate the chat frame of the above content by referring to the first event related to the second event.

The conversation agent 324 transmits the generated conversation frame to the conversation engine manager 323 (530), and the conversation engine manager 323 delivers the conversation frame to the action agent 325 (540).

The action agent 325 selects the first search result from among the three search results searched by the first event based on the attribute value of the order condition category in the dialog frame. The action agent 325 transmits the selection result of "program ABC / 2013.06.04 17:00 / channel 4" to the dialog engine manager 323 (550).

The dialog engine manager 323 transmits the received selection result to the display apparatus 100 (560). In this case, the conversation engine manager 323 may additionally transmit a system utterance command created by the conversation agent 324 to the display apparatus 100 with reference to the selection result.

According to the received command, the display apparatus 100 outputs a performer of “program ABC” broadcasted on 2013.06.04 17:00 on channel 4 through system utterance.

According to the above embodiment, in a system implemented by the display apparatus 100 and the interactive server 300, when a plurality of results satisfying targets and conditions input by the user through utterance are plural, the user may use a remote controller or a gesture. Even if inputs of a non-speech method such as, etc. are mixed, the user's convenience can be improved by providing a result suitable for the user's original intention.

On the other hand, in the above embodiment, a case in which the interactive server 300 stores the history of events by the user's utterance method input and non-speech method input of the display apparatus 100 has been described, but the spirit of the present invention This is not limited to this. For example, a configuration in which the display device () stores the event history is possible, and this will be described with reference to FIGS. 8 and 9.

8 and 9 are flowcharts illustrating a method of controlling the display apparatus 100 according to the second embodiment of the present invention.

As shown in FIG. 8, in step S300, the display apparatus 100 receives a user's utterance. In step S310, the display apparatus 100 transmits a voice command corresponding to the user's speech to the interactive server 300.

In step S320, the display apparatus 100 determines whether there are a plurality of response results of the voice command received from the interactive server 300.

If the response result of the voice command received from the interactive server 300 is one, in step S330, the display apparatus 100 displays the corresponding result.

On the other hand, if there are a plurality of response results of the voice command received from the interactive server 300, in step S340, the display apparatus 100 displays the plurality of response results so that the user can select it.

In step S350, the display apparatus 100 receives selection inputs for a plurality of corresponding results from a user. While the above process is in progress, the display apparatus 100 stores a history of occurrence and processing of an event of a voice command by speech.

As shown in FIG. 9, in step S360, the display apparatus 100 determines whether the received selection input is a speech method or a non-speech method.

In this process, the display apparatus 100 determines whether the selection input received in step S350 is related to the utterance generated in step S300 by referring to the previously stored event history. The display apparatus 100 performs step S360 when it is determined that there is a relationship between the two.

If the received selection input is a speech method, in step S370, the display apparatus 100 transmits a voice command corresponding to the selection input to the interactive server 300 and proceeds to step S400.

On the other hand, if the received selection input is a non-speech method, in step S380, the display apparatus 100 generates a virtual voice command corresponding to the content of the non-speech selection input. In step S390, the display apparatus 100 transmits the generated voice command to the interactive server 300.

In step S400, the display apparatus 100 receives and displays a result corresponding to the transmitted selection input from the interactive server 300.

Meanwhile, in the above embodiment, a case in which the interactive server 300 analyzes and searches a result corresponding to an event occurring in the display apparatus 100 has been described. However, the spirit of the present invention is not limited thereto, and an embodiment in which the display apparatus 100 performs at least one of the STT server 200 and the interactive server 300 in the first embodiment is also possible. .

10 is a block diagram showing the configuration of the processor 620 of the display device 600 according to the third embodiment of the present invention.

As shown in FIG. 10, the display device 600 according to the present embodiment includes a communication interface 610 and a processor 620. Other components included in the display device 600 may be applied to the first embodiment, and detailed descriptions thereof will be omitted.

The processor 620 includes a speech processing engine 621 that processes speech input to the display device 600, a dialogue engine 622 that analyzes a voice command corresponding to the speech, and the dialogue engine 622. It includes a content manager 623 that performs a search based on the received search keyword. Further, the conversation engine 622 includes a conversation engine manager 624, a conversation agent 625, an action agent 626, and a database 627. In the drawings and embodiments, only configurations directly related to the spirit of the invention are described, and illustration and description are omitted for other configurations.

When the user's utterance is input, the voice processing engine 621 transmits a voice signal of the utterance to the STT server 200, and receives a voice command corresponding to the utterance from the STT server 200, and the dialog engine manager 624 Pass it on.

Hereinafter, specific operations by the dialog engine manager 624, the dialog agent 625, the action agent 626, and the content manager 623 are substantially the same as the components of the same name described in the previous embodiment, so the description of the previous embodiment As can be applied, detailed descriptions are omitted.

Alternatively, the display device 600 may perform a function of the STT server 200 by the voice processing engine 621, that is, a function of converting speech into a voice command. In this case, when a user's speech is input, the speech processing engine 621 converts the speech into a voice command and transmits the converted speech to the dialogue engine manager 624.

The above-described embodiments are merely exemplary, and various modifications and equivalent other embodiments are possible for those of ordinary skill in the art. Therefore, the true technical protection scope of the present invention should be determined by the technical idea of the invention described in the following claims.

100: display device
110: communication interface
120: processor
130: display
140: user interface
141: interactive input interface
142: Non-interactive input interface
150: storage
160: controller
200: STT server
300: interactive server
310: server side communication interface
320: server-side processor

Claims (17)

In the electronic device,
A display unit;
Communication interface;
A user interface for receiving user input; And
Controlling the communication interface to transmit a first search request to a server based on a user utterance input received through the user interface,
Controlling the display unit to display first information, corresponding to the user utterance input, received through the communication interface,
While displaying the first information, based on a user input received through the user interface, a second search request that can be processed in relation to the first search request is processed with the server that processed the first search request. Controlling the communication interface to transmit to the same server,
A processor for controlling the display unit to display second information corresponding to the user input received through the communication interface,
The second information is an electronic device corresponding to a search result of the user input related to the user utterance input. The method of claim 1,
The first information includes at least one search result searched based on the user utterance input. The method of claim 1,
The processor is an electronic device that identifies whether the user input is related to the first information. The method of claim 1,
The user input includes a touch input for searching for the second information through an input menu or a UI image. The method of claim 1,
Further comprising a storage unit,
The processor stores the history information of function execution corresponding to the first user utterance input in the storage unit. The method of claim 1,
The processor is an electronic device that determines whether the second search request is a search request that can be processed in connection with the first search request. The method of claim 1,
The processor is configured to receive information on whether the second search request determined by the server is a search request that can be processed in connection with the first search request. The method of claim 1,
The processor,
At least one first search result and keyword related to the user's speech input are received from the server and displayed on the display unit,
An electronic device that receives the user input for selecting one of the displayed at least one keyword. The method of claim 8,
The electronic device, wherein the user input includes a text input. In the control method of an electronic device,
Transmitting a first search request to a server based on the user utterance input received through the user interface;
Displaying first information, which is received through a communication interface, corresponding to the user utterance input;
While displaying the first information, based on a user input received through the user interface, a second search request that can be processed in relation to the first search request is processed with the server that processed the first search request. Transmitting to the same server; And
Displaying the display unit to display second information, corresponding to the user input, received through the communication interface,
The second information is a control method of an electronic device corresponding to a search result of the user input that is related to the user speech input. The method of claim 10,
The first information includes at least one search result searched based on the user's speech input. The method of claim 10,
Transmitting the second search request to the same server as the server that processed the first search request includes identifying whether the user input is related to the first information. The method of claim 10,
The user input is a control method of an electronic device including a touch input for searching for the second information through an input menu or a UI image. The method of claim 10,
And storing history information of function execution corresponding to the user's speech input in a storage unit. The method of claim 10,
The step of transmitting the second search request to the same server as the server that processed the first search request includes determining whether the second search request is a search request that can be processed in relation to the first search request. A method of controlling an electronic device including performing. The method of claim 10,
Transmitting the second search request to the same server as the server that processed the first search request may include whether the second search request determined by the server is a search request that can be processed in relation to the first search request. The method of controlling an electronic device, further comprising receiving information on The method of claim 10,
Receiving and displaying at least one first search result and keyword related to the user utterance input from the server; And
And receiving the user input for selecting one of the displayed at least one keyword.

Images