KR102049833B1 - Interactive server, display apparatus and controlling method thereof - Google Patents

Abstract

An interactive server, a display device, and a control method thereof are disclosed. The interactive server according to the present invention communicates with a display device to receive a spoken voice signal including a first spoken element representing a target and a second spoken element representing an execution command, a plurality of indicators, and a plurality of instructions. A storage unit for storing the information; an extractor for extracting an indicator corresponding to the first speech element and a command corresponding to the second speech element from the storage; and response information corresponding to the spoken voice signal by combining the extracted indicator and command. And a controller configured to generate and transmit the same to the display device, wherein the first speech element is a speech element determined based on a display state of an object displayed on a screen of the display apparatus. Accordingly, the interactive server may grasp the user's intention with respect to the spoken voices of various users, generate response information, and provide the response information to the display apparatus.

Description

Interactive server, display apparatus and controlling method

The present invention relates to an interactive server, a display apparatus, and a control method, and more particularly, to an interactive server, a display apparatus, and a control method for providing response information corresponding to user speech.

In general, a display device capable of speech recognition in an interactive system collects a spoken voice of a user and transmits the collected spoken voice to an external server connected through a network. The external server receiving the spoken voice analyzes the spoken voice to determine the user's intention, generates response information, and transmits the response information to the display apparatus. Accordingly, the display apparatus may execute a function corresponding to the spoken voice of the user or provide the information based on the response information received from the external server.

However, such a conventional interactive system has a limitation in analyzing a user's spoken voice and determining a user's intention based on the analysis result. For example, in the case of a spoken voice that is clearly referred to as “show first content,” the external server analyzes the spoken voice to correctly grasp the user's intention, and generates response information to the display device. Can transmit Therefore, the display device may display the first content requested by the user based on the response information.

However, in the case of an unclear speech voice that is referred to as "show this", the external server has a problem that the user's intention is not clearly understood from the speech speech. In other words, the conventional interactive system has a problem in that the user's speech is restricted because the user's intention is determined only for the predetermined speech voice and the corresponding operation or information is provided.

The present invention has been made in accordance with the above-described needs, and an object of the present invention is to perform an operation corresponding to the speech of various users in an interactive system.

In order to achieve the above object, an interactive server according to an exemplary embodiment of the present invention communicates with a display apparatus to include a spoken voice signal including a first spoken element representing a target and a second spoken element representing an execution command. A communication unit for receiving a plurality of indicators, a storage unit for storing a plurality of indicators and a plurality of instructions, an extractor for extracting an indicator corresponding to the first ignition element and a command corresponding to the second ignition element from the storage unit, and the extraction. And a control unit for generating response information corresponding to the spoken voice signal by combining the indicated indicators and commands and transmitting the response information to the display device, wherein the first spoken element is based on a display state of an object displayed on a screen of the display device. It is a fire element determined by.

The first ignition element may include at least one of pronouns, ordinal numbers, and directions.

The extractor may determine whether there is request information in the first utterance element, and if there is the request information, extracts a command corresponding to the request information from a storage unit, and the controller is further configured to extract the command. Based on the request, content information corresponding to the request information may be added to the response information.

The indicator may be an execution word for relatively referring to the object among the objects displayed on the screen of the display apparatus.

The indicator may be unique identification information of the objects displayed on the screen of the display apparatus, and the extracting unit may determine an object referred to by the first utterance element based on a conversation history of the display apparatus. Unique identification information corresponding to the object may be extracted with the indicator.

The apparatus may further include a voice processor configured to convert the received spoken voice signal into text information.

Meanwhile, according to an embodiment of the present disclosure, in the display device, the display device may include an input unit for receiving a spoken voice of a user, a communication unit for transmitting a spoken voice signal for the spoken voice to a server device, and a display unit for displaying a screen. And, when response information including an indicator and a command is received from the server device, selecting a target referred to by the indicator based on a display state of objects displayed on a screen of the display unit, and selecting the target to the command for the selected target. It includes a control unit for performing a corresponding operation.

The speech signal includes a first speech element representing a target and a second speech element representing an execution command, wherein the first speech element is a speech element determined based on a screen display state of the display device. It may include at least one of an ordinal number and a direction.

Meanwhile, according to an embodiment of the present invention, in the method of controlling an interactive server, the method receives a spoken voice signal including a first speech element representing an object and a second speech element representing an execution command from a display device. And extracting an indicator corresponding to the first speech element and a command corresponding to the second speech element, and combining the extracted indicator and the command to generate response information corresponding to the spoken voice signal. And transmitting to a device, wherein the first utterance element is a utterance element determined based on a display state of an object displayed on a screen of the display device.

The first ignition element may include at least one of pronouns, ordinal numbers, and directions.

The extracting may include determining whether there is request information in the first utterance element, and if there is the request information, extracting a command corresponding to the request information from a storage unit, and transmitting the information. Based on the extracted command, content information corresponding to the request information may be added to the response information.

The indicator may be an execution word for relatively referring to the object among the objects displayed on the screen of the display apparatus.

The indicator may be unique identification information of the objects displayed on the screen of the display apparatus, and the extracting may include determining an object referred to by the first utterance element based on a conversation history of the display apparatus. The unique identification information corresponding to the determined object may be extracted as the indicator.

The method may further include converting the received spoken voice signal into text information.

On the other hand, according to an embodiment of the present invention, in the control method of the display apparatus, the method comprising the steps of receiving a user's spoken voice, transmitting a spoken voice signal for the spoken voice to a server device, the server device Receiving response information including an indicator and a command from the target; selecting an object referred to by the indicator based on a display state of objects displayed on the screen, and performing an operation corresponding to the command on the selected object; It includes.

The speech signal includes a first speech element representing a target and a second speech element representing an execution command, wherein the first speech element is a speech element determined based on a screen display state of the display device. It may include at least one of an ordinal number and a direction.

As described above, according to various embodiments of the present disclosure, in the interactive system, the interactive server may grasp the user's intention with respect to the spoken voices of the various users, generate response information, and provide the response information to the display apparatus.

1 is a first exemplary diagram of an interactive system for providing response information suitable for a user spoken voice according to an embodiment of the present invention;
2 is a second exemplary diagram of an interactive system for providing response information suitable for user spoken voice according to another embodiment of the present invention;
3 is a block diagram of an interactive server according to an embodiment of the present invention;
4 is an exemplary diagram of a spoken voice uttered based on a display state of an object displayed on a screen of a display device according to an embodiment of the present invention;
5 is a block diagram of a display device according to an embodiment of the present invention;
6 is a flowchart illustrating a method for controlling an interactive server according to an embodiment of the present invention;
7 is a flowchart illustrating a control method of a display apparatus according to an exemplary embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a first exemplary diagram of an interactive system for providing response information suitable for a user spoken voice according to an embodiment of the present invention.

As shown in FIG. 1, an interactive system according to an embodiment of the present invention includes a display device 100 and an interactive server 200.

The display device 100 is an internet-enabled device, and may be implemented as various electronic devices such as a smart TV, a mobile phone such as a smartphone, a desktop PC, a notebook, a navigation device, and the like. When the spoken voice of the user is input, the display apparatus 100 performs an operation corresponding to the input spoken voice of the user. In detail, the display apparatus 100 may output a response message corresponding to the spoken voice of the user or perform a function corresponding to the spoken voice of the user. That is, the display apparatus 100 analyzes the input spoken voice to determine whether it can be performed internally, and according to the determination result, performs the function corresponding to the input spoken voice or receives it from the interactive server 200. The operation may be performed based on the response information.

For example, when a spoken voice of "volume up" is input, the display apparatus 100 may adjust the volume based on control information corresponding to the input spoken voice among prestored control information.

For another example, when a spoken voice of "how is the weather today" is input, the display apparatus 100 transmits a spoken voice signal (hereinafter referred to as spoken voice) for the input spoken voice to the interactive server 200. do. Here, the spoken voice may be an analog signal. Therefore, the display apparatus 100 converts the spoken voice, which is an analog signal, into a digital signal and transmits the converted voice to the interactive server 200. Thereafter, the display apparatus 100 may output a result of today's weather as an image in voice or text form based on the response information received from the interactive server 200.

The interactive server 200 provides response information suitable for the user's intention based on the user's spoken voice received from the display apparatus 100. In detail, when the user's speech voice is received by the display apparatus 100, the interactive server 200 extracts a speech element from the received speech voice and responds to the user's speech based on the extracted speech element. Information can be generated and transmitted. As described above, the user's spoken voice received from the display apparatus 100 may be a digital signal. Therefore, when the spoken voice converted into a digital signal is received, the interactive server 200 generates the spoken voice as text information, analyzes the generated text information, and extracts a spoken element, based on the extracted spoken element. Response information corresponding to the spoken voice may be generated. Since generating text information from a spoken voice converted into a digital signal is a well-known technique, detailed description thereof will be omitted.

On the other hand, the utterance element is a key keyword for performing an operation requested by the user in the utterance voice of the user, and can classify the utterance elements extracted by the target domain, the user action, and the main features. have. As described above, when the text information on the user's speech voice of "how is the weather today" is generated, the interactive server 200 extracts the speech elements of "today", "weather", "how?" Can be. Thereafter, the interactive server 200 classifies "today" and "weather" among the extracted utterance elements as ignition elements for the main features (hereinafter referred to as the first ignition element), and "What?" It can be classified into a ignition element (hereinafter referred to as a second ignition element). In addition, the interactive server 200 may classify the utterance element (hereinafter referred to as the third utterance element) for the target region as belonging to a domain called web search based on the extracted utterance element. As such, when the first to third speech elements are classified from the text information of the user's speech voice, the interactive server 200 receives weather information from an external server (not shown) providing various contents, and includes the same. The response information is generated and transmitted to the display apparatus 100. Accordingly, the display apparatus 100 may display today's weather information through at least one of voice and text based on the response information received from the interactive server 200.

Meanwhile, the above-described interactive server 200 may include the first server 10 generating the spoken voice of the user converted into the digital signal as text information and the second response information corresponding to the spoken voice generated from the text information. Server 20 may be included. Hereinafter, an interactive system that provides response information suitable for a user's spoken voice through the display apparatus 100 and the first and second servers 10 and 20 will be described in detail.

2 is a second exemplary diagram of an interactive system providing response information suitable for a user spoken voice according to another embodiment of the present invention.

As shown in FIG. 2, when a spoken voice is input from a user, the display apparatus 100 converts the input spoken voice into a digital signal and transmits the converted spoken voice to a first server 10. When the spoken voice converted into the digital signal is received, the first server 10 generates text information regarding the spoken voice of the user and transmits the text information to the display apparatus 100 according to a specific pattern for the pre-stored various spoken voices.

The display apparatus 100 that receives the text information on the user's spoken voice from the first server 10 transmits the text information on the user's spoken voice to the second server 20. The second server 20 which has received text information on the spoken voice of the user analyzes the received text information, extracts a spoken element, and performs an operation corresponding to the spoken voice of the user based on the extracted spoken element. The response information may be generated and transmitted to the display apparatus 100.

Up to now, the operation of providing the response information corresponding to the spoken voice of the user in the interactive system including the display apparatus 100 and the interactive server 200 according to the present invention has been briefly described. Hereinafter, each configuration of the above-described display apparatus 100 and the interactive server 200 will be described in detail.

3 is a block diagram of an interactive server according to an embodiment of the present invention.

As illustrated in FIG. 3, the interactive server 200 includes a communication unit 210, a voice processing unit 220, a storage unit 230, an extraction unit 240, and a control unit 250.

The communicator 210 communicates with the display apparatus 100 that provides a spoken voice of a user. In particular, the communicator 210 may receive a digital signal for a spoken voice including a first spoken element representing a target and a second spoken element representing an execution command. Here, the first speech element is a speech element classified as a main feature in the speech of the user. When the first speech element is a speech element determined based on a display state of objects displayed on the screen of the display apparatus 100, the first speech element may be a speech element representing a target. That is, the first speech element may be a speech element representing the object through at least one of pronouns, ordinal numbers, and directions. The second speech element is a speech element that is classified as an objective function in the speech of the user. In the present invention, such an objective function will be referred to as an execution command.

For example, in the case of a utterance voice of "play this", "this" may be a first utterance element representing a pronoun and "execute" may be a second utterance element representing an execution command. When a digital signal for a spoken voice including such first and second spoken elements is received, the voice processor 220 converts the received spoken voice into text information. According to an embodiment, the speech processing unit 220 may convert the received speech of the user into text using a speech to text (STT) algorithm. However, the present invention is not limited thereto, and the communication unit 210 may receive text information on the user's spoken voice from the display apparatus 100. In this case, the display apparatus 100 receives the text information on the input voice of the user from the same as the above-described first server 10 and transmits it to the interactive server 200. Therefore, the communicator 210 may receive text information regarding the spoken voice of the user from the display apparatus 100. As such, when the text information regarding the spoken voice of the user is received from the display apparatus 100, the above-described configuration of the voice processor 220 may be omitted.

As such, when the spoken voice of the user is converted into text information or when the text information on the spoken voice of the user is received from the display apparatus 100, the extractor 240 corresponds to the first spoken element from the storage 230. The command corresponding to the indicator and the second speech element is extracted. Here, the storage unit 230 is a storage medium that stores various programs necessary for operating the interactive server 200, and may be implemented as a memory, a hard disk drvie (HDD), or the like. For example, the storage unit 230 may include a ROM for storing a program for performing an operation of the controller 250, a RAM for temporarily storing data according to an operation of the controller 250, and the like. . In addition, the storage unit 230 may further include an electrically erasable and programmable ROM (EEROM) for storing various reference data. In particular, the storage unit 230 may store a plurality of indicators and a plurality of commands. Here, the plurality of indicators and the plurality of commands are execution information for performing an operation in a form that can be interpreted by the display apparatus 100 based on the spoken elements extracted from the spoken voice of the user. In detail, the indicator is an execution word for relatively referring to an object among objects displayed on the screen of the display apparatus 100. In other words, the indicator is an execution word for performing an operation in an interpretable form on the display apparatus 100 based on a first utterance element representing an object such as a pronoun, an ordinal number, and a direction among utterance elements extracted from the utterance voice of the user. to be.

Therefore, the storage unit 230 may store a table in which an indicator is matched for each first utterance element indicating a target, as shown in Table 1 below.

First ignition element indicator this $ this $ next $ this $ + 1 third $ 3rd $

As shown in Table 1, the indicator corresponding to the first utterance element "this" may be $ this $, the indicator corresponding to the first ignition element "next" may be $ this $ + 1, and " The indicator corresponding to the first utterance element of "third" may be $ 3rd $. The plurality of commands stored in the storage 230 may include a second command indicating an execution command among utterance elements extracted from the user's spoken voice. Execution information for causing the display apparatus 100 to perform an operation based on the utterance element. Therefore, the storage unit 230 may store a table in which the command is matched for each second utterance element representing the execution command as shown in Table 2 below.

Second ignition element command Run it (run) execute Show me what show

As shown in Table 2, the command corresponding to the second utterance element "execute" may be "execute", and the command corresponding to the second utterance element "show me" or "what" may be "show". As described in Tables 1 and 2, the storage unit 230 may store an instruction for each of the first utterance elements representing the target and a command for each of the second utterance elements representing the execution command. Therefore, the extractor 240 extracts the first and second speech elements from the user's speech voice converted into text information through the speech processor 220, and an indicator corresponding to the extracted first and second speech elements; The command may be extracted from the storage 230.

The controller 250 generates response information corresponding to the user's spoken voice by combining an indicator and a command corresponding to the first and second speech elements extracted from the extractor 240, and then displays the display apparatus ( 100). For example, in the case of the user's spoken voice of "play this", the extraction unit 240 extracts the first speech element representing the object "this" and the second speech element representing the execution command "run". can do. When the first and second utterance elements are extracted, the extractor 240 extracts an indicator and a command corresponding to the first and second utterance elements extracted from the storage 230. That is, the extraction unit 240, based on the tables disclosed in Table 1 and Table 2, the extracting unit 240 indicates an execution command of the indicators "$ this $" and "execute" corresponding to the first ignition element indicating the object "this". 2 The command "execute" corresponding to the utterance element can be extracted.

As such, when the indicators and instructions corresponding to the first and second utterance elements are extracted, the controller 250 generates the execution command script by combining the extracted indicators and instructions. That is, the controller 250 combines the indicator "$ this $" corresponding to the first ignition element and the instruction "execute" corresponding to the second ignition element indicating the execution command "execute" and executes "execute ($ this $). You can create an executable command script called.

For another example, in the case of the user's spoken voice of "play the third one", the extraction unit 240 is the first speech element representing the object "third" and the second speech representing the execution command "execute". You can extract elements. When the first and second utterance elements are extracted, the extractor 240 extracts an indicator and a command corresponding to the first and second utterance elements extracted from the storage 230. That is, the extraction unit 240 is the first to display the execution command of the instructions "$ 3rd $" and "execute" corresponding to the first ignition element indicating the target "third" based on the table disclosed in Table 1 and Table 2 2 The command "execute" corresponding to the utterance element can be extracted.

As such, when the indicators and instructions corresponding to the first and second utterance elements are extracted, the controller 250 generates the execution command script by combining the extracted indicators and instructions. That is, the controller 250 combines the indicator "$ 3rd $" corresponding to the first ignition element and the instruction "execute" corresponding to the second ignition element indicating the execution command "execute" and executes "execute ($ 3rd $ You can create an executable command script called.

For another example, in the case of the user's spoken voice " please execute next ", the extraction unit 240 has a first speech element representing the object " next " and a second speech representing the execution command " execute " You can extract elements. When the first and second utterance elements are extracted, the extractor 240 extracts an indicator and a command corresponding to the first and second utterance elements extracted from the storage 230. That is, the extractor 240 executes the instructions "$ this $ + 1" and the "execute" command corresponding to the first ignition element representing the object "next" based on the table disclosed in Table 1 and Table 2. The command “execute” corresponding to the second utterance element represented may be extracted.

As such, when the indicators and instructions corresponding to the first and second utterance elements are extracted, the controller 250 generates the execution command script by combining the extracted indicators and instructions. That is, the controller 250 includes the indicator "$ this $ + 1" corresponding to the first utterance element indicating the object "next" and the instruction "execute" corresponding to the second ignition element indicating the execution command "execute". Can be combined to create an executable command script called "execute ($ this $ + 1)".

When such an execution command script is generated, the controller 250 transmits response information including the generated execution command script to the display apparatus 100. Accordingly, the display apparatus 100 selects an object corresponding to the object indicated by the user from among the objects displayed on the screen based on the execution command script included in the response information received from the interactive server 200, and selects the selected object. Can be displayed.

Hereinafter, the speech voice including the first and second speech elements based on the display state of the object displayed on the screen of the display apparatus 100 will be described in detail.

4 is an exemplary diagram of a spoken voice uttered based on a display state of an object displayed on a screen of a display device according to an exemplary embodiment of the present invention.

As illustrated in FIG. 4, the display apparatus 100 may receive and display the content 410 through a channel requested by the user. In addition, the display apparatus 100 may display a content list 420 for content requested by a user on the screen based on a user command input through a remote controller or a user spoken voice. As illustrated, the content information 421 to 425 for the first content broadcast so far may be displayed on the content list 420. Meanwhile, the user may speak the first content corresponding to the specific event by referring to the content information 421 to 425 displayed on the content list 420. For example, a user may speak "Please do this" to watch the first content corresponding to the first content information 421 or to watch the first content corresponding to the third content information 423. You can say, "Do the third one."

As described above, the user can view the first utterance element representing the first content (target) of the turn and the first content of the turn from the content list 420 displayed on the screen of the display apparatus 100. A utterance may be included that includes a second utterance element representing an execution command. Therefore, when the indicator and the command corresponding to the first and second spoken elements included in the spoken voice are extracted from the storage 230, the controller 250 generates an execution command script using a combination of the extracted indicator and the command. can do.

Meanwhile, according to an additional aspect of the present invention, the extractor 240 determines whether there is request information in the first utterance element, and if there is request information, extracts a command corresponding to the request information. Thereafter, the controller 250 may transmit content information corresponding to the request information to the response information based on the command extracted from the extractor 240 and transmit the content information to the display apparatus 100. To this end, the storage unit 230 may further store a command for the request information. For example, the request information "detail information" may be stored in correspondence with the command "detail information", and the request information "title" may be stored in correspondence with the command "title".

For example, in the case of the user's spoken voice of "what is the title", the extractor 240 may display the first speech element of "this" and "title" and the second speech element representing the execution command of "what". Can be extracted. Here, the extracted first utterance element "this" is a utterance element indicating the target, the first utterance element "title" may be a utterance element indicating the request information. As such, when the first and second ignition elements are extracted, the extractor 240 extracts from the storage 230 the first indicators "$ this $" and "title" corresponding to the first ignition element "this". Commands “title” corresponding to the utterance element and “show” corresponding to the second utterance element “what” may be extracted. As such, when the indicators and instructions corresponding to the first and second utterance elements are extracted, the controller 250 generates the execution command script by combining the extracted indicators and instructions. That is, the controller 250 includes the commands "title" and "what" corresponding to the first "ignition element" indicating the request information "indicator" $ this $ "and" title "corresponding to the first speech element indicating the object" this ". An executable command script called show (title) at ($ this $) may be generated by combining the command "show" corresponding to the second firing element representing the execution command.

When such an execution command script is generated, the controller 250 determines whether a command indicating request information is included in the generated execution command script. If there is a command indicating request information, the controller 250 determines whether to obtain content information corresponding to the request information based on the conversation history information with the display apparatus 100 stored in the storage 230. For example, the control unit 250 displays the response information including the content information on the action movie based on the user's spoken voice called "Show action movie" received before the user's spoken voice "What is the title of this?" 100 can be sent. Thereafter, as described above, when the execution command script including the command indicating the request information is generated, the controller 250 stores the title information of the corresponding content on the basis of the previous conversation history information, the EPG information stored in the storage 230. Or from an external server (not shown). Thereafter, the controller 250 may generate response information including the generated execution command script and title information and transmit the generated response information to the display apparatus 100.

However, the present invention is not limited thereto, and the controller 250 may transmit response information to the execution command script including the command indicating the request information to the display apparatus 100. In this case, the display apparatus 100 analyzes the execution command script included in the response information received from the interactive server 200, selects an object corresponding to the object indicated by the indicator among the objects displayed on the screen, and selects the selected object. An operation corresponding to the command may be performed with respect to. Accordingly, the display apparatus 100 may obtain title information of content corresponding to the selected object from pre-stored EPG information or may receive and output the information through an external server (not shown).

Meanwhile, according to an additional aspect of the present disclosure, the indicator stored in the storage 230 may be unique identification information of objects displayed on the screen of the display apparatus 100. In this case, the extractor 240 determines an object indicated by the first spoken element extracted from the spoken voice of the user based on the conversation history of the display apparatus 100, and indicates the unique identification information corresponding to the determined object. Can be extracted with

In detail, the display apparatus 100 and the interactive server 200 may share unique identification information about the content displayed on the screen of the display apparatus 100. Here, each unique identification information is information for identifying the content currently displayed on the display apparatus 100 and the content to be provided according to a user's request. For example, as described with reference to FIG. 4, the display apparatus 100 may display the content list 420 including the content 410 and the content information 421 to 425 for each episode on the screen. In this case, the content 410 may be given unique identification information (# 1234) indicating that it is currently being displayed, and the content list 420 may have unique identification information (# 5678) different from the currently displayed content (410). Can be given.

Accordingly, when the first and second speech elements are extracted from the speech of the user, the extractor 240 determines an object referred to from the first speech element among the extracted speech elements, and identifies unique identification information corresponding to the determined object. May be obtained from the storage 230 and determined as an indicator. For example, in the case of a spoken voice of "play this", the extractor 240 may extract the first spoken element of "this". As such, when the first ignition element is extracted, the extractor 240 may extract the indicator $ this $ corresponding to the first ignition element "this" from the storage unit 230. When such an indicator is extracted, the extractor 240 may determine that the object referred to from the first utterance element is a different object from the content 210 currently displayed on the screen of the display apparatus 100 through the extracted indicator. have. Therefore, the extractor 240 may convert $ this $, which is an indicator corresponding to the first utterance element "this", into unique identification information # 5678.

On the other hand, in the case of the spoken voice of "what is the title of what you are watching", the extraction unit 240 may extract the first speech element of "what is what you are watching." As such, when the first utterance element is extracted, the extractor 240 may extract the indicator $ showing_content $ corresponding to the first utterance element that is "walking" from the storage unit 230. When such an indicator is extracted, the extractor 240 may determine that the object referred to from the first utterance element is the content currently displayed on the screen of the display apparatus 100 through the extracted indicator. Accordingly, the extractor 240 may convert $ showing_content $, which is a branch corresponding to the first utterance element of “viewing”, into a unique identification number # 1234.

Up to now, each configuration in the interactive server 200 according to the present invention has been described in detail. Hereinafter, each configuration of the display apparatus 100 that receives a user's spoken voice will be described in detail.

5 is a block diagram of a display apparatus according to an embodiment of the present invention.

As shown in FIG. 5, the display apparatus 100 includes an input unit 110, a communication unit 120, a display unit 130, and a controller 140.

The input unit 110 receives a spoken speech voice from a user. In detail, when an analog voice of a user is input, the input unit 110 samples the input user speech and converts it into a digital signal. At this time, the input unit 110, if there is noise (for example, air conditioner sound, cleaner sound, etc.) in the input voice of the user, removes the noise, and converts the user's speech voice from which the noise is removed into a digital signal. This is preferable. In addition, the input unit 110 may perform a function of receiving various user manipulations and transmitting them to the controller 130. In this case, the input unit 110 inputs a user manipulation command through a touch pad or a keypad or a touch screen provided with various function keys, numeric keys, special keys, character keys, and the like. I can receive it.

The communication unit 120 transmits the user's spoken voice input through the input unit 110 to a server device (hereinafter referred to as an interactive server), and receives response information corresponding to the voiced voice. Here, the communication unit 120 may include various communication modules such as a short range wireless communication module (not shown), a wireless communication module (not shown), and the like. Here, the short range wireless communication module (not shown) is a communication module for performing wireless communication with the interactive server 200 located at a short distance and an external server (not shown) providing content, for example, Bluetooth, Zigbee, etc. Can be. The wireless communication module (not shown) is a module connected to an external network according to a wireless communication protocol such as Wi-Fi or IEEE to perform communication. In addition, the wireless communication module further includes a mobile communication module for accessing and communicating with the mobile communication network according to various mobile communication standards such as 3rd generation (3G), 3rd generation partnership project (3GPP), long term evolution (LTE), and the like. You may.

The display unit 130 may be implemented as a liquid crystal display (LCD), an organic light emitting diode (OLED), a plasma display panel (PDP), or the like. It is possible to provide various display screens that can be provided through. In particular, the display unit 161 displays content or content information corresponding to the spoken voice of the user based on the response information received from the interactive server 200.

The controller 140 controls the overall configuration of the display apparatus 100. In particular, when the controller 140 receives the response information including the execution command script generated by the combination of the indicator and the command from the interactive server 200, the controller 140 is based on the display state of the objects displayed on the screen of the display 130. To select the object indicated by the indicator. Thereafter, the controller 140 performs an operation corresponding to the command on the selected target.

In detail, as described with reference to FIG. 4, the display 130 may receive and display the content 410 through a channel requested by the user according to a control command of the controller 140. In addition, the display apparatus 100 may display a content list 420 for content requested by a user on the screen based on a user command input through a remote controller or a user's spoken voice. In addition, the display apparatus 100 may display a highlight on the first content information 421 of the individual content information 421 to 425 displayed on the content list 420 based on a preset condition. That is, when the display apparatus 100 first displays the content list 420 including the content information 421 to 425 on the screen for the first time, the display apparatus 100 displays the first content information located at the top based on a preset condition ( The highlight may be displayed so that 421 may be preferentially selected.

Meanwhile, the user may speak the first content corresponding to the specific event by referring to the content information 421 to 425 displayed on the content list 420. For example, the user may speak "Please execute this" in order to watch the first content corresponding to the first content information 421. Accordingly, the display apparatus 100 may receive a spoken voice of "do this". As described above, the display apparatus 100 according to the present invention includes a first utterance element representing a target referring to one of the plurality of objects in a state where a plurality of objects are displayed on the screen, and an execution command for executing the target. A spoken voice including two speech elements may be input. Here, the first utterance element may indicate an object through at least one of pronouns, ordinal numbers, and directions. For example, a spoken voice of "play this" may include a first spoken element that represents a subject in pronouns, and a "speak of please" may include a first spoken element that represents a subject in an ordinal number. Can be.

As such, when an utterance voice including a first utterance element representing a target and a second utterance element indicating an execution command for executing the target is input, the communication unit 120 inputs the uttered voice input to the interactive server 200. Send it. The interactive server 200 receiving the spoken voice extracts an indicator and a command corresponding to the first and second spoken elements included in the received spoken voice, and generates an execution command script using a combination of the extracted indicator and the command. As in the above example, when a spoken voice of "execute this" is received, the interactive server 200 "instructs" and "execute" the indicators "$ this $" corresponding to the first speech element representing the object "this". Extracts the command "execute" corresponding to the second utterance element representing the execution command. Thereafter, the interactive server 200 generates an execution command script by combining the extracted indicators and commands corresponding to the first and second utterance elements. That is, the interactive server 200 combines the indicator "$ this $" corresponding to the first utterance element and the instruction "execute" corresponding to the second utterance element indicating the execution command "execute", and executes "execute ($ Create an executable command script called this $). Thereafter, the interactive server 200 generates response information including an execution command script called "execute ($ this $)" and transmits the response information to the display apparatus 100.

When such response information is received, the controller 140 may interpret an execution command script included in the received response information and perform an operation corresponding to the spoken voice of the user. As in the above example, when response information including an execution command script called "execute ($ this $)" is received, the controller 140 interprets the execution command script to select one of the objects displayed on the screen. , Executes the selected object. In detail, the controller 140 interprets the execution command script to classify the indicator and the command. That is, the control unit 140 may be classified as "$ this $" and the command "execute" from the execution command script called "execute ($ this $)".

As such, when the indicator and the command are separated from the execution command script, the controller 140 may select one of a plurality of objects displayed on the screen based on the separated indicator. As shown in FIG. 4, the display unit 130 displays the content 410 received through the channel requested by the user, and simultaneously displays the contents information 421 to 425 of the first content requested by the user. The content list 420 may be displayed. In addition, the display 130 may display a highlight on the first content information 421 of the content information 421 to 425 included in the content list 420 based on a preset condition. That is, when the display unit 130 displays the content list 420 including the content information 421 to 425 for each time on the screen for the first time, the display 130 displays the first content information located at the top based on a preset condition ( The highlight may be displayed so that 421 may be preferentially selected. Meanwhile, when the user's operation command is input through the input unit 110 while the first content information 421 is highlighted, the display unit 130 displays the content information corresponding to the input user's operation command (1 time). One of the remaining content information 422 to 425 except for the content information 421 may be displayed. In this case, the content information marked with highlights may be preferentially selected in response to a user's operation command.

Therefore, when the indicator distinguished from the execution command script is "$ this $", the controller 140 may determine that the content information is currently highlighted. That is, as shown in FIG. 4, when the highlight is displayed in the first content information 421, the controller 140 may select the first content information 421 in which the highlight is displayed based on the “$ this $” indicator. As such, when the first content information 421 is selected, the controller 140 displays the first content corresponding to the first content information 421 selected based on “execute” which is a command classified from the execution command script. Can be received from (not shown) and displayed.

As in another example described above, the interactive server 200 generates response information including an execution command script called "execute ($ 3rd $)" from the user's spoken voice of "play the third one" and displays the display device ( 100). When such response information is received, the controller 140 interprets the execution command script included in the received response information to classify the indicator and the command. That is, the controller 140 may distinguish the indicator "$ 3rd $" and the command "execute" from the execution command script called "execute ($ 3rd $)". On the other hand, as shown in FIG. 4, if the highlight is displayed in the first-order content information 421, the controller 140 displays the third position based on the first-time content information 421 in which the highlight is displayed based on the "$ 3rd $" indicator. The third round content information 423 can be selected. As such, when the third round of content information 423 is selected, the controller 14 displays the first content corresponding to the third round of content information 421 selected based on "execute" which is a command separated from the execution command script. Can be received from (not shown) and displayed.

Up to now, each configuration of the display apparatus 100 that recognizes a user's spoken voice and performs an operation based on response information corresponding to the recognized spoken voice has been described in detail. Hereinafter, a control method of the interactive server 200 that provides response information corresponding to the spoken voice of the user and the display apparatus 100 that performs an operation based on the response information will be described in detail.

6 is a flowchart illustrating a control method of an interactive server according to an embodiment of the present invention.

As illustrated in FIG. 6, the interactive server 200 receives a spoken voice of a user including a first speech element representing a target and a second speech element representing an execution command from the display apparatus 100 (S610). . Here, the user's spoken voice is a voice signal converted from an analog voice signal to a digital signal. The first speech element is a speech element that is classified as a main feature in the speech of the user. When the first speech element is a speech element determined based on a display state of objects displayed on the screen of the display apparatus 100, the first speech element may be a speech element representing a target. That is, the first speech element may be a speech element representing the object through at least one of pronouns, ordinal numbers, and directions. The second speech element may be a speech element classified as an execution command in the speech of the user.

For example, in the case of a utterance voice of "play this", "this" may be a first utterance element representing a pronoun and "execute" may be a second utterance element representing an execution command. When the digital signal for the spoken voice including the first and second spoken elements is received, the interactive server 200 converts the received spoken voice into text information (S620). According to an embodiment, the interactive server 200 may convert the received speech of the user into text using a speech to text (STT) algorithm. However, the present invention is not limited thereto, and the interactive server 200 may receive text information on the spoken voice of the user from the display apparatus 100. In this case, the display apparatus 100 receives text information on the input voice of the user from the ASR server such as the first server 10 and transmits the text information to the interactive server 200. Therefore, the interactive server 200 may receive text information on the spoken voice of the user from the display apparatus 100.

As such, when the spoken voice of the user is converted into text information or when the text information on the spoken voice of the user is received from the display apparatus 100, the interactive server 200 performs a first speech from the spoken voice converted into text information. The indicator corresponding to the element and the instruction corresponding to the second spoken element are extracted (S630). In detail, the interactive server 200 may store a plurality of indicators and a plurality of commands. Here, the plurality of indicators and the plurality of commands are execution information for performing an operation in a form that can be interpreted by the display apparatus 100 based on the spoken elements extracted from the spoken voice of the user. More specifically, the indicator may be an execution word for relatively referring to an object among objects displayed on the screen of the display apparatus 100. In other words, the indicator is an execution word for performing an operation in a form that can be interpreted by the display apparatus 100 based on a first speech element representing an object such as a pronoun, an ordinal number, and a direction among speech elements extracted from the speech of the user. . The command is execution information for performing an operation in a form that can be interpreted by the display apparatus 100 based on a second speech element indicating an execution command among speech elements extracted from a user's speech voice. Thus, as shown in Tables 1 and 2, the interactive server 200 may store a table in which an indicator for each first utterance element indicating a target and a command for each second utterance element indicating an execution command match. Accordingly, the interactive server 200 may extract an indicator and a command corresponding to the first and second speech elements from the pre-stored table.

When the indicators and commands corresponding to the first and second speech elements are extracted, the interactive server 200 generates response information corresponding to the spoken voice by combining the extracted indicators and commands and outputs the response information to the display apparatus 100. Transmit (S640).

For example, in the case of the user's spoken voice of "play this", the interactive server 200 displays the first speech element representing the object "this" and the second speech element representing the execution command "run this". Can be extracted. When such first and second utterance elements are extracted, the interactive server 200 extracts an indicator and a command corresponding to the first and second utterance elements from a previously stored table. That is, the interactive server 200 indicates an execution command of the indicators "$ this $" and "execute" corresponding to the first ignition element representing the object "this", as shown in Tables 1 and 2, respectively. The command “execute” corresponding to the second speech element may be extracted. Thereafter, the interactive server 200 may generate the execution command script called "execute ($ this $)" by combining the extracted indicator and the command.

For another example, in the case of the user's spoken voice " execute a third one, " the interactive server 200 can display a first utterance element representing the object " third " and a second command " execute command " Ignition elements can be extracted. When such first and second utterance elements are extracted, the interactive server 200 extracts an indicator and a command corresponding to the first and second utterance elements from a previously stored table. That is, the interactive server 200 indicates the execution commands of the indicators "$ 3rd $" and "execute" corresponding to the first utterance element indicating the object "third", as shown in Tables 1 and 2, respectively. The command “execute” corresponding to the second speech element may be extracted. Thereafter, the interactive server 200 may generate the execution command script called "execute ($ 3rd $)" by combining the extracted indicator and the command.

When the execution command script is generated, the interactive server 200 generates response information including the generated execution command script and transmits the generated response information to the display apparatus 100. Accordingly, the display apparatus 100 selects an object corresponding to the object indicated by the user from among the objects displayed on the screen based on the execution command script included in the response information received from the interactive server 200, and selects the selected object. Can be displayed.

Meanwhile, the interactive server 200 determines whether there is request information in the first utterance element, and if there is request information, extracts a command corresponding to the request information. Thereafter, the interactive server 200 may add content information corresponding to the request information to the response information based on the extracted command and transmit the content information to the display apparatus 100. To this end, the interactive server 200 may further store a table matching the command for each request information. For example, the interactive server 200 may match the request information of "detailed information" with the command "detail informaion" and store it, and may match the request information of "title" with the command of "titile" and store it. .

For example, in the case of a user's spoken voice of "what is the title", the interactive server 200 may display a first speech element of "this" and "title" and a second speech element representing an execution command of "what". Can be extracted. Here, the extracted first utterance element "this" is a utterance element indicating the target, the first utterance element "title" may be a utterance element indicating the request information. As such, when the first and second utterance elements are extracted, the interactive server 200 refers to the pre-stored table and displays the indicators "$ this $" and "title" corresponding to the first utterance element "this". The command “show” corresponding to the first utterance element “title” and “what” corresponding to the first utterance element may be extracted. As such, when the indicators and commands corresponding to the first and second utterance elements are extracted, the interactive server 200 combines the extracted indicators and commands to execute an execution command script called show (title) at ($ this $). Can be generated.

When such an execution command script is generated, the interactive server 200 determines whether a command indicating request information is included in the previously generated execution command script. As a result of the determination, if there is a command in the execution command script, the interactive server 200 determines whether to obtain content information for the request information based on the conversation history information with the pre-stored display apparatus 100. For example, the interactive server 200 may include response information including content information about the action movie based on the user's spoken voice of “Show Action Movie” received before the user's spoken voice of “This title is Moya”. May be generated and transmitted to the display apparatus 100. Subsequently, when the user's spoken voice of "this title is Moya" is received, the interactive server 200 executes the command for executing the spoken voice of the user through the above-described steps. Create a script. Subsequently, if the interactive server 200 includes a command for requesting information in the previously generated execution command script, the interactive server 200 obtains title information on the corresponding content from the EPG information based on the conversation history information with the pre-stored display apparatus 100. Acquire or receive from an external server (not shown). Thereafter, the interactive server 200 generates response information including previously generated execution command script and title information and transmits the generated response information to the display apparatus 100.

However, the present invention is not limited thereto, and the interactive server 200 may transmit response information about an execution command script including a command indicating request information to the display apparatus 100. In this case, the display apparatus 100 analyzes the execution command script included in the response information received from the interactive server 200, selects an object corresponding to the object indicated by the indicator among the objects displayed on the screen, and selects the selected object. An operation corresponding to the command may be performed with respect to. Accordingly, the display apparatus 100 may obtain title information of content corresponding to the selected object from pre-stored EPG information or may receive and output the information through an external server (not shown).

Meanwhile, according to an additional aspect of the present invention, the indicator stored in the interactive server 200 may be unique identification information of the objects displayed on the screen of the display apparatus 100. Here, each unique identification information is information for identifying the content currently displayed on the display apparatus 100 and the content to be provided according to a user's request. For example, as described with reference to FIG. 4, the display apparatus 100 may display the content list 420 including the content 410 and the content information 421 to 425 for each episode on the screen. In this case, the content 410 may be given unique identification information (# 1234) indicating that it is currently being displayed, and the content list 420 may have unique identification information (# 5678) different from the currently displayed content (410). Can be given.

Therefore, when the first and second speech elements are extracted from the user's speech voice, the interactive server 200 determines an object referred to from the first speech elements among the extracted speech elements, and determines among the pre-stored unique identification information. Unique identification information corresponding to the object may be obtained and determined as an indicator. For example, in the case of a spoken voice of "play this", the interactive server 200 may extract the first spoken element of "this". As such, when the first utterance element is extracted, the interactive server 200 may extract the indicator $ this $ corresponding to the first utterance element "this" from among the pre-stored first utterance element-specific indicators. When such an indicator is extracted, the interactive server 200 may determine that the object referred to from the first utterance element is a different object from the content 210 currently displayed on the screen of the display apparatus 100 through the extracted indicator. Can be. Accordingly, the interactive server 200 may convert $ this $, which is an indicator corresponding to the first utterance element "this", into unique identification information # 5678.

Hereinafter, a control method of the display apparatus 100 that performs an operation based on response information corresponding to a user's spoken voice will be described in detail.

7 is a flowchart illustrating a control method of a display apparatus according to an exemplary embodiment.

As shown in FIG. 7, the display apparatus 100 receives an input voice of a user (S710). When the user's spoken voice is input, the display apparatus 100 transmits the input user's spoken voice to the interactive server 200 (S720). In detail, when the analog speech user's speech is input, the display apparatus 100 converts the input speech of the user into a digital signal. In this case, the display apparatus 100 may determine whether there is noise in the input speech of the user, and if there is noise, convert the speech of the user from which the noise is removed into a digital signal.

When the user's spoken voice is converted into a digital signal, the display apparatus 100 transmits the user's spoken voice converted into a digital signal to the interactive server 200 and receives corresponding response information (S730). When the response information is received, the display apparatus 100 selects a target indicated by the indicator included in the response information based on the display state of the objects displayed on the screen, and corresponds to a command included in the response information for the selected target. The operation is performed (S740).

In detail, as described with reference to FIG. 4, the display apparatus 100 may receive and display the content 410 through a channel requested by the user. In addition, the display apparatus 100 may display a content list 420 for content requested by a user on the screen based on a user command input through a remote controller or a user's spoken voice. In addition, the display apparatus 100 may display a highlight on the first content information 421 of the individual content information 421 to 425 displayed on the content list 420 based on a preset condition. That is, when the display apparatus 100 first displays the content list 420 including the content information 421 to 425 on the screen for the first time, the display apparatus 100 displays the first content information located at the top based on a preset condition ( The highlight may be displayed so that 421 may be preferentially selected.

On the other hand, the user may utter "Do this" to watch the first content corresponding to the first content information 421 displayed on the content list 420. Accordingly, the display apparatus 100 may receive a user's spoken voice of "do this". As described above, the display apparatus 100 according to the present invention includes a first utterance element representing a target referring to one of the plurality of objects in a state where a plurality of objects are displayed on the screen, and an execution command for executing the target. A spoken voice including two speech elements may be input. Here, the first utterance element may indicate an object through at least one of pronouns, ordinal numbers, and directions. For example, a spoken voice of "play this" may include a first spoken element that represents a subject in pronouns, and a "speak of please" may include a first spoken element that represents a subject in an ordinal number. Can be.

As such, when an utterance voice including a first utterance element representing a target and a second utterance element representing a silham command for executing the target is input, the display apparatus 100 receives a utterance input to the interactive server 200. The voice is converted into a digital signal and transmitted. Accordingly, as described above, the interactive server 200 extracts an indicator and a command corresponding to the first and second speech elements included in the input speech voice, and combines the extracted indicator and the command to execute an execution command script. Create As in the above example, when a spoken voice of "execute this" is received, the interactive server 200 "instructs" and "execute" the indicators "$ this $" corresponding to the first speech element representing the object "this". Extract command execute corresponding to the second utterance element representing the execution command. Thereafter, the interactive server 200 generates an execution command script called "execute ($ this $)" by combining the extracted indicators and commands corresponding to the first and second utterance elements. Thereafter, the interactive server 200 generates response information including an execution command script called "execute ($ this $)" and transmits the response information to the display apparatus 100.

When such response information is received, the display apparatus 100 may interpret an execution command script included in the received response information and perform an operation corresponding to the spoken voice of the user. As in the above example, when the response information including the execution command script called "execute ($ this $)" is received, the display apparatus 100 interprets the execution command script so that the indicator is $ this $ and the command is execute. Can be distinguished.

As such, when the indicator and the command are separated from the execution command script, the display apparatus 100 may select one of a plurality of objects displayed on the screen based on the separated indicator. As illustrated in FIG. 4, the display apparatus 100 may display a highlight on the first content information 421 among the content information 421 to 425 included in the content list 420. Here, the first round content information 421 in which the highlight is displayed may be a reference for selecting an object corresponding to a target indicated by the user. Therefore, when the indicator distinguished from the execution command script is "$ this $", the display apparatus 100 may determine that the first content information 421 is highlighted and select the first content information 421. . As such, when the first content information 421 is selected, the display apparatus 100 displays the first content corresponding to the first content information 421 based on the "execute" command separated from the execution command script. It may be received from the display (not shown).

In another example, the display apparatus 100 may include an execution command script called "execute ($ this $ + 1)" in response to a user's spoken voice of "execute next" from the interactive server 200. Response information may be received. In this case, the display apparatus 100 may distinguish the command “$ this $ + 1” from the command “execute” by analyzing the execution command script included in the received response information. On the other hand, as shown in FIG. 4, if the highlight is displayed in the first content information 421, the display apparatus 100 refers to the first content information 421 in which the highlight is displayed based on the "$ this $ + 1" indicator. Next, the second content information 422 located next may be selected. As such, when the second content information 422 is selected, the display apparatus 100 displays the first content corresponding to the second content information 422 based on the "execute" command separated from the execution command script. It may be received from the display (not shown).

So far I looked at the center of the preferred embodiment for the present invention.

While the above has been shown and described with respect to preferred embodiments of the present invention, the present invention is not limited to the specific embodiments described above, it is usually in the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

10: first server 20: second server
100: display device 110: input unit
120,210: communication unit 130: display unit
140,250: control unit 200: interactive server
220: voice processing unit 230: storage unit
240: extraction unit

Claims (14)

An input unit for receiving a voice command and converting the voice command into a digital signal;
Communication unit for performing communication with the server;
A display unit configured to display a content list including a plurality of objects on a screen; And
Control the communication unit to transmit the digital signal to the server,
Receiving, from the server, response information for performing an operation corresponding to the voice command obtained based on text information corresponding to the voice command, through the communication unit,
And a controller configured to identify a target object among the plurality of objects in the content list based on the response information, and to control the display to display the target object on the screen.
The text information includes a first command element representing the target object and including an ordinal number and a second command element representing an execution command,
The response information may include first information relatively representing the target object among the plurality of objects based on the first command element, and second information that is execution information for performing the operation based on the second command element. Include,
The control unit,
And identifying the target object at a position corresponding to the ordinal number among the plurality of objects based on the first information, and performing the operation on the identified target object based on the second information. The method of claim 1,
The control unit,
And identifying the target object at a position corresponding to the ordinal based on one of the plurality of objects. The method of claim 2,
The control unit,
And identifying the target object at a position corresponding to the ordinal based on a focus displayed on one of the plurality of objects. The method of claim 1,
And the first command element further comprises at least one of a pronoun or a direction. The method of claim 1,
The control unit,
And when the target object is identified, controlling the display to display the other object and the target object differently in the content list. The method of claim 1,
Each of the plurality of objects includes an image representing corresponding content. The method of claim 1,
And the response information includes an execution command script generated by combining the first information and the second information. In the control method of the display device,
Displaying a content list including a plurality of objects on a screen;
Receiving a voice command;
Converting the voice command into a digital signal;
Transmitting the digital signal to a server;
Receiving, from the server, response information for performing an operation corresponding to the voice command obtained based on text information corresponding to the voice command;
Identifying a target object among the plurality of objects in the content list based on the response information; And
Displaying the target object on the screen;
The text information includes a first command element representing the target object and including an ordinal number and a second command element representing an execution command,
The response information may include first information relatively representing the target object among the plurality of objects based on the first command element, and second information that is execution information for performing the operation based on the second command element. Include,
Identifying the target object,
Identify the target object at a position corresponding to the ordinal number of the plurality of objects based on the first information,
The control method,
And performing the operation on the identified target object based on the second information. The method of claim 8,
Identifying the target object,
And identifying the target object at a position corresponding to the ordinal based on one of the plurality of objects. The method of claim 9,
Identifying the target object,
And identifying the target object at a position corresponding to the ordinal based on a focus displayed on one of the plurality of objects. The method of claim 8,
And the first command element further comprises at least one of a pronoun or a direction. The method of claim 8,
And displaying the target object differently from the remaining objects in the content list when the target object is identified. The method of claim 8,
Each of the plurality of objects includes an image representing corresponding content. The method of claim 8,
And the response information includes an execution command script generated by combining the first information and the second information.

Images