KR102524675B1 - Display apparatus and controlling method thereof - Google Patents

Abstract

A display device is disclosed. The present display device controls the display to display a UI screen including a display and a plurality of text objects, and controls the display so that a preset number is displayed together for a text object different from a predetermined language among the plurality of text objects, and a processor for performing an operation related to a text object corresponding to the displayed number when the result of recognizing the voice spoken by the user includes the displayed number.

Description

Display device and its control method { DISPLAY APPARATUS AND CONTROLLING METHOD THEREOF }

The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device providing voice recognition control for contents composed of various languages and a control method thereof.

Thanks to the development of electronic technology, various types of display devices have been developed and supplied. In particular, various electronic devices such as TVs, mobile phones, PCs, notebook PCs, and PDAs have been widely used in most households.

Meanwhile, in recent years, a technology using voice recognition has been developed in order to more conveniently and intuitively control a display device.

Conventional display devices controlled according to the user's voice perform voice recognition using a voice recognition engine. Since there are different voice recognition engines for each language, it is necessary to determine in advance which voice recognition engine to use for voice recognition. there was. Therefore, in general, the system language of the display device is determined as a language to be used for voice recognition.

However, for example, when the language used in the hyperlink text displayed on the display device is English and the system language of the display device is Korean, even if the user utters a voice corresponding to the hyperlink text, the voice is passed through the Korean voice recognition engine to Korean language. Since it is converted into text, there was a problem that the corresponding hyperlink text could not be selected after all.

As such, conventionally, when the system language and the language actually displayed on the display device are different from each other, there is a limitation in controlling the display device with voice.

The present disclosure has been made in accordance with the above-described needs, and an object of the present disclosure is to provide a display device and a control method for providing voice recognition control for contents composed of various languages.

A display device according to an embodiment of the present disclosure for achieving the above object controls the display to display a UI screen including a display and a plurality of text objects, and selects a predetermined language among the plurality of text objects. A processor for controlling the display so that preset numbers are displayed together for different text objects, and performing an operation related to a text object corresponding to the displayed numbers when a recognition result of a voice spoken by a user includes the displayed numbers do.

In this case, the processor may set the language set in the setting menu of the display device to the predetermined language. Alternatively, the most frequently used language of the plurality of text objects may be set as the predetermined language.

Meanwhile, the UI screen is a web page, and the processor may set a language corresponding to language information of the web page to the predetermined language.

On the other hand, the processor may determine that a text object composed of two or more languages among the plurality of text objects is a text object different from the predetermined language when the content ratio of the predetermined language is less than the predetermined ratio.

Meanwhile, the processor may control the display to display the preset number adjacent to a text object corresponding to the preset number.

Meanwhile, the display device according to the present disclosure further includes a communication unit that communicates with an external device, and the processor displays the predetermined number while receiving a signal corresponding to selection of a specific button of the external device. You can control the display.

In this case, the external device includes a microphone, the communication unit receives a voice signal corresponding to a voice input through the microphone of the external device, and the processor determines that the recognition result of the received voice signal is If the displayed number is included, an operation related to the text object corresponding to the displayed number can be performed.

In this case, if the recognition result of the received voice signal includes text corresponding to any one of the plurality of text objects, the processor may perform an operation related to the corresponding text object.

Meanwhile, an operation related to the text object may be a display operation of a web page having a URL address corresponding to the text object or an application program execution operation corresponding to the text object.

Meanwhile, the plurality of text objects are included in the execution screen of the first application, and the processor determines that the object corresponding to the recognition result of the voice spoken by the user while the execution screen of the first application is displayed is the first application. If it is determined that there is no execution screen of , a second application different from the first application may be executed to perform an operation corresponding to the result of recognizing the voice.

In this case, the second application is an application that provides a search result for a search word, and the processor determines that an object corresponding to a result of recognizing a voice spoken by a user while an execution screen of the first application is displayed is displayed. If it is determined that there is no on the execution screen of , the second application may be executed to provide a search result using text corresponding to the voice recognition result as a search term.

Meanwhile, the display device according to the present disclosure further includes a communication unit communicating with a server that performs voice recognition for a plurality of different languages, and the processor includes a voice signal corresponding to the voice spoken by the user and the predetermined The communication unit may be controlled to provide language information to the server, and if a voice recognition result received from the server includes the displayed number, an operation related to a text object corresponding to the displayed number may be performed.

In this case, if the voice recognition result received from the server includes text corresponding to any one of the plurality of text objects, the processor may perform an operation related to the corresponding text object.

Meanwhile, a control method of a display device according to an embodiment of the present disclosure includes displaying a plurality of text objects, displaying a preset number for text objects different from a predetermined language among the plurality of text objects, and and performing an operation related to a text object corresponding to the displayed number if the result of recognizing the voice spoken by the user includes the displayed number.

In this case, the control method of the display device according to the present disclosure may further include setting a language most frequently used for the plurality of text objects as the predetermined language.

Meanwhile, the plurality of text objects are included in the web page, and the control method of the display device according to the present disclosure may further include setting a language corresponding to language information of the web page to the predetermined language. .

On the other hand, in the control method of the display device according to the present disclosure, for a text object composed of two or more languages among the plurality of text objects, when the ratio of the predetermined language is less than the predetermined ratio, the text object is different from the predetermined language. It may further include the step of determining that it is.

Meanwhile, in the step of displaying the preset number together, the preset number may be displayed adjacent to a text object corresponding to the preset number.

Meanwhile, in the step of displaying the preset number together, the preset number may be displayed while a signal corresponding to selection of a specific button of the external device is received from the external device.

Meanwhile, in the step of performing an operation related to the text object, a web page having a URL address corresponding to the text object may be displayed or an application program corresponding to the text object may be executed.

Meanwhile, the plurality of text objects are included in the execution screen of the first application, and the control method of the display device according to the present disclosure corresponds to the recognition result of the user's voice while the execution screen of the first application is displayed. The method may further include performing an operation corresponding to a result of recognizing the voice by executing a second application different from the first application when it is determined that the object is not present on the execution screen of the first application.

Meanwhile, a method for controlling a display device according to the present disclosure includes providing a voice signal corresponding to a voice spoken by the user and information on the predetermined language to a server performing voice recognition for a plurality of different languages. In the step of performing an operation related to the text object, if the voice recognition result received from the server includes the displayed number, an operation related to the text object corresponding to the displayed number may be performed.

Meanwhile, in a computer readable recording medium storing a program for executing a method for controlling a display device according to an embodiment of the present disclosure, the method for controlling the display device controls the display device to display a plurality of text objects. controlling the display device to display a predetermined number for text objects different from a predetermined language among the plurality of text objects, and if the result of recognizing the voice spoken by the user includes the displayed number, and performing an operation related to the text object corresponding to the displayed number.

1 and 2 are views for explaining a method for inputting a voice command in a display device according to various embodiments of the present disclosure;
3 is a diagram for explaining a voice recognition system according to an embodiment of the present disclosure;
4 is a block diagram for explaining the configuration of a display device according to an embodiment of the present disclosure;
5 to 7 are views for explaining a number display method for object selection according to various embodiments of the present disclosure;
8 to 9 are views for explaining a voice search method according to various embodiments of the present disclosure;
10 is a block diagram for explaining the configuration of a display device according to another embodiment of the present disclosure, and
11 is a flowchart for explaining a control method of a display device according to an embodiment of the present disclosure.

Prior to a detailed description of the present disclosure, the method of describing the present specification and drawings will be described.

First, the terms used in the present specification and claims are general terms selected in consideration of functions in various embodiments of the present disclosure. It may change depending on the emergence of new technologies, etc. In addition, some terms are arbitrarily selected by the applicant. These terms may be interpreted as the meanings defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the art.

In addition, the same reference numerals or numerals in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of description and understanding, the same reference numerals or symbols are used in different embodiments. That is, even if all components having the same reference numerals are shown in a plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in the present specification and claims, terms including ordinal numbers such as “first” and “second” may be used to distinguish between elements. These ordinal numbers are used to distinguish the same or similar components from each other, and the meaning of the term should not be construed as being limited due to the use of these ordinal numbers. For example, the order of use or arrangement of elements associated with such ordinal numbers should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In the embodiments of the present disclosure, terms such as “module,” “unit,” and “part” are terms used to refer to components that perform at least one function or operation, and these components are hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except for cases where each of them needs to be implemented with separate specific hardware, so that at least one processor can be implemented as

Also, in an embodiment of the present disclosure, when a part is said to be connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that it may further include other components without excluding other components unless otherwise stated.

Hereinafter, the present disclosure will be described in detail using the accompanying drawings.

1 is a diagram for explaining a display apparatus controlled according to voice recognition according to an embodiment of the present disclosure.

Referring to FIG. 1, the display device 100 may be a TV as shown in FIG. 1, but this is only an example, and a smartphone, desktop PC, laptop computer, smart watch, navigation device, refrigerator, etc. It can be implemented in any device.

The display apparatus 100 may perform an operation based on a result of recognizing a voice spoken by a user. For example, if the user says "change to channel 7", the program of channel 7 can be displayed, and if the user says "turn off the power", the power can be turned off. Also, the display device 100 may operate as if talking to a user. For example, a user may output a voice or text message saying "The title of the program you are inquiring about is ○○○" in response to a voice asking "What is the name of the program currently being broadcast?", and the user may ask "How is the weather today?" ", the message "Tell me the region you want" can be output through voice or text, and when the user says "Seoul", the message "The temperature of Seoul is ○○" can be output through voice or text. there is.

As shown in FIG. 1 , the display apparatus 100 may receive a user's voice through a microphone connected to the display apparatus 100 or included in the display apparatus 100 . Alternatively, the display apparatus 100 may receive a voice signal corresponding to a voice input through a microphone of the external device from the external device. This will be described with reference to FIG. 2 .

2 is a diagram for explaining a display system according to an embodiment of the present disclosure.

Referring to FIG. 2 , the display system includes a display device 100 and an external device 200 .

The display device 100 is a device that operates according to a voice recognition result as described in FIG. 1 .

Although FIG. 2 shows an example in which the external device 200 is implemented as a remote control, it is also possible to implement the external device 200 as an electronic device such as a smart phone, a tablet PC, and a smart watch.

The external device 200 is a device including a microphone and can transmit a voice signal corresponding to a voice input through the microphone to the display device 100 . For example, the external device 200 may transmit a voice signal to the display device 100 using a wireless communication method such as infrared (IR), RF, Bluetooth, or Wi-Fi.

The microphone of the external device 200 may be activated only when there is a preset event to save power. For example, while pressing the microphone button 210 of the external device 200, the microphone is activated, and when the microphone button 210 is released, the microphone is deactivated. That is, voice can be received only while the microphone button 210 is pressed.

Voice recognition for a voice input through the microphone of the display device 100 or the microphone of the external device 200 may be performed through an external server. 3 is a diagram for explaining an embodiment related to this.

Referring to FIG. 3 , the voice recognition system 2000 includes a display device 100 and a server 300 .

The display device 100 is a device that operates according to a voice recognition result as described in FIG. 1 . As described above, the display apparatus 100 may transmit a voice signal corresponding to a voice input through the microphone of the display apparatus 100 or the microphone of the external apparatus 200 to the server 300 .

The display apparatus 100 may transmit to the server 300 information indicating which language should be used to recognize the voice signal (hereinafter referred to as 'language information') along with the voice signal. Even with the same voice signal, different voice recognition results may be obtained depending on which language voice recognition engine is used.

The server 300 may perform voice recognition for a plurality of different languages. The server 300 may include several voice recognition engines corresponding to each of several languages. For example, the server 300 may include a Korean voice recognition engine, an English voice recognition engine, a Japanese voice recognition engine, and the like. When a voice signal and language information are received from the display apparatus 100, the server 300 may perform voice recognition on the voice signal using a voice recognition engine corresponding to the language information.

The server 300 may transmit the voice recognition result to the display device 100, and the display device 100 may perform an operation corresponding to the voice recognition result received from the server 300.

For example, if the text included in the voice recognition result received from the server 300 matches the text object displayed on the display device 100, the display apparatus 100 may perform an operation related to the corresponding text object. For example, if there is a text object matching text included in the voice recognition result in the web page, the display apparatus 100 may display a web page having a URL address corresponding to the corresponding text object. However, this is only an example, and UI objects provided by various applications of the display apparatus 100 may be selected by voice recognition and corresponding operations may be performed.

Meanwhile, although FIG. 3 shows one server 300, a plurality of servers corresponding to each of a plurality of languages may exist. For example, a server in charge of Korean voice recognition and a server in charge of English voice recognition may exist separately.

Meanwhile, in the above example, it has been described that voice recognition is performed in the server 300, which is a separate device from the display device 100, but according to another example, the display device 100 performs the function of the server 300. It is also possible. That is, it is possible that the above-described display device 100 and the server 300 are implemented as a single product.

4 is a block diagram for explaining the configuration of a display apparatus 100 according to an embodiment of the present disclosure.

The display device 100 includes a display 110 and a processor 120.

The display 110 may be implemented as, for example, a liquid crystal display (LCD), and in some cases, a cathode-ray tube (CRT), a plasma display panel (PDP), organic light emitting diodes (OLED), or a transparent OLED (TOLED). etc. can be implemented. Also, the display 110 may be implemented in the form of a touch screen capable of detecting a user's touch manipulation.

The processor 120 is a component for controlling the overall operation of the display device 100 .

For example, the processor 120 may include a CPU, RAM, ROM, and a system bus. Here, the ROM is a configuration in which a command set for system booting is stored, and the CPU copies the operating system stored in the storage unit of the display device 100 to the RAM according to the command stored in the ROM, and executes the O / S to boot the system let it When booting is completed, the CPU copies various applications stored in the storage unit to RAM and executes them to perform various operations. In the above, the processor 120 has been described as including only one CPU, but may be implemented with a plurality of CPUs (or DSPs, SoCs, etc.).

When a user command for selecting an object displayed on the display 110 is input, the processor 120 may perform an operation related to the object selected by the user command. Here, the object may be any object that can be selected, and may be, for example, a hyperlink or an icon. The operation associated with the selected object may be, for example, an operation of displaying a page, document, image, etc. connected to a hyperlink, an operation of executing a program corresponding to an icon, and the like.

A user command for selecting an object may be, for example, a command input through various input devices (ex. mouse, keyboard, touch pad, etc.) connected to the display device 100 or a voice command corresponding to a voice uttered by the user. there is.

Although not shown in FIG. 4 , the display apparatus 100 may further include a voice receiver for receiving voice input. The voice receiving unit may include a microphone to generate a voice signal by directly receiving a voice spoken by a user, or may receive an electrical voice signal from the external device 200 . In the latter case, the voice receiver may be implemented as a communication unit for performing wired or wireless communication with the external device 200 . Such a voice receiver may not be included in the display device 100 in some cases. For example, a voice signal corresponding to a voice input through a microphone of the external device 200 is transmitted to the server 300 via a device other than the display device 100 or directly from the external device 200 to the server. 300, and the display device 100 may be implemented in a form of receiving only the voice recognition result from the server 300.

Among the text objects displayed on the display 110, the processor 120 may control the display 110 to display numerals for text objects different from a predetermined language.

Here, the predetermined language means a language that is a basis for voice recognition (a language of a voice recognition engine to be used for voice recognition), and can be manually set by a user or can be set automatically. To describe the case of manually setting a language, for example, a language set as a language to be used (or a system language) in a setting menu provided by the display apparatus 100 may be set as a language based on voice recognition.

According to an embodiment of automatically setting a language as a basis for voice recognition, the processor 120 identifies a language most frequently used in a text object currently displayed on the display 110, and selects the language as a basis for voice recognition. The language can be set automatically.

Specifically, the processor 120 analyzes the type of characters (eg, Hangul or the alphabet) included in each of the plurality of text objects currently displayed on the display 110, and the type of character most frequently used overall in the plurality of text objects. A language corresponding to may be set as a language that is a basis for voice recognition.

According to another embodiment, if objects currently displayed on the display 110 are objects of a web page, the processor 120 may set a language corresponding to language information of the corresponding web page as a language that is a basis for voice recognition. Language information of the web page can be checked, for example, in the lang attribute of HTML (eg, <html lang="en">).

When a language based on voice recognition is set, the processor 120 may control the display 110 to display an arbitrary number for a text object different from a language based on voice recognition. A user may select a text object by speaking any number displayed on the display 110 . Also, since an image object cannot be selected by voice, the processor 120 may control the display 110 to display an arbitrary number for the image object as well.

The processor 120 may determine that a text object composed of only a language other than a language used as a basis for voice recognition is a text object different from a language to be used for voice recognition. In addition, the processor 120 may determine that a text object composed of two or more languages is a text object different from the language to be used for voice recognition when the content ratio of the language underlying voice recognition is less than a preset ratio. This will be described in more detail with reference to FIG. 5 .

5 illustrates that a specific screen is displayed on the display 110 .

Referring to FIG. 5 , a UI screen including a plurality of text objects 51 to 59 is displayed on the display 110 . Assume that the language that is the basis of voice recognition is set to English. The processor 120 may control the display 110 to display random numbers ① to ⑥ together with respect to the text objects 51 to 56 composed of languages other than English. Numbers ① to ⑥ may be displayed at positions adjacent to the corresponding text objects 51 to 56. And, for the text objects 51 and 58 composed of English, by displaying specific icons 57a and 58a together, by uttering the text included in the text objects 51 and 58, the text objects 51, 58) can be selected. The icons 57a and 58a may be expressed as "T" as shown in FIG. 5, but are not limited thereto, and may be expressed in various forms, such as "Text".

For the text object 59 composed of two or more languages, the processor 120 checks whether the English content ratio is less than a predetermined ratio (eg, 50%), and controls the display 110 to display numbers together if it is less than can do. The text object 59 shown in FIG. 5 is composed of Korean and English, but since the proportion of English exceeds a predetermined ratio (eg 50%), numbers are not displayed together. Instead, an icon 59a indicating that the text object can be selected may be displayed adjacent to the text object 59 by uttering the text included in the text object.

Meanwhile, in FIG. 5, the numbers are shown to have a shape such as “①”, but there is no limit to the shape of the numbers. For example, “1” may be included in a rectangle instead of a circle, or only “1” may be displayed. According to another embodiment of the present disclosure, it may be displayed as a word of a language that is the basis of voice recognition. If the language that is the basis of voice recognition is English, it may be displayed as "one", and the basis of voice recognition If the language is Spanish, it may be displayed as "uno".

Meanwhile, although not shown in FIG. 5 , a phrase encouraging the user to say a number, such as “You can select an object corresponding to the number you have spoken,” may be additionally displayed on the display 110 together with the display of the number.

According to another embodiment of the present disclosure, for a text object composed of two or more languages, if the language of the first word is different from the language to be used for voice recognition, the processor 120 may set the text object to be different from the language used for voice recognition. can be judged. This embodiment will be described with reference to FIG. 6 .

6 illustrates that a specific screen is displayed on the display 110 .

Referring to FIG. 6 , a UI screen including a plurality of text objects 61 to 63 is displayed on the display 110 . Assume that the language to be used for voice recognition is set to Korean. For the text object 61 composed of two or more languages, the processor 120 is different from the language based on voice recognition because the language of the first word "AAA" is English, not Korean, which is the language based on voice recognition. It can be determined as a text object. Accordingly, the processor 120 may control the display 110 to display the number ① together with the text object 61 .

According to the embodiment described with reference to FIG. 6 , even if a text object composed of two or more languages includes a language based on voice recognition in a predetermined ratio or more, if the first word is different from the language based on voice recognition, a number is entered. display Conversely, even if the text object composed of two or more languages contains less than a preset ratio of the language underlying voice recognition, if the first word is the same as the language underlying voice recognition, a number is not displayed. This is because there is a high possibility that the user will say the first word of the text object to select the text object.

Meanwhile, according to another embodiment of the present disclosure, since an image object cannot be selected by voice, a number may be displayed on the image object. This embodiment will be described with reference to FIG. 7 below.

7 illustrates that a specific screen is displayed on the display 110 .

Referring to FIG. 7 , a first image object 71 , a second image object 72 , a third image object 74 , a first text object 73 , and a second text object 75 are displayed on the display 110 . is indicated on The processor 120 may control the display 110 to display the number ① together with the first image object 71 .

Meanwhile, according to another embodiment of the present disclosure, when a plurality of objects displayed on the display 110 are objects having URL links, the processor 120 compares the URL links of the plurality of objects, and the same URL link In the case where there are objects having , if all of the corresponding objects are objects that cannot be selected by voice recognition, the display 110 can be controlled to display a number on only one object, and any one of these objects can be voiced. If an object can be selected through recognition, the display 110 may be controlled not to display a number.

More specifically, a plurality of objects that cannot be selected by voice recognition (ie, text objects or image objects different from the language used for voice recognition) are displayed on the display 110, and they have the same URL. In the case of having an address link, a number can be displayed only in one object. Referring to FIG. 7, the second image object 72 is an object that cannot be selected by voice, and the text object 73 is composed of English, a language different from Korean, which is the language underlying voice recognition. Therefore, both the second image object 72 and the first text object 73 cannot be selected by voice, but the second image object 72 and the first text object 73 are connected to the same URL address when selected. Therefore, the number (②) may be displayed only on the second image object 72, which is either one of the second image object 72 and the first text object 73. Alternatively, numbers may be displayed on the text object 73 instead of the second image object 72 . This is to minimize the number of numbers displayed on the display 110 .

In order to minimize the number of numbers displayed on the display 110, according to another embodiment of the present disclosure, a plurality of objects having the same URL address are displayed on the display 110, and among them If any one of the text objects is the same as the language that is the basis of voice recognition, numbers are not displayed for all of them. Referring to FIG. 7 , the processor 120 compares the URL address of the third image object 74 and the ULR address of the second text object 75, determines that they are the same, and determines that the URL address of the second text object 75 is the same. ) is determined to be the same text object as Korean, which is the language underlying voice recognition, the display 110 is controlled so as not to display numbers on the third image object 74 .

When the recognition result of the voice spoken by the user includes specific text displayed on the display 110, the processor 120 may perform an operation related to a text object corresponding to the text. Referring to FIG. 5 , when the user says “Voice recognition”, the processor 120 may control the display 110 to display a page of a URL address corresponding to the text object 59 .

Meanwhile, according to an embodiment of the present disclosure, when the recognition result of a voice spoken by a user includes text commonly included in two or more text objects among text objects displayed on the display 110, the processor 120 determines the corresponding text object. A number is displayed on each of the text objects, and when the user utters the displayed number, an operation related to the text object corresponding to the number can be performed. Referring to FIG. 5 , when “Speech recognition” is included in the result of voice recognition uttered by the user, the processor 120 searches for a text object including “Speech recognition” among text objects displayed on the screen. do. When a plurality of text objects 57 and 58 are searched for, the processor 120 may control the display 110 to display a random number next to the text objects 57 and 58 . For example, the number ⑦ can be displayed next to the text object 57, the number ⑧ can be displayed next to the text object 58, and the user can select the text object 57 by saying the number “7”. When the recognition result of the voice spoken by the user includes a number displayed on the display 110, the processor 120 may perform an operation related to a text object or an image object corresponding to the included number. Referring to FIG. 6 , if the user says “work”, the processor 120 may control the display 110 to display a page of a URL address corresponding to the text object 61 .

The voice uttered by the user may be input through the microphone of the display device 100 or the external device 200 through the microphone. In the latter case, the display device 100 may include a communication unit for communicating with the external device 200 including a microphone, and the communication unit receives a voice signal corresponding to a voice input through the microphone of the external device 200. can do. When the recognition result of the voice signal received from the external device 200 through the communication unit includes a number displayed on the display 110, the processor 120 may perform an operation related to a text object corresponding to the corresponding number. Referring to FIG. 6, when the user says “work” into the microphone of the external device 200, the external device 200 transmits a voice signal to the display device 200, and the processor 120 transmits the received voice signal. The display 110 may be controlled to display a page of a URL address corresponding to the text object 61 based on a voice recognition result for .

Meanwhile, numbers displayed corresponding to text or image objects may be displayed only for a certain period of time. According to an embodiment, the processor 120 may control the display 110 to display numbers while a signal corresponding to selection of a specific button is received from the external device 200 . That is, numbers may be displayed only while the user presses a specific button of the external device 200 . Here, the specific button may be, for example, the microphone button 210 of the external device 200 described in FIG. 2 .

According to another embodiment, the processor 120 displays numbers when the voice input through the microphone of the display device 100 includes a preset keyword (eg, "Hi TV"), and When a predetermined time elapses in a state in which no voice is input through the microphone, the displayed numbers may be removed.

On the other hand, in the above embodiments, it has been described that numbers are displayed, but numbers are not necessarily displayed, and any words (words with meaning or words without meaning) that the user can see and read can be used. For example, it is also possible to display a, b, c... instead of 1, 2, 3... According to another embodiment of the present disclosure, when there is a search window in the web page displayed on the display 110, the user can easily perform a search by uttering a word to be searched for and a specific keyword that executes the search function. For example, if there is a search bar on the web page displayed on the display 110, the search result for “○○○” is displayed on the display 110 simply by saying “Search ○○○” or “Search ○○○”. It can be.

To this end, the processor 120 may detect a search word input window in the web page displayed on the display 110 . Specifically, the processor 120 may search for an object capable of being input from among constituent objects of the web page displayed on the display 110 . An input tag on HTML is an object that can be input. An input tag has various attributes, among which type attributes clearly define the characteristics of an input. If the type is "search", the object clearly corresponds to the search word input window.

However, in the case of an object whose type is "text", it cannot be immediately determined whether or not it is a search word input window. This is because general input objects also have a text type, so it cannot be distinguished whether the corresponding object is a search word input window or a general input window. Therefore, in this case, a separate process of determining whether the search word input window is required is required.

In the case of an object whose type is "text", information on additional attributes of the corresponding object is referred to in order to determine whether it is a search word input window. If there is a "search" keyword in title or aria-label, it can be determined that the object is a search word input window.

Further, the processor 120 determines whether a specific keyword is included in the recognition result of the voice spoken by the user. Here, the specific keyword may be “search” or “find”. If it is determined that a specific keyword is included, the processor 120 checks the location of the specific keyword in order to more accurately determine the user's intention. If at least one word exists before or after the specific keyword, it is highly likely that the user's intention is to search for the at least one word. If the voice recognition result includes only a specific keyword such as “search” or “find”, it is highly likely that the user does not intend to search.

This process of determining the user's intention may be performed in the display apparatus 100 or may be performed in the server 300 to provide the result to the display apparatus 100 .

When the user's search intention is determined, the processor 120 selects words other than the specific keyword as a search word, inputs the selected search word into the detected search word input window according to the above method, and performs a search. For example, as shown in FIG. 8 , when a web page including a search word input window 810 is displayed on the display 110, the processor 120 detects the search word input window 810, and the user “searches for a dog”. When a voice is uttered, the processor 120 selects “dog” as a search word from the voice recognition result for the uttered voice and inputs it into the detected search word input window 810 to perform a search.

Meanwhile, the operation of detecting a search word input box in the web page displayed on the display 110 may be performed after it is determined that a specific keyword is included in the voice recognition result, or may be performed in advance.

9 is a diagram for explaining another example of a search word input method. 9 is for explaining a method of performing a search when there are a plurality of search word input windows in one web page.

Referring to FIG. 9 , a case in which there are two search windows in one web page is shown. The first search word input window 910 is for news search, and the second search word input window 920 is for stock search. The processor 120 performs a search through the search word input window displayed when the user utters a voice including the search word, based on the information about the arrangement position of the object and the information about the layout of the current screen. For example, in a situation where the first search word input window 910 is displayed on the display 110, when the user utters a search word and a voice including a specific keyword, the processor 120 inputs the search word into the first search word input window 910, In a situation where the second search word input window 920 is displayed on the display 110 by scrolling in a downward direction, when the user utters a voice including a search word and a specific keyword, the processor 120 displays the second search word input window 920 You can enter a search term. That is, when there are a plurality of search word input windows in one web page, a search can be performed with the search word input window displayed on the current screen.

Voice control is performed based on the screen displayed on the display 110 , that is, a function according to a voice command is performed using an application corresponding to the screen being displayed on the display 110 . However, if the input voice command does not match an object included in the currently displayed screen or has a function different from that of the application currently displaying the screen, another application may be executed to perform the function according to the corresponding voice command.

For example, if the currently running application is a web browsing application and a voice uttered by the user does not match an object in a web page displayed by the web browsing application, the processor 120 executes another preset application to allow the user to A search function corresponding to the uttered voice may be performed. Here, another preset application is an application that provides a search function, for example, an application that provides search results for text corresponding to voice using the search engine of Google™, and search results for VOD content corresponding to text corresponding to voice. It may be an application that provides. On the other hand, before this other application is executed, the processor 120 displays a UI for obtaining consent from the user, such as "There is no result matching ○○○ on the current screen. Do you want to search for ○○○ on the Internet?" It can be displayed, and after user agreement is input in the UI, an Internet search application or the like can be executed to provide search results.

The display device 100 may include a voice processing unit that processes a voice recognition result received from the server 300 and an application unit that executes an application installed in the display device 100 . The voice processing unit provides the voice recognition result received from the server 300 to the application unit. When the first application of the application unit is executed and the recognition result is provided while the screen of the first application is displayed on the display 110, the first application performs the above-described operation based on the voice recognition result provided from the voice processing unit. can be done For example, text or image object search corresponding to the number included in the voice recognition result, text object search corresponding to the word included in the voice recognition result, search after entering a keyword in the search box when “search” is included in the voice recognition result. You can perform operations such as running . If there is no operation to be performed using the voice recognition result provided by the first application from the voice processing unit, that is, if there is no text or image object corresponding to the voice recognition result or there is no search window, the first application is sent to the voice processing unit. Notifying this, the voice processing unit may control the application unit to execute a second application capable of performing an operation related to a voice recognition result. For example, the second application is an application that provides search results for a specific search term. The application unit may execute the second application and provide a search result using text included in the voice recognition result as a search term.

FIG. 10 is a block diagram showing the configuration of a display apparatus 100 implemented as a TV. In describing FIG. 10 , a description of a configuration overlapping with the configuration described in FIG. 4 will be omitted.

Referring to FIG. 10 , the display device 100 includes, for example, an analog TV, a digital TV, a 3D-TV, a smart TV, an LED TV, an OLED TV, a plasma TV, a monitor, and a curved screen having a fixed curvature screen. A curved TV, a flexible TV having a screen with a fixed curvature, a bended TV having a screen with a fixed curvature, and/or a curvature variable TV capable of changing the curvature of the current screen by a received user input, etc. It may be implemented as, but is not limited thereto.

The display device 100 includes a display 110, a processor 120, a tuner 130, a communication unit 140, a microphone 150, an input/output unit 160, an audio output unit 170, and a storage unit 180. ).

The tuner 130 tunes only the frequency of a channel to be received by the display device 100 among many radio wave components through amplification, mixing, resonance, etc. of broadcast signals received by wire or wirelessly ( can be selected by tuning. The broadcast signal may include video, audio, and additional data (eg, Electronic Program Guide (EPG)).

The tuner 130 may receive video, audio, and data in a frequency band corresponding to a channel number corresponding to a user input.

The tuner 130 may receive broadcast signals from various sources such as terrestrial broadcasting, cable broadcasting, or satellite broadcasting. The tuner 130 may receive broadcast signals from various sources such as analog broadcasting or digital broadcasting.

The tuner 130 is implemented as an all-in-one with the display device 100 or a separate device having a tuner unit electrically connected to the display device 100 (for example, a set-top box (set-top box)). top box) and a tuner connected to the input/output unit 160).

The communication unit 140 is a component that performs communication with various types of external devices according to various types of communication methods. The communication unit 140 may be connected to an external device through a local area network (LAN) or an Internet network, and wireless communication (eg, Z-wave, 4LoWPAN, RFID, LTE D2D, BLE, GPRS, Weightless , Edge Zigbee, ANT+, NFC, IrDA, DECT, WLAN, Bluetooth, Wi-Fi, Wi-Fi Direct, GSM, UMTS, LTE, WiBRO, etc.) The communication unit 140 includes various communication chips such as a Wi-Fi chip 141, a Bluetooth chip 142, an NFC chip 143, and a wireless communication chip 144. The Wi-Fi chip 141, the Bluetooth chip 142, and the NFC chip 143 each perform communication using a WiFi method, a Bluetooth method, and an NFC method. The wireless communication chip 174 refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE). In addition, the communication unit 140 includes a light receiving unit 145 capable of receiving a control signal (eg, an IR pulse) from the external device 200 .

The processor 120 may transmit a voice signal and language information (information on a language that is a basis for voice recognition) to the server 300 through the communication unit 140, and the server 300 may transmit a language corresponding to the language information. When a result of voice recognition performed on the voice signal is transmitted using the voice recognition engine of , the result of voice recognition may be received through the communication unit 140 .

The microphone 150 may receive a voice uttered by a user and generate a voice signal corresponding to the received voice. The microphone 150 may be integrally implemented with the display device 100 or may be separated. The separated microphone 150 may be electrically connected to the display device 100 .

When the display device 100 does not have a microphone, the display device 100 may receive a voice signal corresponding to a voice input through the microphone of the external device 200 from the external device 200 through the communication unit 140. there is. The communication unit 140 may receive a voice signal from the external device 200 through a communication method such as Wi-Fi or Bluetooth.

The input/output unit 160 is a component for connecting with an external device. The input/output unit 160 may include at least one of a high-definition multimedia interface port 161, a component input jack 162, and a USB port 163. In addition to those shown, the input/output unit 180 may include at least one of ports such as RGB, DVI, HDMI, DP, and Thunderbolt.

The audio output unit 170 is a component for outputting audio, for example, audio included in a broadcast signal received through the tuner 130 or input through the communication unit 140 or the input/output unit 160. Audio or audio included in an audio file stored in the storage unit 180 may be output. The audio output unit 170 may include a speaker 171 and a headphone output terminal 172 .

The storage unit 180 may include various application programs, data, and software modules for driving and controlling the display device 100 under the control of the processor 120 . For example, the storage unit 180 may include a web parsing module for parsing web content data received through an Internet network, a JavaScript module, a graphic processing module, a voice recognition result processing module, and an input processing module.

When voice recognition is performed by the display device 100 itself rather than the external server 300, voice recognition modules including various voice recognition engines suitable for various languages may be stored in the storage unit 180.

The storage unit 180 may store data for composing various UI screens provided by the display 110 . Also, the storage unit 180 may store data for generating control signals corresponding to various user interactions.

The storage unit 180 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). Meanwhile, the storage unit 180 may be implemented as a storage medium in the display device 100 as well as an external storage medium such as a micro SD card, a USB memory, or a web server through a network.

The processor 120 controls the overall operation of the display device 100 and the flow of signals between internal components of the display device 100 and processes data.

Processor 120 includes RAM 121 , ROM 122 , CPU 123 and bus 124 . RAM 121 , ROM 122 , CPU 123 , etc. may be connected to each other through a bus 124 . The processor 120 may be implemented as a system on chip (SoC).

The CPU 123 accesses the storage unit 180 and performs booting using the O/S stored in the storage unit 180 . In addition, various operations are performed using various programs, contents, and data stored in the storage unit 180 .

The ROM 122 stores command sets for system booting and the like. When a turn-on command is input and power is supplied, the CPU 123 copies the O/S stored in the storage 180 to the RAM 121 according to the command stored in the ROM 122 and executes the O/S. boot up the system When booting is completed, the CPU 123 copies various application programs stored in the storage unit 180 to the RAM 121 and executes the copied application programs in the RAM 121 to perform various operations.

The processor 120 may perform various operations using modules stored in the storage unit 180 . For example, the processor 120 may parse and process web content data received through an internet network to display the overall layout of the corresponding content and each object on the display 110 .

When the voice recognition function is activated, the processor 120 analyzes the objects of the web content to find objects that can be controlled by voice, and pre-processes information such as the location of the object, an operation related to the object, whether or not text is included in the object, and the like. It is possible to store the preprocessing results in the storage unit 180.

Further, the processor 120 may control the display 110 to identify and display objects controllable (selectable) by voice based on the pre-processed object information. For example, the processor 120 may control the display 110 to display the colors of objects controllable by voice differently from other objects.

The processor 120 may recognize the voice input through the microphone 150 as text using a voice recognition engine. In this case, the processor 120 uses a voice recognition engine of a predetermined language (a language set as a language to be based on voice recognition). Alternatively, the processor 120 may transmit a voice signal and information about a language that is the basis of voice recognition to the server 300 and receive text as a voice recognition result from the server 300 .

Also, the processor 120 may search for an object corresponding to the voice recognition result among the pre-processed objects, and display that the object has been selected at the position of the searched object. For example, the processor 120 may control the display 110 to highlight an object selected by voice. Further, the processor 120 may perform an operation related to an object corresponding to a voice recognition result based on the pre-processed object information and output the result through the display 110 or the audio output unit 170 .

11 is a flowchart for explaining a control method of the display apparatus 100 according to an embodiment of the present disclosure. The flowchart shown in FIG. 11 may consist of operations processed in the display apparatus 100 described herein. Therefore, even if the content is omitted below, the description of the display device 100 can be applied to the flowchart shown in FIG. 11 .

Referring to FIG. 11 , the display device 100 first displays a UI screen including a plurality of text objects (S1110).

Then, among the plurality of text objects displayed on the display device, text objects different from the predetermined language are displayed together with preset numbers (S1120). Here, the predetermined language means a predetermined language as a basis for voice recognition. The language to be the basis of voice recognition may be set as a default language, set manually by a user, or set automatically based on a language constituting objects displayed on the display device 100 . In the case of automatic setting, for example, optical character recognition (OCR) may be applied to objects displayed on the display device 100 to determine the language constituting the objects.

Then, if the recognition result of the voice spoken by the user includes the displayed number, an operation related to the text object corresponding to the displayed number is performed (S1130).

The result of recognizing the voice spoken by the user can be obtained by the display device's own voice recognition, or can be received by requesting voice recognition from an external server that performs voice recognition for a plurality of different languages. In the latter case, the display apparatus 100 provides information on a voice signal corresponding to a voice uttered by a user and a language set as a language underlying voice recognition to an external server, and the voice recognition result received from the external server is If the displayed number is included, an operation related to the text object corresponding to the displayed number can be performed.

For example, if the text object is hyperlinked text in a web page, a web page displaying operation of a URL address corresponding to the text object may be performed, and if the text object is an icon for running an application, the corresponding application may be executed. .

Meanwhile, the UI screen including the plurality of text objects may be an execution screen of the first application. The execution screen of the first application may be any screen provided by the first application. If it is determined that the object corresponding to the recognition result of the voice spoken by the user while the running screen of the first application is displayed is not on the running screen of the first application, the display device executes a second application different from the first application. Thus, an operation corresponding to the result of recognizing the voice may be performed. Here, the first application may be a web browsing application, and the second application may be an application that searches various sources, such as the Internet, data stored in a display device, VOD content, channel information (eg EPG), and the like. For example, if there is no object corresponding to voice recognition in the currently displayed web page, the display device executes another application to provide search results (eg, Google search results, VOD search results, channel search results, etc.) corresponding to voice recognition. can do.

According to various embodiments described above, voice control for objects configured in various languages is possible, and voice search can be more easily performed.

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. According to the hardware implementation, the embodiments described in this disclosure are application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). ), processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. In some cases, the embodiments described herein may be implemented by the processor 120 itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing processing operations in the display apparatus 100 according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. can When the computer instructions stored in the non-transitory computer readable medium are executed by the processor of the specific device, the processing operation in the display device 100 according to various embodiments described above is performed by the specific device.

A non-transitory computer readable medium is a medium that stores data semi-permanently and is readable by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of the non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is common in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: display device
110: display
120: processor

Claims (21)

In the display device,
display; and
Control the display to display a UI screen including a plurality of text objects;
Identifying a first text object including text in a first language, which is a basis for voice recognition, and a second text object including text in a second language different from the first language, among the plurality of text objects;
For the second text object among the plurality of text objects, control the display to display a preset number adjacent to the second text object;
and a processor configured to perform an operation related to the second text object when a result of the voice recognition for the voice spoken by the user includes the preset number. delete According to claim 1,
The UI screen is a web page,
the processor,
A display device configured to set a language corresponding to the language information of the web page as the first language. According to claim 1,
the processor,
A display device that determines that a text object composed of two or more languages among the plurality of text objects is the second text object when the content ratio of the first language is less than a preset ratio. delete According to claim 1,
It further includes; a communication unit that performs communication with an external device;
the processor,
A display device that controls the display to display the preset number while receiving a signal corresponding to selection of a specific button of the external device. According to claim 6,
The external device includes a microphone,
The communication department,
Receiving a voice signal corresponding to a voice input through a microphone of the external device;
the processor,
and performing an operation related to the second text object when a result of the voice recognition for the received voice signal includes the preset number. According to claim 7,
the processor,
and performing an operation related to the first text object when a result of the voice recognition for the received voice signal corresponds to the first text object. According to claim 1,
The operation related to the second text object,
Displaying a web page of a URL address corresponding to the second text object or executing an application program corresponding to the second text object. According to claim 1,
The plurality of text objects are included in the execution screen of the first application,
the processor,
If it is determined that the object corresponding to the result of the voice recognition for the voice uttered by the user while the running screen of the first application is displayed is not on the running screen of the first application, a second application different from the first application A display device that executes an operation corresponding to a result of the voice recognition. According to claim 10,
The second application is an application that provides search results for a search term,
the processor,
When it is determined that the object corresponding to the result of the voice recognition for the voice uttered by the user while the running screen of the first application is displayed is not on the running screen of the first application, the second application is executed to recognize the voice. A display device that provides a search result using text corresponding to the result as a search term. According to claim 1,
A communication unit configured to communicate with a server that performs voice recognition for a plurality of different languages; further comprising;
the processor,
Controls the communication unit to provide a voice signal corresponding to a voice spoken by the user and information on the first language to the server, and when a result of voice recognition received from the server includes the preset number, the device A display device that performs an operation related to the second text object corresponding to the set number. According to claim 12,
the processor,
When a result of voice recognition received from the server corresponds to the first text object, performing an operation related to the first text object. In the control method of the display device,
displaying a UI screen including a plurality of text objects;
identifying a first text object including text in a first language, which is a basis for voice recognition, and a second text object including text in a second language different from the first language, among the plurality of text objects;
displaying a predetermined number adjacent to the second text object among the plurality of text objects; and
and performing an operation related to the second text object when a result of the voice recognition for the voice spoken by the user includes the preset number. delete According to claim 14,
The UI screen is a web page,
The control method of the display device,
The method of controlling a display device further comprising setting a language corresponding to the language information of the web page as the first language. According to claim 14,
For a text object composed of two or more languages among the plurality of text objects, determining that the second text object is the second text object when the content rate of the first language is less than a preset rate; Control method of a display device further comprising. delete According to claim 14,
In the step of displaying the preset number,
A control method of a display device displaying the preset number while receiving a signal corresponding to selection of a specific button of the external device from an external device. A computer-readable recording medium storing a program for executing a control method of a display device,
The control method of the display device,
controlling the display device to display a plurality of text objects;
identifying a first text object including text in a first language, which is a basis for voice recognition, and a second text object including text in a second language different from the first language, among the plurality of text objects;
controlling the display device to display a predetermined number adjacent to the second text object among the plurality of text objects; and
and performing an operation related to the second text object when a result of the voice recognition for the voice uttered by the user includes the preset number. According to claim 1,
the processor,
A display device for controlling the display so that an icon indicating that the first text object is capable of voice recognition is displayed together with respect to the first text object among the plurality of text objects.

Images