KR20220073723A - Display apparatus, voice acquiring apparatus and voice recognition method thereof - Google Patents

Abstract

The present invention relates to a display device, a voice acquisition device, and a voice recognition method thereof, comprising: a display unit for displaying an image; a communication unit configured to communicate with a plurality of external devices; The communication unit includes a voice recognition engine for recognizing a voice uttered by a user, receives a voice signal from a voice acquisition unit, and receives a candidate command for recognizing the received voice signal from at least one of the plurality of external devices Includes a control unit for controlling the. In this way, it is possible to increase the utilization of various voice acquisition equipment, to provide linked services using them, and to dynamically register/delete candidate commands for voice recognition. It is possible to increase the efficiency of the voice recognition system.

Description

Display device, voice acquisition device, and voice recognition method thereof

The present invention relates to a display device, a voice acquisition device, and a voice recognition method thereof, and more particularly, to a display device for recognizing a voice uttered by a user, a voice acquisition device, and a voice recognition method thereof.

The use of electronic devices having a voice recognition function for recognizing a voice uttered by a user is gradually increasing. Voice recognition is being actively used not only in PCs and mobile communication devices, but also in various electronic devices such as digital TVs, air conditioners, and home theaters.

In order to perform this voice recognition function, a microphone for receiving a voice uttered by a user and a voice recognition engine for recognizing the inputted voice are provided in a main device such as a TV, and the voice recognition engine stores the voice input into the microphone in advance. Compared with the command, the uttered voice may be recognized according to the result.

However, in the case of a conventional electronic device having a voice recognition function, since a means for receiving a voice uttered by a user is fixed, it is difficult to utilize various input means capable of voice input such as a mobile phone. In addition, although the recognition rate increases as the number of candidate commands increases, as the number of candidate commands increases, the number of comparison targets increases, which slows down the speech recognition processing speed, and the storage capacity of the main device is limited, so candidate commands are continuously added. has the disadvantage of being difficult.

A display apparatus according to an embodiment of the present invention includes: a display unit for displaying an image; a communication unit configured to communicate with a plurality of external devices; The communication unit includes a voice recognition engine for recognizing a voice uttered by a user, receives a voice signal from a voice acquisition unit, and receives a candidate command for recognizing the received voice signal from at least one of the plurality of external devices Includes a control unit for controlling the.

A plurality of voice acquisition units may be provided, and when a voice input is sensed from at least one of the plurality of voice acquisition units, the control unit may receive a voice signal from the voice acquisition unit in which the voice input is sensed.

The voice acquisition unit includes at least one of a built-in microphone provided in the display device, a first external microphone provided in at least one of the plurality of external devices, and a second external microphone separated from the display device and the plurality of external devices can do.

The external device may include at least one application, and the candidate command may be managed by the application.

It manages the candidate command and may further include a built-in application provided in the display device.

A storage unit for storing the received candidate command may be further included, and the speech recognition engine may recognize a voice using the received candidate command.

When it is detected that a wakeup keyword for any one of the plurality of voice acquisition units is uttered, the control unit activates the voice acquisition unit corresponding to the detected wakeup keyword, and receives a voice signal from the activated voice acquisition unit can receive

When a trigger signal according to operation of a predetermined button provided in any one of the plurality of voice acquisition units is input, the control unit activates the voice acquisition unit corresponding to the input trigger signal, and receives a voice signal from the activated voice acquisition unit can receive

The control unit may control the display unit to display a voice recognition result for the voice signal and a candidate command according to the voice recognition result.

The display unit may further display application information for managing the candidate command.

The voice recognition engine may recognize the voice by determining a command matching or similar to the received voice signal from among the received candidate commands.

On the other hand, the voice acquisition device according to an embodiment of the present invention, the communication unit for performing communication with a display device having a voice recognition function; a voice acquisition unit receiving the voice uttered by the user; a voice converter for converting the input voice into an electrical voice signal; and a controller for controlling the communication unit to transmit the converted voice signal and a candidate command for recognizing the voice signal to the display device.

It further includes at least one application, and the candidate command may be managed by the application.

On the other hand, the voice recognition method of the display apparatus according to an embodiment of the present invention, the step of receiving a voice signal from the voice acquisition unit; receiving a candidate command for recognizing the received voice signal from at least one of a plurality of external devices; and recognizing the voice uttered by the user according to the received voice signal and the candidate command.

The method may further include detecting a voice input from at least one of the plurality of voice acquisition units, wherein the receiving of the voice signal may include receiving a voice signal from the voice acquisition unit in which the voice input has been sensed.

The voice acquisition unit includes at least one of a built-in microphone provided in the display device, a first external microphone provided in at least one of the plurality of external devices, and a second external microphone separated from the display device and the plurality of external devices can do.

The external device may include at least one application, and the candidate command may be managed by the application.

It manages the candidate command and may further include a built-in application provided in the display device.

The method may further include storing the received candidate command, and the step of recognizing the voice may recognize a voice using the received candidate command.

The detecting of the voice input may include: detecting that a wakeup keyword for any one of the plurality of voice acquisition units is uttered; It may include activating a voice acquisition unit corresponding to the detected wakeup keyword.

The step of detecting the voice input may include: detecting that a trigger signal according to manipulation of a predetermined button provided in any one of the plurality of voice acquisition units is input; It may include activating the voice acquisition unit corresponding to the input trigger signal.

The method may further include displaying a voice recognition result for the voice signal and a candidate command according to the voice recognition result.

The displaying may further display application information for managing the candidate command.

In the step of recognizing the voice, the voice may be recognized by determining a command matching or similar to the received voice signal from among the received candidate commands.

1 is an exemplary view of a voice recognition system according to an embodiment of the present invention,
2 is a block diagram showing the configuration of a voice recognition system according to an embodiment of the present invention;
3 is a view for explaining an example of performing voice recognition according to an embodiment of the present invention;
4 is a view showing an example of a screen displayed as a result of voice recognition of FIG. 3;
5 is a view for explaining an example of performing voice recognition according to another embodiment of the present invention,
6 is a flowchart illustrating a voice recognition method of a voice recognition system according to an embodiment of the present invention;
7 is a flowchart illustrating the process of detecting the voice input of FIG. 6 in more detail;
8 is a flowchart illustrating a process of performing the voice recognition of FIG. 6 in more detail.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram of a voice recognition system according to an embodiment of the present invention.

1, the voice recognition system according to the embodiment of the present invention includes a main device 100, a plurality of voice acquisition devices 201, ..., 202, and a plurality of external devices 301, 302, .. . , 303). The main device 100, the plurality of voice acquisition devices 201, 202, and the plurality of external devices 301, 302, 303 are connected to enable mutual communication.

The main device 100 is provided with a voice acquisition unit 140 such as a microphone into which the voice uttered by the user is input and a voice recognition engine 181 for recognizing the input voice, and through the communication unit 160, the voice acquisition device ( 201 and 202) and external devices 301, 302, and 303. In addition, one or more built-in applications 171 and 172 (native applications) are provided in the main device 100 , and the main device 100 is driven to perform various functions (services). In the built-in applications 171 and 172 of this embodiment, candidate commands corresponding to the functions to be performed, that is, included in a service scenario that can be performed, are stored in advance, and the candidate commands stored in the built-in applications 171 and 172 are voice recognition engines during voice recognition. It is transmitted to 181, so that the voice recognition engine 181 can perform voice recognition.

The voice acquisition devices 201 and 202 include a voice acquisition unit such as a microphone for receiving a voice uttered by a user, and a voice signal corresponding to the input voice is transmitted to the main device 100 for voice recognition.

The voice acquisition devices 201 and 202 receive the voice uttered by the user, convert it into an electrical voice signal, and transmit it to the main device 100 . The voice acquisition devices 201 and 202 are external devices capable of wireless communication with the main device 100, and the wireless communication is wireless LAN, radio frequency (RF) communication, Bluetooth, Zigbee, Including infrared (IR) communication and the like.

One or more external applications (Dev. Applications) are provided in the external devices 301 , 302 , and 303 to perform functions (services) required by each of the external devices 301 , 302 , 303 . Candidate commands corresponding to functions performed by the corresponding external devices 301, 302, and 303 are stored in advance in the external application of this embodiment, and the candidate commands stored in the external application are transmitted to the voice recognition engine 181 during voice recognition. , so that the voice recognition engine 181 can perform voice recognition.

Here, the candidate commands registered in advance in the built-in applications 171 and 172 and the external application may be commands related to functions/operations of the corresponding applications. For example, when the main device 100 is a TV, candidate commands related to channel movement, volume control, etc. of the TV may be stored in any one of the built-in applications 171 and 172, and the external device 302 is an air conditioner. In this case, candidate commands related to temperature control (up/down), wind strength control (strong/weak wind/breeze), etc. may be stored in an application included in the external device 302 .

The present invention may be implemented such that an external device or a voice acquisition device includes both a voice acquisition unit and an external application, as in the external device 1 301 of FIG. 1 . In this case, when a voice is input from the voice acquisition unit provided in the external device 1 301 , a candidate command stored in advance in the external application of the external device 1 301 is transmitted to the voice recognition engine 181 of the main device 100 . , voice recognition may be performed.

The voice recognition system of this embodiment includes at least one voice acquisition unit, and when a voice input is sensed from the voice acquisition unit, the voice acquisition unit in which the voice input is sensed is activated to receive a voice stream. When a plurality of voice acquisition units are provided, a voice stream may be received by activating a voice acquisition unit in which a voice input is sensed among the plurality of voice acquisition units. Here, the plurality of voice acquisition units include a built-in microphone provided in the main device 100 , a first external microphone provided in at least one of the plurality of external devices 301 , 302 , and 303 , the main device 100 and a plurality of external devices It may include a second external microphone provided in a separate voice acquisition device (201, 202) separated from (301, 302, 303).

When it is detected that a wakeup keyword for any one of the plurality of voice acquisition units is uttered, the main device 100 activates the voice acquisition unit corresponding to the detected wakeup keyword, and from the activated voice acquisition unit A voice signal can be received. In addition, the main device 100 activates the voice acquisition unit corresponding to the input trigger signal when a trigger signal according to operation (event occurrence) of a predetermined button provided in any one of the plurality of voice acquisition units is input, and , it is possible to receive a voice signal from the activated voice acquisition unit.

The main device 100 of this embodiment operates in the voice recognition mode, and when any one of the voice acquisition units is activated by a wake-up keyword or a trigger signal, the other voice acquisition unit is deactivated to prevent an error in voice recognition from occurring. have. The main device 100 can operate in a remote or short-distance voice recognition mode accordingly. In addition, a UI indicating a voice acquisition unit connected to the display unit 130 to be described later is displayed to help the user's convenience.

And, the main device 100 may receive a candidate command for recognizing the received voice signal from at least one of the plurality of external devices (301, 302, 303). The received candidate command may be transmitted to the voice recognition engine 181 for voice recognition.

The plurality of external devices 301 , 302 , and 303 include at least one application, and a candidate command may be managed by each corresponding application. Also, the main device 100 further includes built-in applications 171 and 172, and the built-in applications 171 and 172 manage corresponding candidate commands. Candidate commands managed by the built-in applications 171 and 172 can be delivered to the voice recognition engine 181 for voice recognition.

The main device 100 of the present invention may be implemented as a display device such as a TV as shown in FIG. 2. FIG. 2 is a block diagram showing the configuration of a voice recognition system according to an embodiment of the present invention.

The display apparatus 100 processes an image signal provided from an external image supply source (not shown) according to a preset image processing process so that the image can be displayed as an image.

In the system of this embodiment, the display device 100 is implemented as a TV or set top box that processes a broadcast image based on a broadcast signal/broadcast information/broadcast data received from a transmission device of a broadcast station. . However, since the spirit of the present invention is not limited to implementation examples of the display apparatus 100 , various types of implementation examples capable of processing images in addition to TVs and set-top boxes may be applied to the display apparatus 100 .

In addition, the display device 100 is not limited to the type of image displayable to the broadcast image, for example, the display device 100 is a video based on signals / data received from various types of image sources (not shown), Processing can be performed to display images such as still images, applications, on-screen displays (OSDs), and graphic user interfaces (GUIs) for controlling various operations.

According to an embodiment of the present invention, the display apparatus 100 may be implemented as a smart TV. A smart TV can receive and display broadcast signals in real time, and has a web browser function to display real-time broadcast signals and simultaneously search and consume various contents through the Internet. to be. In addition, since the smart TV includes an open software platform, it can provide interactive services to users. Accordingly, the smart TV can provide a user with various contents, for example, an application that provides a predetermined service through an open software platform. These applications are applications that can provide various types of services, and include, for example, applications that provide services such as SNS, finance, news, weather, maps, music, movies, games, and e-books.

A voice recognition engine 181 for recognizing a user's voice is provided in the display device 100 of the present embodiment. A command corresponding to the recognized voice, that is, a control command, is transmitted to a corresponding application to perform an operation according to a corresponding function. Here, when the application corresponding to the control command is any one of the built-in applications 171 and 172 , the display apparatus 100 performs an operation according to the control command by the corresponding application. In addition, when the application corresponding to the control command is an external application, the control command is transmitted to the external devices 301 , 302 , 303 including the external application, and the external devices 301 , 302 , 303 are executed by the corresponding application. An operation according to a control command can be performed.

Meanwhile, although FIG. 2 illustrates a case in which the main device is implemented as a display device such as a TV, the present invention is not limited thereto, and the main device may be implemented with various electronic devices, for example, a PC.

In the embodiment shown in FIG. 2 , a plurality of voice acquisition devices are provided, for example, a mobile phone 200 and a remote controller 300 . Here, the remote control 300 serves as both a voice acquisition device and an external device, and the mobile phone 200 may be a smartphone having a voice acquisition function.

The remote control 300 transmits a preset command (control command) to a corresponding device by a user's manipulation. The remote control of the present embodiment may be preset to transmit a command to the display apparatus 100 or an external device, and in some cases may be implemented as an integrated remote control for transmitting a command to a plurality of devices. Also, the remote control 300 may include a TV remote control and/or an air conditioner remote control.

On the other hand, the voice acquisition device of the present invention can be implemented in various devices that can receive the voice uttered by the user, such as a mobile phone 200 and the remote control 300, as well as a portable terminal device, a microphone transmitter, and the like.

In the embodiment shown in FIG. 2 , a plurality of external devices are provided, for example, a remote controller 300 and an air conditioner 400 are implemented as an example. As described above, the remote control 300 performs both the roles of the voice acquisition device and the external device.

In the embodiment of FIG. 2 , the external device is implemented with the remote control 300 and the air conditioner 400 as an example, but the external device of the present invention is not limited thereto, and various electronic devices capable of wireless communication, for example, a home device It can be implemented as a theater, radio, VCR, DVD, washing machine, refrigerator, robot vacuum cleaner, etc. In addition, if the external device is provided with a voice acquisition unit such as a microphone, it may serve as a voice acquisition device.

Applications 372 and 472 for performing a function are provided in the external device of this embodiment, and candidate commands are stored and managed in advance in each application 372 and 472, and the candidate commands are displayed on the display device 100 for voice recognition. can be sent to

The external device, that is, the remote control 300 and the air conditioner 400 performs an operation corresponding to the control command received from the display apparatus 100 according to the voice recognition result.

Hereinafter, each configuration of the voice recognition system will be described in more detail with reference to FIG. 2 .

As shown in FIG. 2 , the display apparatus 100 includes an image receiving unit 110 for receiving an image signal, an image processing unit 120 for processing an image signal received by the image receiving unit 110 , and an image processing unit 120 . The display unit 130 for displaying the image signal processed by the image as an image, the first voice acquisition unit 140 for receiving the voice uttered by the user, the first voice conversion unit for converting the input voice into an electrical voice signal ( 150 ), a first communication unit 160 performing communication with an external device, a first storage unit 170 storing various data, and a first control unit 180 controlling the display apparatus 100 .

The image receiving unit 110 receives the image signal and transmits it to the image processing unit 120 , and may be implemented in various ways in response to the standard of the received image signal and the implementation form of the display apparatus 100 . For example, the image receiving unit 110 wirelessly receives a radio frequency (RF) signal transmitted from a broadcasting station (not shown), or composite video, component video, super video, SCART. , a video signal according to a high definition multimedia interface (HDMI) standard, etc. may be received by wire. When the video signal is a broadcast signal, the video receiver 110 includes a tuner that tunes the broadcast signal for each channel.

Also, the image signal may be input from an external device, for example, the image signal may be input from an external device such as a PC, an AV device, a smart phone, or a smart pad. In addition, the image signal may be derived from data received through a network such as the Internet. In this case, the display apparatus 100 may perform network communication through the first communication unit 160 or may further include a separate network communication unit. Also, the image signal may be derived from data stored in the nonvolatile first storage unit 170 such as a flash memory or a hard disk. The first storage unit 170 may be provided inside or outside the display apparatus 100 , and when provided outside, may further include a connection unit (not shown) to which the first storage unit 170 is connected.

The image processing unit 120 performs various preset image processing processes on the image signal. The image processing unit 120 outputs the image signal on which this process has been performed to the display unit 130 , so that the image is displayed on the display unit 130 .

The type of image processing process performed by the image processing unit 120 is not limited, and for example, decoding, de-interlacing, frame refresh rate conversion, and scaling corresponding to various image formats. It may include scaling, noise reduction for image quality improvement, detail enhancement, line scanning, and the like. The image processing unit 120 may be implemented as a group of individual components capable of independently performing each of these processes, or may be implemented as a system-on-chip (SoC) in which several functions are integrated.

The display unit 130 displays an image based on the image signal processed by the image processing unit 120 . The implementation method of the display unit 130 is not limited, and for example, liquid crystal, plasma, light-emitting diode, organic light-emitting diode, surface conduction electron gun (surface). -conduction electron-emitter), carbon nano-tube, nano-crystal, etc. can be implemented in various display methods.

The display unit 130 may additionally include additional components according to its implementation method. For example, when the display unit 130 is a liquid crystal type, the display unit 130 drives a liquid crystal display panel (not shown), a backlight unit (not shown) supplying light thereto, and a panel (not shown). and a panel driving board (not shown).

The display unit 130 of the present invention may display the voice recognition result as information on the recognized voice. Here, the voice recognition result can be displayed in various forms such as text, graphics, and icons, and the text includes letters and numbers. The display unit 130 of the present embodiment may further display candidate commands and application information according to the voice recognition result. This will be described in more detail with reference to FIG. 4 to be described later.

The user can check whether the voice is correctly recognized by the voice recognition result displayed on the display unit 130, and selects a command corresponding to the voice uttered by the user from among the displayed candidate commands by manipulating the user input unit 390 provided in the remote control. Alternatively, information related to the voice recognition result can be selected and checked.

The first voice acquisition unit 140 receives the voice uttered by the user, and may be implemented as a microphone.

The first voice conversion unit 150 converts the voice input from the first voice acquisition unit 140 into an electrical voice signal. The converted voice signal has a PCM (pulse code modulation) state or a compressed audio waveform. Here, the first voice converter 150 may be implemented as an A/D converter that digitally converts the user's input voice.

On the other hand, when the first voice acquisition unit 140 is a digital microphone, since separate A/D conversion is not required, the first voice acquisition unit 140 may include the first voice conversion unit 150 . .

The first communication unit 160 communicates with the voice acquisition device and an external device, that is, the mobile phone 200, the remote control 300, and the air conditioner 400 of FIG. 2 . The first communication unit 160 of the present embodiment may perform wireless communication, and the wireless communication includes at least one of infrared communication, RF, ZigBee, and Bluetooth.

The first storage unit 170 stores unrestricted data according to the control of the first control unit 180 . The first storage unit 170 is implemented as a non-volatile storage medium such as a flash-memory or a hard-disc drive. The first storage unit 170 is accessed by the first control unit 180 , and reading/writing/modification/deletion/update of data by the first control unit 180 is performed.

The data stored in the first storage unit 170 includes, for example, an operating system for driving the display apparatus 100 , and various applications executable on the operating system, image data, additional data, and the like.

Various data for recognizing a voice uttered by a user may be stored in the first storage unit 170 of the present embodiment. For example, the first storage unit 170 may store the command table 171 including candidate commands (hereinafter also referred to as command candidate groups) as voice recognition target information corresponding to the received voice signal. In the command table 171 , candidate commands may be managed for each application.

One or more applications for performing a function of the display apparatus 100, for example, applications 1 and 2 (172, 173), are further stored in the first storage unit 170 of the present embodiment. Applications 1 and 2 172 and 173 are driven under the control of the first controller 180 to be described later, and perform various functions of the display apparatus 100 . In FIG. 2 , a case in which two applications 172 and 173 are embedded in the display apparatus 100 is illustrated as an example, but since the functions performed by the display apparatus 100 are various, the display apparatus 100 of the present invention includes three More than one application can be embedded.

Applications 1 and 2 ( 172 , 173 ) manage candidate commands corresponding to functions to be performed, respectively. Candidate commands managed by applications 1 and 2 (172, 173) may be registered/deleted in the command table 171 as needed.

When candidate commands of the first and/or second applications 172 and 173 are registered in the command table 171 , the voice recognition engine 181 performs voice recognition using the candidate commands registered in the command table 171 . will do

Candidate commands that can be registered/deleted in the command table 171 of the present embodiment include candidate commands managed by application 3 372 of the remote controller 300 and applications managed by application 4 472 of the air conditioner 400 to be described later. may include

The first controller 180 performs control operations for various components of the display apparatus 100 . For example, the first control unit 180 controls the entire operation of the display apparatus 100 by performing an image processing process processed by the image processing unit 120 and a control operation corresponding to a command from the remote control unit 300 . control

The first control unit 180 may be implemented, for example, in a form in which software is combined with a CPU.

The first controller 180 includes a voice recognition engine 181 for recognizing the voice uttered by the user. The voice recognition function of the voice recognition engine 181 may be performed using a known voice recognition algorithm. For example, the voice recognition engine 181 extracts a voice feature vector of a voice signal, compares the extracted voice feature vector with candidate commands stored in the command table 171 of the first storage unit 170 to obtain a voice. can recognize In addition, when there is no command matching the extracted voice feature vector among the candidate commands stored in the command table 171 , the voice may be recognized by correcting the voice recognition result with a command with high similarity. Here, when there are a plurality of candidate commands having a high degree of similarity, the first control unit 180 may display a plurality of candidate commands on the display unit 130 , and a user may select any one command.

The voice recognition engine 181 of this embodiment is described as an example implemented as an embedded voice recognition engine 181 residing in the CPU, but the present invention is not limited thereto. For example, the voice recognition engine 181 may be implemented as a device embedded in the display device 100 separate from the CPU, that is, a separate chip such as a microcomputer.

Meanwhile, the present invention includes a case in which the voice recognition engine 181 is provided in a server physically separated from the display device 100 (hereinafter, also referred to as a cloud server (not shown)). The cloud server communicates with the display device 100 through a network such as the Internet. Here, the network may be a wired or wireless network. In this case, the voice recognition engine 181 may be implemented as an embedded voice recognition engine residing in the CPU of the cloud server, or a device embedded in the cloud server separately from the CPU, that is, a separate chip such as a microcomputer. .

The first controller 180 may perform an operation corresponding to the recognition result of the voice recognition engine 181 . For example, if the display device 100 is a TV, when the user is watching a movie or news, the voice recognition engine 181 recognizes that the volume is up, down, or loud or low, the first controller Reference numeral 180 may adjust the sound volume (volume) of a movie or news to correspond thereto.

As a result of the recognition of the voice recognition engine 181 , the first controller 180 provides a control command corresponding to the recognized voice when the recognized voice is a voice for controlling an external device such as the remote control 300 or the air conditioner 400 . The first communication unit 160 may be controlled to transmit to the corresponding external device. For example, when the voice recognition engine 181 recognizes that the temperature rises, the first controller 180 recognizes it as a voice for the air conditioner and transmits a command to increase the temperature of the air conditioner to the air conditioner 400 . The first communication unit 160 may be controlled.

Hereinafter, a detailed configuration of the mobile phone 200 will be described.

As shown in FIG. 2 , the mobile phone 200 is a voice acquisition device, and includes a second voice acquisition unit 240 that receives a voice uttered by a user, and a second voice converter that converts the input voice into an electrical voice signal. 250 , a second communication unit 260 for communicating with the outside, a second storage unit 270 for storing non-limited data, and a second control unit 280 for controlling the mobile phone 200 .

The second voice acquisition unit 240 receives the voice uttered by the user, and may be implemented as a microphone. The second voice conversion unit 250 converts the voice input from the second voice acquisition unit 240 into an electrical voice signal. The converted voice signal has a PCM (pulse code modulation) state or a compressed audio waveform. Here, the first voice converter 250 may be implemented as an A/D converter that digitally converts the user's input voice.

On the other hand, when the second voice acquisition unit 240 is a digital microphone, since separate A/D conversion is not required, the second voice acquisition unit 240 may include the second voice conversion unit 240 . .

The second communication unit 260 communicates with the display apparatus 100 . The second communication unit 260 of this embodiment performs wired or wireless communication, and the wireless communication includes at least one of RF, ZigBee, and Bluetooth.

The second communication unit 260 of this embodiment transmits the voice signal transmitted from the second voice conversion unit 250 to the display apparatus 100 .

The second storage unit 270 stores unrestricted data according to the control of the second control unit 280 . The second storage unit 270 is implemented as a non-volatile storage medium such as a flash-memory. The second storage unit 270 is accessed by the second control unit 280 , and reading/writing/modification/deletion/update of data by the second control unit 280 is performed.

The data stored in the second storage unit 270 includes, for example, an operating system for driving the mobile phone 200, various applications executable on the operating system, image data, additional data, and the like.

The second control unit 280 performs control operations for various components of the mobile phone 200 . For example, the second controller 280 may generate a command corresponding to a user's manipulation, perform an operation corresponding to the generated command, and display the result on a display unit (not shown).

The second control unit 280 may be implemented in a form in which software is combined with, for example, a micro controller unit (MCU).

When the voice uttered by the user is input through the second voice acquisition unit 240, the second control unit 280 of this embodiment controls the second voice conversion unit 250 to convert it into an electrical voice signal, and converts it The second communication unit 260 is controlled to transmit the received voice signal to the display apparatus 100 .

Hereinafter, a detailed configuration of the remote control 300 will be described.

As shown in FIG. 2 , the remote control 300 is a voice acquisition device and an external device at the same time as a user input unit 330 for receiving a user's operation, and a third voice acquisition unit 340 for receiving a voice uttered by the user. , a third voice conversion unit 350 for converting the input voice into an electrical voice signal, a third communication unit 360 for communicating with the outside, a third storage unit 370 for storing unlimited data, and a remote control and a third control unit 380 for controlling 300 .

The user input unit 330 transmits various preset control commands or non-limited information to the third control unit 380 according to a user's manipulation and input. The user input unit 330 may be implemented as a button including a menu key, a number key, etc. installed on the outside of the remote control 300 . When the remote control 300 is a TV remote control, the user input unit 330 may further include a touch sensing unit for receiving a user's touch input and/or a motion sensing unit for sensing a motion of the remote control 300 .

The third voice acquisition unit 340 receives the voice uttered by the user, and may be implemented as a microphone.

The third voice conversion unit 350 converts the voice input from the third voice acquisition unit 340 into an electrical voice signal. The converted voice signal has a PCM (pulse code modulation) state or a compressed audio waveform. Here, the third voice converter 350 may be implemented as an A/D converter that digitally converts the user's input voice.

On the other hand, when the third voice acquisition unit 340 is a digital microphone, since separate A/D conversion is not required, the third voice acquisition unit 340 may include the third voice conversion unit 350 . .

The third communication unit 360 communicates with the display apparatus 100 . The third communication unit 360 of this embodiment performs wireless communication, and the wireless communication includes at least one of infrared communication, RF, ZigBee, and Bluetooth.

The third communication unit 360 of this embodiment transmits the voice signal transmitted from the third voice conversion unit 350 to the display apparatus 100, and is performed by the application 3 372 in the third storage unit 370 to be described later. The managed candidate command is transmitted to the display apparatus 100 .

Unlimited data is stored in the third storage unit 370 under the control of the third control unit 380 . The third storage unit 370 is implemented as a non-volatile storage medium such as a flash-memory. The third storage unit 370 is accessed by the third control unit 380 , and reading/writing/modification/deletion/update of data by the third control unit 380 is performed.

The data stored in the third storage unit 370 includes, for example, an operating system for driving the remote control 300 , various applications executable on the operating system, image data, additional data, and the like.

In the third storage unit 370 of this embodiment, one or more applications for performing a function of the remote control 300, for example, application 3 372 are further stored. The application 3 372 is driven under the control of the third controller 380 to be described later, and performs various functions of the remote controller 300 . In the present invention, application 3 ( 372 ) and application 4 ( 472 ), which will be described later, are referred to as external applications to distinguish them from applications built in the display apparatus 100 , 172 and 173 .

In FIG. 2 , a case in which one application 372 is provided in the remote control 300 is illustrated as an example, but the present invention also includes a case in which two or more applications are provided in the remote control 300 .

The application 3 372 manages a candidate instruction corresponding to a function to be performed. Candidate commands managed by the application 3 372 may be registered/deleted in the command table 171 of the display apparatus 100 as needed.

The third controller 380 performs control operations for various components of the remote control 300 . For example, the third control unit 260 may control the third communication unit 360 to generate a command corresponding to a user's manipulation of the user input unit 330 and transmit the generated command to the display apparatus 100 . can

The third control unit 380 may be implemented in a form in which software is combined with, for example, a micro controller unit (MCU).

When the voice uttered by the user is inputted through the third voice acquisition unit 340, the third control unit 380 of this embodiment controls the third voice conversion unit 350 to convert it into an electrical voice signal, and converts it The third communication unit 360 is controlled to transmit the received voice signal to the display apparatus 100 .

The third control unit 380 may transmit a candidate command managed by the application 3 372 of the third storage unit 370 to the display apparatus 100 while communicating with the display apparatus 100 . The transmitted candidate command is registered in the command table 171 of the display device 100 and is used for the voice recognition operation of the voice recognition engine 181 .

In addition, when a control command according to voice recognition is received from the display apparatus 100 , the third control unit 380 may receive it through the third communication unit 360 and perform an operation corresponding to the received control command. have.

Hereinafter, a specific configuration of the air conditioner 400 will be described.

As shown in FIG. 2 , the air conditioner 400 is an external device, and includes a fourth communication unit 340 for communicating with the outside, a fourth storage unit 470 for storing unrestricted data, and the air conditioner 400 . It includes a fourth control unit 480 to control.

The fourth communication unit 460 communicates with the display apparatus 100 . The fourth communication unit 460 of this embodiment performs wireless communication, and the wireless communication includes at least one of RF, ZigBee, and Bluetooth.

The fourth communication unit 460 of this embodiment transmits to the display apparatus 100 a candidate command managed by the application 4 472 in the fourth storage unit 370 to be described later.

The fourth storage unit 470 stores unrestricted data according to the control of the fourth control unit 480 . The fourth storage unit 470 is implemented as a non-volatile storage medium such as a flash-memory. The fourth storage unit 470 is accessed by the fourth control unit 480 , and reading/writing/modification/deletion/update of data by the fourth control unit 480 is performed.

The data stored in the fourth storage unit 470 includes, for example, an operating system for driving the mobile phone 200, various applications executable on the operating system, additional data, and the like.

One or more applications (external applications) for performing the function of the air conditioner 300, for example, application 4 472, are further stored in the fourth storage unit 470 of the present embodiment. The application 4 472 is driven under the control of the fourth controller 480 to be described later, and performs various functions of the air conditioner 400 .

In FIG. 2 , a case in which one application 472 is provided in the air conditioner 400 is illustrated as an example, but the present invention also includes a case in which two or more applications are provided in the air conditioner 400 .

Application 4 (472) manages a candidate command corresponding to the function to be performed. Candidate commands managed by the application 4 472 may be registered/deleted in the command table 171 of the display apparatus 100 as needed.

The fourth control unit 460 performs control operations for various components of the air conditioner 400 . For example, the fourth controller 260 may receive a control command corresponding to a user's manipulation of the air conditioner remote control, and perform a control operation according to the received control command, for example, temperature control.

The fourth control unit 460 may be implemented in a form in which software is combined with, for example, a micro controller unit (MCU).

The fourth control unit 460 of the present embodiment may transmit a candidate command managed by the application 4 472 of the fourth storage unit 470 to the display apparatus 100 while performing communication with the display apparatus 100 . . The transmitted candidate command is registered in the command table 171 of the display device 100 and is used for the voice recognition operation of the voice recognition engine 181 .

In addition, when a control command according to voice recognition is received from the display apparatus 100 , the fourth control unit 480 may receive it through the fourth communication unit 460 and perform an operation corresponding to the received control command. have.

According to the embodiment of Figure 2, the first control unit 180 of the display device 100 as the main device of the voice recognition system of the present invention receives the input voice from at least one of the plurality of voice acquisition units (140, 240, 340). Upon detection, the first communication unit 140 is controlled to receive a voice signal from the voice acquisition unit in which the voice input has been sensed. In addition, the first control unit 180 transmits a candidate command for recognizing the received voice signal through the first communication unit 140 at least one of the external applications 372 and 472 of the plurality of external devices 300 and 400 or built-in. The received candidate commands received from the applications 172 and 173 are registered in the command table 171 of the first storage unit 170 . The voice recognition engine 181 compares the candidate command registered in the command table 171 with the voice signal to perform voice recognition.

In this way, as the main device according to the present embodiment, the display device 100 detects voice input from various devices capable of receiving a user's utterance, so that the utilization of various voice acquisition devices can be increased. In addition, since candidate commands for voice recognition are received and used from a running application, it is possible to dynamically register/delete candidate commands for voice recognition, thereby preventing unnecessary and infinite increase of candidate commands provided by the main device. can

On the other hand, the display device 100 according to the embodiment of the present invention receives a voice recognition input from the voice acquisition unit 140 provided in the display device 100, internal applications (171, 172, 173) and external applications (372, 472) It may be implemented to receive a candidate command from at least one of the voice recognition engine 181 to perform voice recognition.

Hereinafter, with reference to FIGS. 3 and 4, an embodiment of performing voice recognition in the voice recognition system of the present invention will be described in more detail.

3 is a diagram for explaining an example of performing voice recognition according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating an example of a screen displayed as a result of voice recognition of FIG. 3 .

As shown in FIG. 3 , the display apparatus 100 may receive candidate commands stored in the command table 171 from one or more applications (including built-in applications and external applications).

Specifically, referring to FIG. 3 , commands A and B are transmitted from application 1 172 (built-in application) to the command table 171 501 , and the transmitted commands A and B are stored in the command table 171 . becomes (502). The voice recognition engine 181 registers the commands A and B stored in the command table 171 as candidate commands ( 504 ).

In addition, commands C and D are transmitted from the application 3 372 (external application) to the command table 171 ( 505 ), and the transmitted commands C and D are registered in the command table 171 ( 507 ). The voice recognition engine 181 registers the commands C and D stored in the command table 171 as candidate commands (508).

Accordingly, in the voice recognition engine 181, A, B, C, and D transmitted from the application 1 172 and the application 3 372 are registered as candidate commands.

In this way, in a state where A, B, C, and D are registered as candidate commands, the second voice acquisition unit 240 separated from the display apparatus 100 may detect that voice A is input. The sensed voice A is converted into a voice signal by the second voice conversion unit 250 and transmitted to the voice recognition engine 181 through the second and first communication units 260 and 160 (509).

The voice recognition engine 181 compares the voice signal for voice A with the registered candidate commands A, B, C, and D, and determines a matching or similar command to recognize voice A (510).

The first controller 180 transmits the recognition result to the display unit 130 ( 511 ), and a voice recognition result screen as shown in FIG. 4 may be displayed on the display unit 130 .

As shown in FIG. 4 , the display unit 130 has a UI indicating the voice recognition result A ( 60 ) and candidate commands A, B, C, D ( 61 , 62 , 63 , 64 ) according to the voice recognition result. can be displayed. In addition, a UI indicating application information (application 1) 65 for managing command A according to the voice recognition result may be further displayed on the display unit 130 .

The user can check not only the voice recognition result but also the candidate command through the UI displayed on the screen as shown in FIG. 4 , and when the voice recognition result does not match the intention of the speech, the user can select any one of the candidate commands. In addition, it is possible to obtain information about the device related to the voice recognition result through the application information.

The first controller 180 transmits a control command according to the voice recognition result as shown in FIG. 4 to the application 1 172 (512). Application 1 172 performs a control operation according to the recognized voice A under the control of the first controller 180 . For example, when the volume of voice A is turned down, the volume of the display apparatus 100 is adjusted to be small.

3 and 4, the display apparatus 100 of the present invention pre-registers commands (eg, A, B, C, D) of some applications as candidate words according to a preset condition, and When a voice utterance is detected, a voice may be recognized using a candidate command registered in advance, the result may be displayed, and a corresponding control operation may be executed.

In the embodiment of FIGS. 3 and 4 , the command of application 1 172 and application 3 372 is registered as candidate commands and the user's voice is received through the second voice acquisition unit 240 as an example. , the present invention can receive commands from various built-in and external applications to dynamically register/delete candidate commands, and includes all cases of receiving voices from various voice acquisition units.

Hereinafter, with reference to FIG. 5, another embodiment for performing voice recognition in the voice recognition system of the present invention will be described in more detail.

5 is a view for explaining an example of performing voice recognition according to another embodiment of the present invention.

As shown in FIG. 5 , the third voice acquisition unit 340 separated from the display apparatus 100 may detect that E is voice input. The sensed voice E is converted into a voice signal by the third voice conversion unit 350, and is transmitted to the voice recognition engine 181 through the third and first communication units 360 and 160 (701).

The display apparatus 100 may receive a registration of a candidate command stored in the command table 171 from the application 3 372 corresponding to the device in which the voice input is detected. Specifically, referring to FIG. 5 , commands E and F are transmitted from the application 3 372 to the command table 171 ( 702 ), and the transmitted commands E, F are stored in the command table 171 ( 703 ). . The voice recognition engine 181 registers the commands E and F stored in the command table 171 as candidate commands (705).

Accordingly, in the voice recognition engine 181 , E and F transmitted from the application 3 372 are registered as candidate commands.

In this way, in a state where E and F are registered as candidate commands, the voice recognition engine 181 compares the voice signal for voice E with the registered candidate commands E and F and determines a match or similar command to recognize voice E. (706).

The first controller 180 transmits the recognition result to the display unit 130 ( 707 ), and a voice recognition result screen may be displayed on the display unit 130 .

The first controller 180 transmits a control command according to the voice recognition result to the application 3 (372) (708). The application 3 372 performs a control operation according to the recognized voice E under the control of the third controller 380 . Here, when the control command according to the voice recognition result is a command to control the display apparatus 100 , the control command may be transmitted to the application 1 172 or the application 2 173 .

As described in FIG. 5 , when a user's voice utterance is detected, the display apparatus 100 of the present invention registers commands (eg, E, F) of an application corresponding to the device to which the voice is input as candidate commands, and , it is possible to recognize a voice using the registered candidate command, display the result, and execute a corresponding control operation. Candidate commands can be dynamically registered/deleted.

In the embodiment of FIG. 5, a case in which the user's voice is input through the third voice acquisition unit 340 and the command of the application 3 (372) is registered as a candidate command has been described as an example, but the present invention is based on various voice acquisition units. Of course, it includes all cases in which a voice input can be received and a candidate command is dynamically registered/deleted by receiving commands from various built-in and external applications.

Hereinafter, a voice recognition method of the voice recognition system according to the present embodiment will be described with reference to the drawings.

6 is a flowchart illustrating a voice recognition method of a voice recognition system according to an embodiment of the present invention.

As shown in FIG. 6 , the voice recognition system of the present invention may detect a voice input from at least one of the plurality of voice acquisition units 140 , 240 , 340 ( S810 ). Here, the sensed user's voice is converted into an electrical voice signal in the corresponding voice conversion units 150 , 250 , 350 .

The first controller 180 receives the voice signal detected and converted in step S810 (S820). Here, when a voice input is sensed by the external voice acquisition units 240 and 340 , a voice signal may be received through the first communication unit 160 .

Candidate commands for voice recognition of the voice signal received in step S820 are registered in the voice recognition engine 181 (S830). Here, the registered candidate command may be stored in advance in the command table 171 , or received from the built-in or external applications 172 , 173 , 372 , and 472 and dynamically stored in the command table 171 .

The voice recognition engine 181 recognizes the voice uttered by the user in step S810 using the candidate command registered in step S830 (S840).

The first control unit 180 displays the voice recognition result of step S840 on the display unit 130 . Here, the display unit 130 may display a voice recognition result for a voice signal, a candidate command according to the voice recognition result, application information, and the like.

The first control unit 180 generates a control command according to the voice recognition result of step S840 and transmits it to an application capable of performing it (S860). Accordingly, an operation according to the generated control command may be performed.

7 is a flowchart illustrating in more detail a process of detecting a voice input of FIG. 6 .

As shown in FIG. 7 , the display apparatus 100, which is the main device, may operate in a voice input mode in which a user input by voice is possible (S811). The voice input mode in the present invention is operated to enable voice input from various voice acquisition units 140 , 240 , 340 .

As one of the methods of detecting a voice input, the first controller 180 may detect an utterance of the wakeup keyword in any one of the plurality of voice acquisition units 140 , 240 , 340 ( S812 ). Here, the wakeup keyword activates a voice input through a specific voice acquisition unit, and may be preset. For example, the first voice acquisition unit 140 of the display apparatus 100 may set a voice related to TV control such as a channel and volume as a wake-up keyword, and the second voice acquisition unit 240 of the mobile phone 200 ) can be set as a wake-up keyword for voice related to calls and contacts.

Meanwhile, as another method of detecting a voice input, the first control unit 180 receives a trigger signal according to operation of a predetermined button (voice input button) from any one of the plurality of voice acquisition units 140, 240, 340. , it is possible to detect a voice input from the corresponding voice acquisition unit (S813). For example, when the user operates a voice input button provided in a specific voice acquisition device, it is sensed that a voice is input from the voice input unit provided in the device.

The first controller 180 activates any one of the plurality of voice acquisition units 140 , 240 , 340 corresponding to the voice input according to the detection result of steps S812 and S813 ( S812 ). By activating any one of the voice acquisition units in this way, it is possible to prevent unnecessary voices from being detected and causing confusion.

Then, a voice signal is received from the voice acquisition unit activated in step S820 to the voice recognition engine 181, and voice recognition can be performed.

8 is a flowchart illustrating a process of performing the voice recognition of FIG. 6 in more detail.

As shown in FIG. 8 , as in step S830 of FIG. 6 , the voice recognition engine 181 may receive and register candidate commands from one or more predetermined applications among the plurality of applications 172 , 372 , and 472 .

The voice recognition engine 181 may determine whether the candidate command registered in step S830 matches/similar to the voice signal received in step S820 of FIG. 6 ( S841 ).

As a result of the determination in step S840, if there is a matching/similar candidate command, the voice recognition engine 181 determines a matching/similar command to perform voice recognition, and the first controller 180 displays the result on the display unit 130 will be displayed in

As a result of the determination in step S840, if there is no matching/similar candidate command, the voice recognition engine 181 may determine whether to receive and register a candidate command of an application different from the application in step S830 (S842). Here, the first control unit 180 may receive and register candidate commands of other applications according to the user's selection or input settings, and may sequentially receive and register candidate commands of a plurality of applications according to a preset order. do. Also, in consideration of the capacity of the first storage unit 170 of the display apparatus 100, previously registered candidate commands may be selectively deleted.

If there is no matching/similar candidate command among the candidate commands registered in step S842, steps S842 and 841 are sequentially repeated to perform voice recognition.

On the other hand, if it is decided not to receive and register a candidate command of another application in step S842, the voice recognition engine 181 stops the voice recognition, and the first controller 180 determines that the voice recognition fails as a result of the display unit 130 ) can be displayed.

As described above, according to an embodiment of the present invention, since the main device detects voice input from various devices capable of receiving the user's utterance, it is possible to increase the utilization of various voice acquisition devices and to provide linked services using the same.

In addition, since candidate commands for voice recognition are received from a plurality of applications and used, it is possible to dynamically register/delete candidate commands for voice recognition, thereby preventing unnecessary increases in candidate commands provided by the main device, It is possible to flexibly cope with problems such as processing speed delay or reduction in recognition rate, and the efficiency of the entire voice recognition system can be increased.

In addition, the user can easily recognize not only voice recognition results, candidate commands, and application information, but also information on various voice acquisition equipment and candidate command providing equipment used for voice recognition, thereby further enhancing user convenience.

As mentioned above, although the present invention has been described in detail through preferred embodiments, the present invention is not limited thereto and may be practiced in various ways within the scope of the claims.

100: main device, display device 110: image receiver
120: image processing unit 130: display unit
140: first voice acquisition unit 150: first voice conversion unit
160: first communication unit 170: first storage unit
171: command table 172, 173: application
180: first control unit 161: voice recognition engine
201, 202: voice acquisition device 200: mobile phone
240: second voice acquisition unit 250: second voice conversion unit
260: second communication unit 270: second storage unit
280: second control unit 301, 302, 303: external device
300: remote control 330: user input unit
340: third voice acquisition unit 350: second voice conversion unit
360: third communication unit 370: third storage unit
371: application 380: third control unit
400: air conditioner 460: 4th communication department
470: fourth storage 472: application
180: fourth control unit

Claims (20)

In the display device,
display unit;
voice input receiving unit;
a communication unit configured to communicate with at least one external device; and
at least one processor;
The at least one processor,
When a trigger signal by a user non-voice input input to any one external device having an external voice input receiving unit is received through the communication unit,
Activating a function corresponding to the user voice input received through the external voice input receiving unit of the external device, and deactivating the function corresponding to the user voice input received through the voice input receiving unit after the trigger signal is received,
When a user voice input including a trigger word is received through the voice input receiving unit,
While the function corresponding to the user voice input including the trigger word received through the external voice input receiving unit of the external device is deactivated, corresponding to the user voice input including the trigger word received through the voice input receiving unit enable the function,
A display device that controls to execute a function corresponding to the text acquired through the activated function. According to claim 1,
The external device includes a remote control provided with an external communication unit for communicating with other devices, the display device. According to claim 1,
the at least one processor,
A display apparatus for displaying a UI indicating voice recognition on the display unit based on reception of any one of the trigger signal or the user's voice input including the trigger word. According to claim 1,
The trigger signal by the user's non-voice input includes an input to a predetermined button provided in the external device. According to claim 1,
The voice input receiving unit includes a built-in microphone provided in the display device, and the external voice input receiving unit includes an external microphone provided in the external device,
The at least one processor,
Based on the reception of the trigger signal, control to deactivate the built-in microphone,
Based on the reception of the user's voice input including the trigger word, the display device controls to activate the built-in microphone so that the voice input through the external microphone is not received. According to claim 1,
The at least one processor,
A display apparatus for controlling to execute a function corresponding to the text by a first application related to the function corresponding to the acquired text among a plurality of applications. 7. The method of claim 6,
The at least one processor,
A display device that controls the function corresponding to the text to be executed based on the first application being executed by the display device. 7. The method of claim 6,
The at least one processor,
Based on the execution of the first application by the external device, the display device to control the function corresponding to the text to be executed. 7. The method of claim 6,
The first application is provided on the display device,
The external device includes at least one second application,
The at least one processor,
Receiving the information of the second application from the external device through the communication unit,
A display apparatus for controlling to execute a function corresponding to the text based on the received information. According to claim 1,
A plurality of external devices are provided,
The at least one processor,
A display apparatus for controlling to execute a function corresponding to the acquired text in any one of the plurality of external devices. A method for controlling a display device, comprising:
When a trigger signal by a user non-voice input input to any one external device having an external voice input receiving unit is received through the communication unit, a function corresponding to the user voice input received through the external voice input receiving unit of the external device is provided. activating and deactivating a function corresponding to the user's voice input received through the voice input receiving unit after the trigger signal is received;
When a user voice input including a trigger word is received through the voice input receiving unit, while a function corresponding to the user voice input including the trigger word received through the external voice input receiving unit of the external device is deactivated, the voice activating a function corresponding to the user's voice input including the trigger word received through the input receiving unit; and
and controlling a function corresponding to the text obtained through the activated function to be executed. 12. The method of claim 11,
The method of claim 1, wherein the external device includes a remote control provided with an external communication unit for communicating with other devices. 12. The method of claim 11,
and displaying a UI indicating voice recognition based on reception of any one of the trigger signal or the user voice input including the trigger word. 12. The method of claim 11,
The trigger signal by the user non-voice input includes an input to a predetermined button provided in the external device. 12. The method of claim 11,
The voice input receiving unit includes a built-in microphone provided in the display device, and the external voice input receiving unit includes an external microphone provided in the external device,
deactivating the built-in microphone based on the reception of the trigger signal; and
Based on the reception of the user's voice input including the trigger word, the method further comprising the operation of activating the built-in microphone so that the voice input through the external microphone is not received. 12. The method of claim 11,
The operation of controlling the function corresponding to the text to be executed is,
A method of controlling to execute a function corresponding to the text by a first application related to the function corresponding to the acquired text among a plurality of applications. 17. The method of claim 16,
The operation of controlling the function corresponding to the text to be executed is,
Controlling a function corresponding to the text to be executed based on the first application being executed by the display device. 17. The method of claim 16,
The operation of controlling the function corresponding to the text to be executed is,
A method of controlling to execute a function corresponding to the text based on the execution of the first application by the external device. 17. The method of claim 16,
The first application is provided on the display device,
The external device includes at least one second application,
Further comprising the operation of receiving the information of the second application from the external device,
Controlling a function corresponding to the text to be executed based on the received information. 12. The method of claim 11,
A plurality of external devices are provided,
The operation of controlling the function corresponding to the text to be executed includes controlling the function corresponding to the acquired text to be executed in any one of the plurality of external devices.

Images