KR102340231B1 - Multimedia device and method for controlling the same - Google Patents

Abstract

A multimedia device and a method for controlling the same are disclosed.
A multimedia device according to an embodiment of the present invention includes a voice signal receiving unit for receiving a voice signal, a display unit for displaying an execution screen of a specific application, and a control unit, wherein the execution screen of the specific application is a function of the specific application a plurality of menu items corresponding to , and when a preset condition is satisfied, the controller includes location information of each of the plurality of menu items on the execution screen of the specific application and utterances corresponding to each of the plurality of menu items. When information is requested to the specific application, the voice signal received through the voice signal receiver is converted into text, and specific utterance information corresponding to the converted text exists among the requested utterance information, the plurality of menus A signal for selecting a menu item corresponding to the specific utterance information from among the items is generated.

Description

Multimedia device and its control method

The present invention relates to a multimedia device capable of operating a function by voice recognition and a method for controlling the same.

Multimedia devices such as smart TVs, mobiles, PCs, and tablet PCs are gradually diversifying their functions, and along with the diversification of functions, the user convenience aspect of how convenient to operate these functions is becoming increasingly important. .

Recently, a voice recognition control method for controlling functions of these devices from a distance has been introduced. However, this voice recognition control method has limitations in controlling the device function in detail, and as a premise for enabling control by voice recognition, the process of recognizing text displayed on the screen of the device and extracting the coordinates of the recognized text There is a problem in that a lot of resources and time are consumed.

SUMMARY OF THE INVENTION The present invention aims to solve the above and other problems. Another object is to provide a multimedia device capable of controlling a function corresponding to the image through voice recognition by proposing a method for recognizing an object included in an image as well as text output on the screen of the multimedia device, and a method for controlling the same aim to do

Another object of the present invention is to provide a multimedia device capable of significantly reducing resources and time consumed for voice recognition control by receiving information about a menu item output on a screen of the multimedia device from an application, and a method for controlling the same.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

According to one aspect of the present invention to achieve the above or other object, a multimedia device includes a voice signal receiving unit for receiving a voice signal, a display unit for displaying an execution screen of a specific application, and a control unit, the specific application the execution screen includes a plurality of menu items corresponding to the function of the specific application, and when a preset condition is satisfied, the controller includes location information of each of the plurality of menu items on the execution screen of the specific application and the Requests speech information corresponding to each of a plurality of menu items to the specific application, converts a speech signal received through the speech signal receiver into text, and specific speech information corresponding to the converted text among the requested speech information When is present, a signal for selecting a menu item corresponding to the specific utterance information from among the plurality of menu items is generated.

In addition, according to one aspect of the present invention to achieve the above or other object, a method for controlling a multimedia device includes displaying an execution screen of a specific application, and when a preset condition is satisfied, on the execution screen of the specific application requesting the specific application for location information of each of a plurality of menu items, and executing the text included in the execution screen of the specific application and the specific application based on the requested location information of each of the plurality of menu items Recognizing at least one of an object in an image included in a screen; and storing the recognized text and object corresponding to each menu item based on location information of each of the plurality of requested menu items; converting a voice signal received through a signal receiving unit into text; and when a specific text or a specific object corresponding to the converted text exists among the recognized texts and objects, the specific text or and generating an event for selecting a menu item corresponding to a specific object.

In addition, according to an aspect of the present invention to achieve the above or other object, a method for controlling a multimedia device includes displaying an execution screen of a specific application, and recognizing an object in an image included in the execution screen of the specific application and extracting the location information of the recognized object from the execution screen of the specific application, and storing the extracted location information in correspondence with the recognized object; An event of converting a voice signal into text, and selecting an area of location information corresponding to the specific object from the execution screen of the specific application when a specific object corresponding to the converted text exists among the recognized objects It includes the step of generating

The effects of the multimedia device and the method for controlling the same according to the present invention will be described as follows.

According to at least one of the embodiments of the present invention, a method for recognizing an object included in an image as well as text output on a screen of a multimedia device is proposed to control a function corresponding to the image through voice recognition. It is possible to provide a multimedia device and a method for controlling the same.

In addition, according to at least one of the embodiments of the present invention, a multimedia device capable of significantly reducing resources and time consumed for voice recognition control by receiving information about a menu item output on the screen of the multimedia device from an application, and a multimedia device and the same A control method may be provided.

Further scope of applicability of the present invention will become apparent from the following detailed description. However, it should be understood that the detailed description and specific embodiments such as preferred embodiments of the present invention are given by way of example only, since various changes and modifications within the spirit and scope of the present invention may be clearly understood by those skilled in the art.

1 is a block diagram illustrating a mobile terminal as a multimedia device according to an embodiment of the present invention.
2 is a block diagram illustrating a display device as a multimedia device according to an embodiment of the present invention.
3 is a block diagram illustrating a detailed configuration of the control unit of FIG. 2 according to an embodiment of the present invention.
4 is a diagram illustrating an input means connected to the display device of FIG. 2 according to an embodiment of the present invention.
5 is a block diagram of a multimedia device according to an embodiment of the present invention.
6 is a flowchart illustrating an example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.
7 to 10 are diagrams for explaining an example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.
11 is a flowchart illustrating another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.
12 to 15 are diagrams for explaining another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.
16 is a flowchart illustrating another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.
17 is a diagram for explaining another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The multimedia device described in this specification includes a mobile terminal (eg, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, personal digital assistants (PDA), a portable multimedia player (PMP), Navigation, slate PC, tablet PC, ultrabook, wearable device, for example, watch-type terminal (smartwatch), glass-type terminal (smart glass), HMD (head) mounted display))), display devices, TVs, digital TVs, smart TVs, IPTVs, HBBTVs, personal computers, and the like.

1 is a block diagram illustrating a mobile terminal as a multimedia device according to an embodiment of the present invention.

The mobile terminal 100 includes a wireless communication unit 110 , an input unit 120 , a sensing unit 140 , an output unit 150 , an interface unit 160 , a memory 170 , a control unit 180 , and a power supply unit 190 . and the like. The components shown in FIG. 1 are not essential for implementing the mobile terminal, so the mobile terminal described in this specification may have more or fewer components than those listed above.

More specifically, among the components, the wireless communication unit 110 is between the mobile terminal 100 and the wireless communication system, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 and an external server. It may include one or more modules that enable wireless communication between them. In addition, the wireless communication unit 110 may include one or more modules for connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast reception module 111 , a mobile communication module 112 , a wireless Internet module 113 , a short-range communication module 114 , and a location information module 115 . .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 or an audio input unit for inputting an audio signal, and a user input unit 123 for receiving information from a user, for example, , a touch key, a push key, etc.). The voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity. Sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor: infrared sensor), fingerprint recognition sensor (finger scan sensor), ultrasonic sensor , optical sensors (eg, cameras (see 121)), microphones (see 122), battery gauges, environmental sensors (eg, barometers, hygrometers, thermometers, radiation detection sensors, It may include at least one of a thermal sensor, a gas sensor, etc.) and a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification may combine and utilize information sensed by at least two or more of these sensors.

The output unit 150 is for generating an output related to visual, auditory or tactile sense, and includes at least one of a display unit 151 , a sound output unit 152 , a haptip module 153 , and an optical output unit 154 . can do. The display unit 151 may implement a touch screen by forming a layer structure with the touch sensor or being integrally formed. Such a touch screen may function as the user input unit 123 providing an input interface between the mobile terminal 100 and the user, and may provide an output interface between the mobile terminal 100 and the user.

The interface unit 160 serves as a passage with various types of external devices connected to the mobile terminal 100 . The interface unit 160 connects a device equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In response to the connection of the external device to the interface unit 160 , the mobile terminal 100 may perform appropriate control related to the connected external device.

In addition, the memory 170 stores data supporting various functions of the mobile terminal 100 . The memory 170 may store a plurality of application programs (or applications) driven in the mobile terminal 100 , data for operation of the mobile terminal 100 , and commands. At least some of these application programs may be downloaded from an external server through wireless communication. In addition, at least some of these application programs may exist on the mobile terminal 100 from the time of shipment for basic functions (eg, incoming calls, outgoing functions, message reception, and outgoing functions) of the mobile terminal 100 . Meanwhile, the application program may be stored in the memory 170 , installed on the mobile terminal 100 , and driven to perform an operation (or function) of the mobile terminal by the controller 180 .

In addition to the operation related to the application program, the controller 180 generally controls the overall operation of the mobile terminal 100 . The controller 180 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170 .

In addition, the controller 180 may control at least some of the components discussed with reference to FIG. 1 in order to drive an application program stored in the memory 170 . Furthermore, in order to drive the application program, the controller 180 may operate at least two or more of the components included in the mobile terminal 100 in combination with each other.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power to each component included in the mobile terminal 100 . The power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the respective components may operate in cooperation with each other in order to implement an operation, control, or control method of a mobile terminal according to various embodiments to be described below. Also, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 170 .

Hereinafter, before looking at various embodiments implemented through the mobile terminal 100 as described above, the above-listed components will be described in more detail with reference to FIG. 1 .

First, referring to the wireless communication unit 110 , the broadcast reception module 111 of the wireless communication unit 110 receives a broadcast signal and/or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more of the broadcast reception modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The broadcast management server may mean a server that generates and transmits a broadcast signal and/or broadcast-related information or a server that receives a previously generated broadcast signal and/or broadcast-related information and transmits the received broadcast signal and/or broadcast-related information to the terminal. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, as well as a TV broadcast signal or a broadcast signal in which a data broadcast signal is combined with a radio broadcast signal.

The broadcast signal may be encoded according to at least one of technical standards (or a broadcast method, for example, ISO, IEC, DVB, ATSC, etc.) for transmitting and receiving digital broadcast signals, and the broadcast reception module 111 is configured to The digital broadcast signal can be received using a method suitable for technical standards set by technical standards.

The broadcast related information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112 .

The broadcast-related information may exist in various forms, for example, an Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB) or an Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld (DVB-H). A broadcast signal and/or broadcast related information received through the broadcast reception module 111 may be stored in the memory 160 .

Mobile communication module 112, the technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV -Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (DO), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced, etc.) transmits and receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to.

The wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be built-in or external to the mobile terminal 100 . The wireless Internet module 113 is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

As wireless Internet technology, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), etc., and the wireless Internet module ( 113) transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above.

From the point of view that wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is made through a mobile communication network, the wireless Internet module 113 performs wireless Internet access through the mobile communication network. ) may be understood as a type of the mobile communication module 112 .

Short-range communication module 114 is for short-range communication, Bluetooth (Bluetooth ^TM , RFID (Radio Frequency Identification), infrared communication (Infrared Data Association; IrDA), UWB (Ultra Wideband), ZigBee, NFC ( Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and at least one of Wireless USB (Wireless Universal Serial Bus) technologies may be used to support short-distance communication. is between the mobile terminal 100 and the wireless communication system, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 and the other mobile terminal 100, Alternatively, wireless communication between networks in which external servers are located may be supported, and the local area network may be wireless personal area networks.

Here, the other mobile terminal 100 is a wearable device capable of exchanging (or interworking) data with the mobile terminal 100 according to the present invention, for example, a smart watch, smart glasses. (smart glass) or HMD (head mounted display)). The short-range communication module 114 may detect (or recognize) a wearable device capable of communicating with the mobile terminal 100 in the vicinity of the mobile terminal 100 . Furthermore, when the detected wearable device is a device authenticated to communicate with the mobile terminal 100 according to the present invention, the controller 180 transmits at least a portion of data processed by the mobile terminal 100 to the short-range communication module ( 114) to transmit to the wearable device. Accordingly, the user of the wearable device may use data processed by the mobile terminal 100 through the wearable device. For example, according to this, when a call is received in the mobile terminal 100, the user performs a phone call through the wearable device, or when a message is received in the mobile terminal 100, the user receives the received call through the wearable device. It is possible to check the message.

The location information module 115 is a module for obtaining a location (or current location) of a mobile terminal, and a representative example thereof includes a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, if the mobile terminal utilizes a GPS module, it can acquire the location of the mobile terminal by using a signal transmitted from a GPS satellite. As another example, if the mobile terminal utilizes the Wi-Fi module, the location of the mobile terminal may be obtained based on information of the Wi-Fi module and a wireless access point (AP) that transmits or receives a wireless signal. If necessary, the location information module 115 may perform any function of the other modules of the wireless communication unit 110 to obtain data on the location of the mobile terminal as a substitute or additionally. The location information module 115 is a module used to obtain the location (or current location) of the mobile terminal, and is not limited to a module that directly calculates or obtains the location of the mobile terminal.

Next, the input unit 120 is for input of image information (or signal), audio information (or signal), data, or information input from a user. For input of image information, the mobile terminal 100 has one Alternatively, a plurality of cameras 121 may be provided. The camera 121 processes an image frame such as a still image or a moving image obtained by an image sensor in a video call mode or a shooting mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170 . On the other hand, the plurality of cameras 121 provided in the mobile terminal 100 may be arranged to form a matrix structure, and through the cameras 121 forming the matrix structure as described above, various angles or focal points are applied to the mobile terminal 100 . A plurality of image information with can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes an external sound signal as electrical voice data. The processed voice data may be utilized in various ways according to a function (or an application program being executed) being performed by the mobile terminal 100 . Meanwhile, various noise removal algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone 122 .

The user input unit 123 is for receiving information from the user, and when information is input through the user input unit 123 , the controller 180 may control the operation of the mobile terminal 100 to correspond to the input information. . Such a user input unit 123 is a mechanical input means (or a mechanical key, for example, a button located on the front, rear or side of the mobile terminal 100, a dome switch (dome switch), a jog wheel, jog switch, etc.) and a touch input means. As an example, the touch input means consists of a virtual key, a soft key, or a visual key displayed on the touch screen through software processing, or a part other than the touch screen. On the other hand, the virtual key or the visual key may be displayed on the touch screen while having various shapes, and for example, graphic, text ), an icon, a video, or a combination thereof.

On the other hand, the sensing unit 140 senses at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information, and generates a sensing signal corresponding thereto. The controller 180 may control the driving or operation of the mobile terminal 100 or perform data processing, function, or operation related to an application program installed in the mobile terminal 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor 141 may be disposed in an inner region of the mobile terminal covered by the touch screen described above or near the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the proximity sensor 141 may be configured to detect the proximity of an object having conductivity by a change in an electric field according to the proximity of the object. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of description, the act of approaching an object on the touch screen without contacting it so that the object is recognized that it is located on the touch screen is called “proximity touch”, and the touch The act of actually touching an object on the screen is called "contact touch". The position where the object is touched in proximity on the touch screen means a position where the object is perpendicular to the touch screen when the object is touched in proximity. The proximity sensor 141 may detect a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.) have. Meanwhile, the controller 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further, provides visual information corresponding to the processed data. It can be printed on the touch screen. Furthermore, the controller 180 may control the mobile terminal 100 to process different operations or data (or information) according to whether a touch to the same point on the touch screen is a proximity touch or a contact touch. .

The touch sensor receives a touch (or touch input) applied to the touch screen (or the display unit 151) using at least one of various touch methods such as a resistive film method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method. detect

As an example, the touch sensor may be configured to convert a change in pressure applied to a specific part of the touch screen or a change in capacitance occurring in a specific part of the touch screen into an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, an electrostatic capacitance at the time of touch, etc. in which a touch object applying a touch on the touch screen is touched on the touch sensor. Here, the touch object is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen, a stylus pen, or a pointer.

As such, when there is a touch input to the touch sensor, a signal(s) corresponding thereto is sent to the touch controller. The touch controller processes the signal(s) and then transmits corresponding data to the controller 180 . Accordingly, the controller 180 can know which area of the display unit 151 has been touched, and the like. Here, the touch controller may be a component separate from the controller 180 , or may be the controller 180 itself.

Meanwhile, the controller 180 may perform different controls or may perform the same control according to the type of the touch object that touches the touch screen (or a touch key provided in addition to the touch screen). Whether to perform different control or the same control according to the type of the touch object may be determined according to the current operating state of the mobile terminal 100 or a running application program.

On the other hand, the above-described touch sensor and proximity sensor independently or in combination, a short (or tap) touch on the touch screen (short touch), long touch (long touch), multi touch (multi touch), drag touch (drag touch) ), flick touch, pinch-in touch, pinch-out touch, swype touch, hovering touch, etc. It can sense touch.

The ultrasonic sensor may recognize position information of a sensing target by using ultrasonic waves. Meanwhile, the controller 180 may calculate the position of the wave source through information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source may be calculated using the property that light is much faster than ultrasonic waves, that is, the time at which light arrives at the optical sensor is much faster than the time at which ultrasonic waves reach the ultrasonic sensor. More specifically, the position of the wave source may be calculated using a time difference from the time that the ultrasonic wave arrives using light as a reference signal.

On the other hand, the camera 121 as seen in the configuration of the input unit 120 includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to detect a touch of a sensing target for a 3D stereoscopic image. The photo sensor may be stacked on the display device, and the photo sensor is configured to scan the motion of the sensing target close to the touch screen. More specifically, the photo sensor mounts photo diodes and transistors (TRs) in rows/columns and scans the contents placed on the photo sensors using electrical signals that change according to the amount of light applied to the photo diodes. That is, the photo sensor calculates the coordinates of the sensing target according to the amount of change in light, and through this, location information of the sensing target can be obtained.

The display unit 151 displays (outputs) information processed by the mobile terminal 100 . For example, the display unit 151 may display execution screen information of an application program driven in the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information. .

Also, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

A three-dimensional display method such as a stereoscopic method (glasses method), an auto stereoscopic method (glassesless method), or a projection method (holographic method) may be applied to the stereoscopic display unit.

In general, a 3D stereoscopic image consists of a left image (image for left eye) and a right image (image for right eye). Depending on the method in which the left and right images are merged into a 3D stereoscopic image, a top-down method in which the left and right images are placed up and down in one frame, and the left and right images in one frame L-to-R (left-to-right, side by side) method, which arranges the pieces of the left and right images in tiles Alternatively, it is divided into an interlaced method in which rows are alternately arranged, and a time sequential (frame by frame) method in which a left image and a right image are displayed alternately by time.

In addition, the 3D thumbnail image may be generated as a single image by generating a left image thumbnail and a right image thumbnail from the left image and the right image of the original image frame, respectively, and combining them. In general, a thumbnail means a reduced image or a reduced still image. The generated left image thumbnail and right image thumbnail may be displayed with a left and right distance difference on the screen by a depth corresponding to the parallax between the left image and the right image, thereby representing a three-dimensional sense of space.

The left image and the right image necessary for realizing the 3D stereoscopic image may be displayed on the stereoscopic display unit by the stereoscopic processing unit. The stereoscopic processing unit receives a 3D image (a reference view image and an extended view image) and sets a left image and a right image therefrom, or receives a 2D image and converts it into a left image and a right image.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output unit 152 also outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, etc.) performed in the mobile terminal 100 . The sound output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A representative example of the tactile effect generated by the haptic module 153 may be vibration. The intensity and pattern of vibration generated by the haptic module 153 may be controlled by a user's selection or setting of the controller. For example, the haptic module 153 may synthesize and output different vibrations or output them sequentially.

In addition to vibration, the haptic module 153 is configured to respond to stimuli such as a pin arrangement that moves vertically with respect to the contact skin surface, a jet force or suction force of air through a nozzle or an inlet, rubbing against the skin surface, contact of an electrode, electrostatic force, etc. Various tactile effects can be generated, such as the effect of heat absorption and the effect of reproducing a feeling of coolness and warmth using an element capable of absorbing heat or generating heat.

The haptic module 153 may not only deliver a tactile effect through direct contact, but may also be implemented so that the user can feel the tactile effect through a muscle sense such as a finger or arm. Two or more haptic modules 153 may be provided according to the configuration of the mobile terminal 100 .

The light output unit 154 outputs a signal for notifying the occurrence of an event by using the light of the light source of the mobile terminal 100 . Examples of the event occurring in the mobile terminal 100 may be message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output from the optical output unit 154 is implemented as the mobile terminal emits light of a single color or a plurality of colors toward the front or rear. The signal output may be terminated when the mobile terminal detects the user's event confirmation.

The interface unit 160 serves as a passage with all external devices connected to the mobile terminal 100 . The interface unit 160 receives data from an external device, receives power and transmits it to each component inside the mobile terminal 100 , or transmits data inside the mobile terminal 100 to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module (port), an audio I/O (Input/Output) port, a video I/O (Input/Output) port, an earphone port, etc. may be included in the interface unit 160 .

On the other hand, the identification module is a chip storing various types of information for authenticating the use authority of the mobile terminal 100, a user identification module (UIM), a subscriber identity module (subscriber identity module; SIM), universal user authentication It may include a universal subscriber identity module (USIM) and the like. A device equipped with an identification module (hereinafter, 'identification device') may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal 100 through the interface unit 160 .

In addition, when the mobile terminal 100 is connected to an external cradle, the interface unit 160 becomes a passage through which power from the cradle is supplied to the mobile terminal 100, or is input from the cradle by a user. Various command signals may be a path through which the mobile terminal 100 is transmitted. Various command signals or the power input from the cradle may be operated as signals for recognizing that the mobile terminal 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 , and may temporarily store input/output data (eg, a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibration and sound of various patterns output when a touch input on the touch screen is input.

Memory 170 is a flash memory type (flash memory type), hard disk type (hard disk type), SSD type (Solid State Disk type), SDD type (Silicon Disk Drive type), multimedia card micro type (multimedia card micro type) ), card-type memory (such as SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EEPROM) -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The mobile terminal 100 may be operated in relation to a web storage that performs a storage function of the memory 170 on the Internet.

Meanwhile, as described above, the controller 180 controls the operation related to the application program and the general operation of the mobile terminal 100 in general. For example, if the state of the mobile terminal satisfies a set condition, the controller 180 may execute or release a lock state restricting input of a user's control command to applications.

In addition, the controller 180 performs control and processing related to voice calls, data communication, video calls, etc., or performs pattern recognition processing capable of recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively. can Furthermore, in order to implement various embodiments described below on the mobile terminal 100 according to the present invention, the controller 180 may control one or a combination of any one or a plurality of the components described above.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 to supply power required for operation of each component. The power supply unit 190 includes a battery, and the battery may be a built-in battery configured to be rechargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of the interface 160 to which an external charger that supplies power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 uses one or more of an inductive coupling method based on a magnetic induction phenomenon or a resonance coupling method based on an electromagnetic resonance phenomenon from an external wireless power transmitter. power can be transmitted.

Meanwhile, various embodiments below may be implemented in a computer-readable recording medium using, for example, software, hardware, or a combination thereof.

2 is a block diagram illustrating a display device as a multimedia device according to an embodiment of the present invention.

Referring to FIG. 2 , the display device 200 includes a broadcast receiving unit 205 , an external device interface unit 235 , a storage unit 240 , a user input interface unit 250 , a control unit 270 , and a display unit 280 . ), an audio output unit 285 , a power supply unit 290 , and a photographing unit (not shown). Here, the broadcast receiving unit 205 may include at least one tuner 210 , a demodulator 220 , and a network interface unit 230 . However, in some cases, the broadcast receiving unit 205 may include the tuner 210 and the demodulator 220 but not the network interface unit 230 , and vice versa. In addition, although not shown, the broadcast receiving unit 205 includes a multiplexer to provide a signal demodulated by the demodulator 220 through the tuner 210 and a signal received through the network interface unit 230 . can also be multiplexed. In addition, although not shown, the broadcast receiving unit 225 includes a demultiplexer to demultiplex the multiplexed signal or to demultiplex the demodulated signal or the signal that has passed through the network interface unit 230. can

The tuner 210 receives an RF broadcast signal by tuning a channel selected by a user or all channels previously stored among radio frequency (RF) broadcast signals received through an antenna. Also, the tuner 210 converts the received RF broadcast signal into an intermediate frequency (IF) signal or a baseband signal.

For example, if the received RF broadcast signal is a digital broadcast signal, it is converted into a digital IF signal (DIF), and if it is an analog broadcast signal, it is converted into an analog baseband image or audio signal (CVBS/SIF). That is, the tuner 210 may process both a digital broadcast signal and an analog broadcast signal. An analog baseband video or audio signal (CVBS/SIF) output from the tuner 210 may be directly input to the controller 270 .

Also, the tuner 210 may receive an RF broadcast signal of a single carrier or a multiple carrier. Meanwhile, the tuner 210 sequentially tunes and receives the RF broadcast signals of all broadcast channels stored through the channel storage function among the RF broadcast signals received through the antenna, and receives them as an intermediate frequency signal or a baseband signal (DIF: Digital Intermediate). frequency or baseband signal).

The demodulator 220 may receive and demodulate the digital IF signal DIF converted by the tuner 210 , and may perform channel decoding and the like. To this end, the demodulator 220 includes a trellis decoder, a de-interleaver, a Reed-Solomon decoder, or the like, or a convolution decoder, a deinterleaver and a read. - A Solomon decoder, etc. may be provided.

The demodulator 220 may output a stream signal TS after demodulation and channel decoding are performed. In this case, the stream signal may be a signal obtained by multiplexing an image signal, an audio signal, or a data signal. For example, the stream signal may be an MPEG-2 Transport Stream (TS) in which an MPEG-2 standard video signal, a Dolby AC-3 standard audio signal, etc. are multiplexed.

The stream signal output from the demodulator 220 may be input to the controller 270 . The controller 270 may control demultiplexing, image/audio signal processing, and the like, and may control output of an image through the display unit 280 and audio through the audio output unit 285 .

The external device interface unit 235 provides an interface environment between the display device 200 and various external devices. To this end, the external device interface unit 235 may include an A/V input/output unit (not shown) or a wireless communication unit (not shown).

External device interface unit 235, DVD (Digital Versatile Disk), Blu-ray (Blu-ray), game device, camera, camcorder (Camcorder), computer (laptop), tablet PC, smart phone, Bluetooth device (Bluetooth) device) or an external device such as a cloud, and the like, may be connected via wired/wireless. The external device interface unit 235 transmits a signal including data such as an image, video, and audio input through a connected external device to the controller 470 of the digital device. The controller 270 may control the processed image, video, audio, etc. to output a data signal to a connected external device. To this end, the external device interface unit 435 may further include an A/V input/output unit (not shown) or a wireless communication unit (not shown).

The A/V input/output unit includes a USB terminal, a CVBS (Composite Video Banking Sync) terminal, a component terminal, an S-video terminal (analog), It may include a Digital Visual Interface (DVI) terminal, a High Definition Multimedia Interface (HDMI) terminal, an RGB terminal, a D-SUB terminal, and the like.

The wireless communication unit may perform short-range wireless communication with another external device. The display device 200 includes, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Digital Living Network Alliance (DLNA). It may be network-connected with other digital devices according to a communication protocol such as .

In addition, the external device interface unit 235 may be connected to the set-top box (STB) through at least one of the above-described various terminals to perform input/output operations with the set-top-box (STB).

Meanwhile, the external device interface unit 235 may receive an application or an application list in an adjacent external device and transmit it to the control unit 270 or the storage unit 240 .

The network interface unit 230 provides an interface for connecting the display device 200 to a wired/wireless network including an Internet network. The network interface unit 230 may include, for example, an Ethernet terminal for connection with a wired network, and for connection with a wireless network, for example, a WLAN (Wireless LAN) (Wi- Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) communication standards, etc. can be used.

The network interface unit 230 may transmit or receive data with another user or another external device through the connected network or another network linked to the connected network. In particular, some content data stored in the display device 200 may be transmitted to a selected user or a selected external device among other users or other external devices previously registered in the display device 200 .

Meanwhile, the network interface unit 230 may access a predetermined web page through the connected network or another network linked to the connected network. That is, by accessing a predetermined web page through a network, it is possible to transmit or receive data with a corresponding server. In addition, content or data provided by a content provider or network operator may be received. That is, it is possible to receive content such as a movie, advertisement, game, VOD, and broadcast signal and information related thereto from a content provider or a network provider through a network. In addition, it is possible to receive firmware update information and an update file provided by a network operator. It may also transmit data to the Internet or to a content provider or network operator.

Also, the network interface unit 230 may select and receive a desired application from among applications open through a network.

The storage unit 240 may store a program for processing and controlling each signal in the control unit 270 , or may store a signal-processed image, audio, or data signal.

Also, the storage unit 240 may perform a function for temporarily storing an image, audio, or data signal input from the external device interface unit 235 or the network interface unit 230 . The storage unit 240 may store information about a predetermined broadcast channel through a channel storage function.

The storage unit 240 may store an application or an application list input from the external device interface unit 235 or the network interface unit 230 .

Also, the storage unit 240 may store various platforms to be described later.

The storage unit 240 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD). memory, etc.), a RAM (RAM), and a ROM (EEPROM, etc.) may include at least one type of storage medium. The digital device 400 may reproduce content files (movie files, still image files, music files, document files, application files, etc.) stored in the storage unit 240 and provide them to the user.

2 illustrates an embodiment in which the storage unit 440 is provided separately from the control unit 270, the present invention is not limited thereto. In other words, the storage unit 240 may be included in the control unit 270 .

The user input interface unit 250 transmits a signal input by the user to the control unit 270 or transmits a signal of the control unit 270 to the user.

For example, the user input interface unit 250 controls power on/off, channel selection, screen setting, etc. from the remote control device 400 according to various communication methods such as an RF communication method and an infrared (IR) communication method. The signal may be received and processed, or a control signal of the controller 270 may be transmitted to the remote control device 400 .

In addition, the user input interface unit 250 may transmit a control signal input from a local key (not shown) such as a power key, a channel key, a volume key, and a setting value to the control unit 270 .

The user input interface unit 250 transmits a control signal input from a sensing unit (not shown) that senses a user's gesture to the control unit 270 or transmits a signal of the control unit 270 to the sensing unit (not shown) can be transmitted. Here, the sensing unit (not shown) may include a touch sensor, a voice sensor, a position sensor, a motion sensor, and the like.

The controller 270 demultiplexes a stream input through the tuner 210 , the demodulator 220 , or the external device interface 235 or processes the demultiplexed signals to generate a signal for video or audio output. and output.

The image signal processed by the control unit 270 may be input to the display unit 280 and displayed as an image corresponding to the image signal. Also, the image signal processed by the controller 270 may be input to an external output device through the external device interface unit 235 .

The audio signal processed by the controller 270 may be audio output to the audio output unit 285 . Also, the audio signal processed by the controller 270 may be input to an external output device through the external device interface 235 .

Although not shown in FIG. 2 , the controller 270 may include a demultiplexer, an image processor, and the like.

The controller 270 may control the overall operation of the display device 200 . For example, the controller 270 may control the tuner 210 to tune an RF broadcast corresponding to a channel selected by the user or a pre-stored channel.

The control unit 270 may control the display device 200 according to a user command input through the user input interface unit 250 or an internal program. In particular, by accessing a network, a user can download a desired application or an application list into the display device 200 .

For example, the controller 270 controls the tuner 210 to input a signal of a selected channel according to a predetermined channel selection command received through the user input interface 250 . And it processes the video, audio, or data signal of the selected channel. The control unit 270 enables the channel information selected by the user to be output through the display unit 280 or the audio output unit 285 together with the processed image or audio signal.

As another example, the control unit 270 may be configured to receive an external device image playback command received through the user input interface unit 250 from an external device input through the external device interface unit 235, for example, a camera or a camcorder. , an image signal or an audio signal to be output through the display unit 280 or the audio output unit 285 .

Meanwhile, the controller 270 may control the display 280 to display an image. For example, a broadcast image input through the tuner 210 , an external input image input through the external device interface unit 235 , an image input through the network interface unit, or an image stored in the storage unit 240 . , can be controlled to be displayed on the display unit 280 . In this case, the image displayed on the display unit 280 may be a still image or a moving image, and may be a 2D image or a 3D image.

Also, the controller 270 may control to reproduce the content. In this case, the content may be content stored in the display device 200 , received broadcast content, or external input content input from the outside. The content may be at least one of a broadcast image, an external input image, an audio file, a still image, a connected web screen, and a document file.

Meanwhile, when entering the application view item, the controller 270 may control to display an application or an application list downloadable from an internal or external network of the display device 200 .

The controller 270 may control to install and drive an application downloaded from an external network together with various user interfaces. In addition, by the user's selection, it is possible to control the display unit 280 to display an image related to the application being executed.

Meanwhile, although not shown in the drawings, a channel browsing processing unit generating a thumbnail image corresponding to a channel signal or an external input signal may be further provided.

The channel browsing processing unit receives the stream signal TS output from the demodulator 220 or the stream signal output from the external device interface 235, and extracts an image from the input stream signal to generate a thumbnail image. can The generated thumbnail image may be input to the controller 270 as it is or encoded. Also, the generated thumbnail image may be encoded in a stream form and input to the controller 270 . The controller 270 may display a thumbnail list including a plurality of thumbnail images on the display 280 by using the input thumbnail images. Meanwhile, the thumbnail images in the thumbnail list may be updated sequentially or simultaneously. Accordingly, the user can easily grasp the contents of a plurality of broadcast channels.

The display unit 280 converts the image signal, the data signal, the OSD signal processed by the controller 270, or the image signal and the data signal received from the external device interface unit 235 into R, G, and B signals, respectively. generate a drive signal.

The display unit 280 may be a PDP, LCD, OLED, flexible display, or 3D display.

Meanwhile, the display unit 280 may be configured as a touch screen and used as an input device in addition to an output device.

The audio output unit 285 receives a signal processed by the control unit 270 , for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs the audio signal. The audio output unit 285 may be implemented with various types of speakers.

Meanwhile, in order to detect a user's gesture, as described above, a sensing unit (not shown) including at least one of a touch sensor, a voice sensor, a position sensor, and a motion sensor may be further provided in the display device 200 . . The signal sensed by the sensing unit (not shown) may be transmitted to the control unit 270 through the user input interface unit 250 .

Meanwhile, a photographing unit (not shown) for photographing the user may be further provided. Image information captured by the photographing unit (not shown) may be input to the control unit 270 .

The controller 270 may detect a user's gesture by individually or in combination with an image captured by a photographing unit (not shown) or a signal sensed from a sensing unit (not shown).

The power supply 290 supplies corresponding power to the entire display device 200 .

In particular, to supply power to the control unit 270 that can be implemented in the form of a system on chip (SoC), the display unit 280 for displaying an image, and the audio output unit 285 for outputting audio. can

To this end, the power supply unit 290 may include a converter (not shown) that converts AC power into DC power. Meanwhile, for example, when the display unit 280 is implemented as a liquid crystal panel including a plurality of backlight lamps, an inverter capable of Pulse Width Modulation (PWM) operation for luminance variable or dimming driving (inverter) (not shown) may be further provided.

The remote control device 400 transmits a user input to the user input interface unit 250 . To this end, the remote control device 400 may use Bluetooth (Bluetooth), Radio Frequency (RF) communication, infrared (IR) communication, Ultra Wideband (UWB), ZigBee, or the like.

In addition, the remote control device 400 may receive an image, audio, or data signal output from the user input interface unit 250 , and display it on the remote control device 400 or output voice or vibration.

The above-described display device 200 may be a digital broadcast receiver capable of processing a digital broadcast signal of a fixed or mobile ATSC method or a DVB method.

In addition, the display device according to the present invention may omit some of the illustrated components as needed, or may further include a non-illustrated component. Meanwhile, unlike the above, the display device does not include a tuner and a demodulator, and may receive and reproduce content through a network interface unit or an external device interface unit.

3 is a block diagram illustrating a detailed configuration of the control unit of FIG. 2 according to an embodiment of the present invention.

An example of the controller 270 is a demultiplexer 310 , an image processor 320 , an OSD generator 340 , a mixer 350 , and a frame rate converter (FRC) 355 . ), and a formatter 360 . In addition, although not shown, the control unit may further include a voice processing unit and a data processing unit.

The demultiplexer 310 demultiplexes an input stream. For example, the demultiplexer 310 may demultiplex the input MPEG-2 TS video, audio, and data signals. Here, the stream signal input to the demultiplexer 310 may be a stream signal output from a tuner or demodulator or an external device interface unit.

The image processing unit 320 performs image processing of the demultiplexed image signal. To this end, the image processing unit 320 may include an image decoder 325 and a scaler 335 .

The image decoder 325 decodes the demultiplexed image signal, and the scaler 335 scales the resolution of the decoded image signal to be output on the display unit.

The video decoder 325 may support various standards. For example, the video decoder 325 performs the function of an MPEG-2 decoder when the video signal is encoded in the MPEG-2 standard, and when the video signal is encoded in the DMB (Digital Multimedia Broadcasting) method or the H.264 standard. In this case, the function of the H.264 decoder may be performed.

Meanwhile, the image signal decoded by the image processing unit 320 is input to the mixer 350 .

The OSD generator 340 generates OSD data according to a user input or by itself. For example, the OSD generating unit 340 generates data for displaying various data in the form of graphics or text on the screen of the display unit 280 based on the control signal of the user input interface unit. The generated OSD data includes various data such as a user interface screen of a digital device, various menu screens, widgets, icons, and viewing rate information. The OSD generator 340 may generate data for displaying broadcast information based on a caption of a broadcast image or an EPG.

The mixer 350 mixes the OSD data generated by the OSD generator 340 and the image signal processed by the image processor and provides them to the formatter 360 . Because the decoded image signal and the OSD data are mixed, the OSD is overlaid and displayed on the broadcast image or the external input image.

The frame rate converter (FRC) 355 converts a frame rate of an input image. For example, the frame rate converter 355 may convert a frame rate of an input 60Hz image to have a frame rate of, for example, 120Hz or 240Hz according to an output frequency of the display unit. As described above, there may be various methods for converting the frame rate. For example, when the frame rate conversion unit 355 converts the frame rate from 60 Hz to 120 Hz, the same first frame is inserted between the first frame and the second frame, or a second frame predicted from the first frame and the second frame is inserted. It can be converted by inserting 3 frames. As another example, when converting the frame rate from 60 Hz to 240 Hz, the frame rate converter 355 may convert by inserting three more identical frames or predicted frames between existing frames. Meanwhile, when a separate frame conversion is not performed, the frame rate conversion unit 355 may be bypassed.

The formatter 360 changes the output of the input frame rate converter 355 to match the output format of the display unit. For example, the formatter 360 may output R, G, and B data signals, and these R, G, and B data signals are output as a low voltage differential signal (LVDS) or mini-LVDS. can be In addition, when the output of the input frame rate converter 355 is a 3D video signal, the formatter 360 may support the 3D service through the display unit by composing and outputting it in a 3D form to match the output format of the display unit.

Meanwhile, the voice processing unit (not shown) in the control unit may perform voice processing of the demultiplexed voice signal. Such a voice processing unit (not shown) may support processing of various audio formats. For example, even when the audio signal is encoded in a format such as MPEG-2, MPEG-4, AAC, HE-AAC, AC-3, BSAC, it can be processed with a corresponding decoder.

In addition, the voice processing unit (not shown) in the control unit may process a base (Base), a treble (Treble), volume control, and the like.

A data processing unit (not shown) in the control unit may perform data processing of the demultiplexed data signal. For example, the data processing unit may decode the demultiplexed data signal even when it is encoded. Here, the encoded data signal may be EPG information including broadcast information such as start time and end time of a broadcast program aired on each channel.

On the other hand, the above-described digital device is an example according to the present invention, and each component may be integrated, added, or omitted according to the specifications of the actually implemented digital device. That is, if necessary, two or more components may be combined into one component or one component may be subdivided into two or more components. In addition, the function performed in each block is for explaining the embodiment of the present invention, and the specific operation or device does not limit the scope of the present invention.

Meanwhile, the digital device may be an image signal processing device that performs signal processing of an image stored in the device or an input image. As another example of the image signal processing device, the set-top box (STB) in which the display unit 280 and the audio output unit 285 shown in FIG. 2 are excluded, the above-described DVD player, Blu-ray player, game device, computer and the like can be further exemplified.

4 is a diagram illustrating an input means connected to the display device of FIG. 2 according to an embodiment of the present invention.

A front panel (not shown) or a control means (input means) provided on the display device 200 may be used to control the display device 200 .

On the other hand, the control means is a user interface device (UID) capable of wired and wireless communication, mainly implemented for the purpose of controlling the display device 200, a remote control 410, keyboard 430, pointing device 420, A touch-pad or the like is included, but an external input-only control means connected to the display device 200 may also be included. In addition, the control means includes a mobile device such as a smart phone or a tablet PC that controls the display device 200 through mode switching, not for the purpose of controlling the display device 200 . However, in the present specification, a pointing device is described as an example for convenience, but is not limited thereto.

Communication protocols such as Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA), UWB (Ultra Wideband), ZigBee, DLNA (Digital Living Network Alliance), RS, etc. can communicate with the digital device by employing at least one or more as necessary.

The remote control 410 refers to a general input means provided with various key buttons necessary for controlling the display device 200 .

The pointing device 420 is equipped with a gyro sensor, etc. to implement a pointer corresponding to the screen of the display device 200 based on the user's movement, pressure, rotation, etc. ) to transmit a predetermined control command. The pointing device 420 may be named by various names such as a magic remote control and a magic controller.

The keyboard 430, as the display device 200 provides various services such as a web browser, an application, and a social network service (SNS) as an intelligent integrated digital device beyond the conventional broadcasting only, control is possible only with the conventional remote control 410 It was not easy, so it was implemented to improve the convenience of inputting text, etc. by implementing it similar to a PC keyboard.

On the other hand, the control means such as the remote control 410, the pointing device 420, the keyboard 430 is provided with a touch pad if necessary, so that text input, pointer movement, enlargement/reduction of pictures or videos, etc. for more convenient and various control purposes is available for

Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS. 5 to 17 .

5 is a block diagram of a multimedia device according to an embodiment of the present invention.

Referring to FIG. 5 , the multimedia device according to an embodiment of the present invention includes a voice signal receiver 510 , a user input interface 520 , a network interface 530 , a controller 540 , an application module 550 , A memory 560, a display unit 570, and the like are included. However, according to an embodiment, the multimedia device may include fewer or more configuration modules than the aforementioned configuration modules, and two configuration modules are implemented as one configuration module, or one configuration module includes two configuration modules. Of course, it can also be implemented in

The voice signal receiver 510 receives a voice signal. The voice signal receiver 510 may directly receive an external voice signal including a microphone and convert it into an electrical voice signal, or may receive an electrical voice signal from an external device. In the latter case, the voice signal receiver 510 may be implemented as the wireless communication unit 110 described above with reference to FIG. 1 , and the network interface unit 230 and the external device interface unit 235 described above with reference to FIG. 2 . ) and the user interface unit 250 may be implemented as at least one.

The user input interface unit 520 is for receiving a user input for controlling the operation of the multimedia device. According to an embodiment of the present invention, when it is desired to control the operation of the multimedia device through voice recognition, the voice signal receiving unit 510 may also serve as the user input interface unit 520 . The user input interface unit 520 may be implemented as the user input unit 123 described above with reference to FIG. 1 , or may be implemented as the user interface unit 250 described above with reference to FIG. 2 . When the display unit 570 includes a touch screen, the display unit 570 may also serve as the user interface unit 250 .

The network interface unit 530 provides an interface for connecting to a wired/wireless network. The multimedia device may transmit/receive data to and from an external server and an external device through the network interface unit 530 . When the multimedia device receives a voice signal from an external device, the network interface unit 530 may serve as the voice signal receiving unit 510 . The network interface unit 530 may be implemented as the wireless communication unit 110 described above with reference to FIG. 1 , or may be implemented as the network interface unit 230 described above with reference to FIG. 2 .

The control unit 540 controls the operation of each component module and the multimedia device.

An application (or program) is installed in the application module 550 . The application module 550 stores an application and metadata related to the application. The application may be a native application installed when the multimedia device is shipped or updated, or may be an application separately downloaded from an external server and installed by a user of the multimedia device. According to an embodiment, the application module 550 may be implemented as a part of the memory 560 .

The memory 560 may store programs, data, applications, commands, and the like for driving the multimedia device. According to an embodiment of the present invention, the memory 560 may store a voice recognition program for converting a voice signal received through the voice signal receiver 510 into text. According to an embodiment, when it is difficult to convert the voice signal received through the voice signal receiver 510 into text using only the voice recognition program stored in the memory 560 , the controller 540 converts the received voice signal to the network interface unit 530 . ) to an external server, and using the text conversion result transmitted from the external server, it is also possible to control the operation of the multimedia device.

Also, according to an embodiment, the memory 560 may be implemented in the controller 540 . The memory 560 may be implemented as the memory 170 described above with reference to FIG. 1 , or may be implemented with the storage unit 240 described above with reference to FIG. 2 .

The display unit 570 outputs information, data, etc. processed by the multimedia device. The display unit 570 may be implemented as the display unit 151 described above with reference to FIG. 1 , or may be implemented as the display unit 280 described above with reference to FIG. 2 .

6 is a flowchart illustrating an example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention. Hereinafter, an example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention will be described with reference to FIGS. 5 and 6 .

The display unit 570 of the multimedia device displays an execution screen of a specific application (or a specific program) (S610). The execution screen may include a plurality of menu items corresponding to functions of a specific application. The specific application may be a native application installed when the multimedia device is shipped or updated, or may be an application that is separately downloaded from an external server and installed by a user of the multimedia device. The specific application may be an application that can be used independently by itself, or may be an application required to enjoy content or use a predetermined function of the multimedia device. A specific application may be executed by a user's selection, execution may be automatically triggered by a specific event, or may be automatically executed by another application.

The controller 540 recognizes at least one of an object and a text in an image included in the execution screen of the specific application ( S620 ). When the execution screen of the specific application includes text, the text is recognized, and when the image is included, the object included in the image can be recognized.

According to an embodiment, the controller 540 may recognize at least one of an object and a text in an image included in the execution screen of the specific application only when a preset condition is satisfied. For example, the controller 540 may receive a predetermined signal for activating a function for controlling a multimedia device through voice recognition from the remote controller, or a voice signal received through the voice signal receiving unit 510 for voice recognition. At least one of an object and a text in an image included in the execution screen of the specific application may be recognized only when a trigger command for activating a function for controlling the multimedia device is included. In the latter case, the controller 540 converts the voice signal received through the voice signal receiver 510 into text, and when the converted text corresponds to a preset trigger command, an image included in the execution screen of the specific application At least one of My Object and Text can be recognized.

The object may include any one of letters, figures, and numbers included in the image. The memory 560 may store a program for recognizing an object included in an image. For example, the controller 540 may recognize an object in the image using an optical character recognition (OCR) program.

The controller 540 extracts (detects) position information of each of the recognized text and object (S630). Here, the position information of the recognized text and object may be coordinate information indicating where the recognized text and object are located on the displayed execution screen of the specific application.

Then, the controller 540 stores the extracted location information in the memory 560 in correspondence with the recognized text and object (S640).

According to an embodiment, the controller 540 may control the operation of the multimedia device (ie, the function of the multimedia device related to the execution screen of the specific application) through voice recognition so that the user can know that the specific application It is also possible to give a visual effect to the execution screen of

The controller 540 converts the voice signal received through the voice signal receiver 510 into text (S650).

Then, the controller 540 generates an event for selecting a menu item corresponding to the converted text (S660). Specifically, when a specific text or a specific object corresponding to the converted text exists among the recognized texts and objects, the controller 540 is configured to control a position corresponding to the specific text or specific object in the execution screen of the specific application. An event that selects an area corresponding to information is raised. According to an embodiment, even when only at least a part of the converted text is included in the recognized text and object, the controller 540 may include location information corresponding to the recognized text or object corresponding to the at least part of the text. It is also possible to generate an event that selects an area corresponding to .

According to an embodiment, the controller 540 may highlight an area corresponding to location information corresponding to the specific text or specific object in the execution screen of the specific application. Also, according to an embodiment, the controller 540 outputs a pointer corresponding to the movement of the remote controller to an arbitrary area on the execution screen of the specific application, and outputs a pointer corresponding to the converted text among the recognized texts and objects. When a specific text or a specific object exists, the pointer may be moved on an area corresponding to location information corresponding to the specific text or specific object in the execution screen of the specific application.

The control unit 540 may execute a function of a menu item corresponding to the area selected by the event among the functions of the specific application, and output a screen corresponding to the result of the function execution on the display unit 570 have.

Hereinafter, examples of an embodiment of the present invention described above with reference to FIG. 6 will be described in detail with reference to FIGS. 7 to 10 .

7 to 10 are diagrams for explaining an example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention.

Referring to FIG. 7A , an execution screen 700 of a specific application is displayed on the display unit 570 . The execution screen 700 may include a plurality of menu items 710 corresponding to a predetermined function of a specific application. According to an embodiment of the present invention, the user can view the execution screen 700 of a specific application (ie, functions related to a plurality of menu items 710 included in the execution screen 700 of the specific application) through voice recognition. can be controlled

Referring to (b) of FIG. 7 , when a preset condition is satisfied, the controller 540 may recognize at least one of a text included in the execution screen 700 and an object in an image. For example, the control unit 540 may control the text 721 (“User Manual”) included in the execution screen 700 and the object 722 in the image, “TV Installation”, “TV Operation”, “View Broadcast” , "Content", "File Sharing", "Smart TV", "Preferences", "Help"). Then, the controller 540 extracts the position information of each of the recognized text 721 and the object 722 . For example, the control unit 540 extracts the coordinate information _{of (X 1} to X ₂ , Y ₁ to Y ₂ ) as location information of the text 721 (“user manual”) on the execution screen 700 and _{, coordinate information of (X 3} to X ₄ , Y ₃ to Y ₄ ) may be extracted as location information of the object 710 (“TV operation”) on the execution screen 700 . Then, the controller 540 stores the extracted location information in the memory 560 in correspondence with the text 721 or the object 722 .

The user may receive a command (eg, "TV installation", "TV operation", "broadcast view", "content") corresponding to a desired menu item among the plurality of menu items 710 included in the execution screen 700 . , "file sharing", "Smart TV", "environment setting", or "help"), a function corresponding to the corresponding menu item may be executed. Execution of a function through voice recognition will be described in more detail with reference to FIGS. 9 and 10 .

According to an embodiment, the controller 540 may display the specific application on the execution screen 700 through voice recognition after recognition of the text/object included in the execution screen 700 and the identification of these location information is completed. It is also possible to provide a visual effect to inform the user that it can be controlled.

For example, referring to (a) of FIG. 8 , the controller 540 includes a message 810 (eg, “to be executed Please ignite the menu)" may be output to the execution screen 700 .

As another example, referring to (b) of FIG. 8 , the controller 540 corresponds to the recognized text 721 and the object 722 (eg, the recognized text 721 and the object ( 722) may be outputted corresponding to each of the extracted location information. This is to reduce errors in function control by voice recognition by guiding the user to more accurate utterance of commands. The indicator 820 may be located in an area of location information corresponding to each of the recognized text 721 or the object 722 in the execution screen 700 or in the vicinity of the area. The indicator 820 may include the recognized text 721 or the recognized object 722 . The user may execute/control a function corresponding to the text 721 or object 722 by uttering a command matching the text 721 or object 722 corresponding to the indicator 820 .

According to an embodiment, the controller 540 may highlight a menu item that the user wants to control from among the plurality of menu items 710 included in the execution screen 700 through voice recognition, or a pointer corresponding to the movement of the remote controller. can also be used to indicate.

Referring to FIG. 9A , the controller 540 outputs a pointer P corresponding to the movement of the remote controller to an arbitrary area on the execution screen 700 . For example, the arbitrary area may be an area that the user last pointed to on the screen of the display unit 570 using a remote controller or a preset arbitrary area. In FIG. 9A , it is assumed that the user has uttered the command “manipulate TV”.

The controller 540 converts a voice signal corresponding to the “TV operation” received through the voice signal receiver 510 into text, and the converted text is applied to the recognized text 721 and object 722 . to determine whether or not

Referring to FIG. 9B , the controller 540 controls the execution screen ( 700), the pointer P may be moved onto an area corresponding to the specific text or location information corresponding to the specific object. Although not shown, when there is a specific text or a specific object corresponding to the converted text among the recognized text 721 and the object 722 , the control unit 540 is configured to control the specific text in the execution screen 700 . Alternatively, a region corresponding to location information corresponding to a specific object may be highlighted. Also, the controller 540 controls an area corresponding to the location information corresponding to the specific text or the specific object, similarly to the pointer P clicking on the area corresponding to the location information corresponding to the specific text or the specific object. You can trigger an event that selects . According to an embodiment, when the multimedia device is the mobile terminal 100 described above with reference to FIG. 1 , the controller 540 determines that the user has touched an area corresponding to the location information corresponding to the specific text or specific object. Similarly, an event for selecting a region corresponding to location information corresponding to the specific text or specific object may be generated.

And, referring to FIG. 9C , the controller 540 controls a menu item corresponding to a region selected by the event among functions related to a plurality of menu items 710 included in the execution screen 700 . , and outputs a screen 900 corresponding to the result of the execution of the function to the display unit 570 . The user may also control a predetermined function of a specific application related to the screen 900 through voice recognition. This will be described with reference to FIG. 10 .

Referring to FIG. 10A , the controller 540 may recognize at least one of a text included in the screen 900 and an object in an image. As described above with reference to FIG. 7 , the controller 540 may recognize at least one of the text included in the screen 900 and the object in the image only when the preset condition is satisfied, and Unless activation of the control function is released, at least one of a text included in the screen 900 and an object in an image may be recognized when the screen 900 is output.

For example, the control unit 540 may control the text 910 included in the screen 900 , "User's Manual", "Operating TV", "Magic Remote Control", "Search by voice", "Input text by voice"","Screen remote control", "Integrated remote control", "Using keyboard/mouse"). Then, the controller 540 extracts location information of each of the recognized texts 910 . For example, the controller 540 may extract coordinate information _{of (X 5} to X ₇ , Y ₃ ) as location information of the text 910 (“search by voice”) on the screen 900 . . Then, the controller 540 stores the extracted location information in the memory 560 in correspondence with the text 910 . Although not shown, as described above with reference to FIG. 8 , the controller 540 may provide a visual effect to the screen 900 to notify the user that the specific application can be controlled through voice recognition. .

The user may receive a command corresponding to a desired menu item among a plurality of menu items included in the screen 900 (eg, "Search by voice", "Enter text by voice", "Screen remote control", "Integration A function corresponding to the corresponding menu item can be executed by igniting the remote control” or “using the keyboard/mouse”).

Meanwhile, the controller 540 outputs the pointer P corresponding to the movement of the remote controller to an arbitrary area on the screen 900 . For example, the arbitrary area may be an area that the user last pointed to on the screen of the display unit 570 using a remote controller or a preset arbitrary area. And, in FIG. 10A , it is assumed that the user has uttered a command "search by voice".

The controller 540 converts a voice signal corresponding to the “search by voice” received through the voice signal receiver 510 into text, and whether the converted text is included in the recognized text 910 . to judge

Referring to (b) of FIG. 10 , when a specific text corresponding to the converted text exists among the recognized texts 910 , the controller 540 is configured to respond to the specific text in the screen 900 . The pointer P may be moved on the area corresponding to the location information. Also, the controller 540 may highlight the region when the pointer P is positioned on the region corresponding to the location information corresponding to the specific text on the screen 900 . Also, an event for selecting an area corresponding to the location information corresponding to the specific text may be generated similarly to the pointer P clicking on the area corresponding to the location information corresponding to the specific text. According to an embodiment, when the multimedia device is the mobile terminal 100 described above with reference to FIG. 1 , the controller 540 controls the user to touch an area corresponding to the location information corresponding to the specific text, similarly to the An event for selecting an area corresponding to location information corresponding to a specific text may be generated. In addition, the controller 540 executes a function of a menu item corresponding to an area selected by the event among functions related to a plurality of menu items included in the screen 900, and performs a function corresponding to a result of the execution of the function. The screen is output to the display unit 570 .

As described above with reference to FIGS. 6 to 10 , the user may continuously operate a function of a specific application. According to the present embodiment, even when a menu item is included as an image rather than text on the execution screen of a specific application, there is an advantage in that the corresponding menu item can be manipulated through voice recognition.

11 is a flowchart illustrating another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention. Hereinafter, another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention will be described with reference to FIGS. 5 and 11 . A detailed description of the content overlapping with the above in relation to FIGS. 6 to 10 will be omitted.

The display unit 570 of the multimedia device displays an execution screen of a specific application (or a specific program) ( S1110 ). The execution screen may include a plurality of menu items corresponding to functions of a specific application.

The controller 540 requests the specific application for location information on a plurality of menu items included in the execution screen (S1120). The request may include a signal for requesting information necessary to identify the execution screen among execution screens of a specific application and location information of a plurality of menu items included in the execution screen. The application module 550 stores the specific application and metadata related to the specific application. The metadata may include information necessary to identify each of the execution screens of the specific application and location information of a plurality of menu items included in each execution screen. Here, the location information may be coordinate information indicating where each menu item is located on the execution screen.

The controller 540 recognizes at least one of a text included in the execution screen and an object in an image based on the requested location information (S1130). For example, the application module 550 may load location information of a plurality of menu items included in the execution screen into the memory 560 according to the request of the controller 540 . According to an embodiment, the application module 550 may also load the identification information for identifying each of the menu items together with the location information of the plurality of menu items included in the execution screen into the memory 560 . In addition, the controller 540 may recognize at least one of a text and an object in an image based on an area corresponding to the location information of each of the plurality of requested menu items in the execution screen. According to this embodiment, instead of recognizing the text/object in the entire execution screen, based on the requested location information, based on the area corresponding to the location information of each of the plurality of requested menu items in the execution screen Since text/object is recognized, resources and time consumed for voice recognition control can be significantly reduced, and the recognition rate of text/object can be improved.

Then, the controller 540 stores the recognized text and object in the memory 560 in response to each menu item included in the execution screen (S1140). For example, when recognizing a specific text in an area corresponding to the requested location information of the specific menu item on the execution screen, the controller 540 may match the specific text with the location information of the specific menu item to the memory 560 ) can be stored in As another example, when recognizing a specific text in an area corresponding to the requested location information of the specific menu item on the execution screen, the controller 540 may convert the specific text to the specific text corresponding to the location information of the specific menu item. It may be stored in the memory 560 in correspondence with the identification information of the menu item.

According to an embodiment, the controller 540 may control the operation of the multimedia device (ie, the function of the multimedia device related to the execution screen of the specific application) through voice recognition so that the user can know that the specific application It is also possible to give a visual effect to the execution screen of

The controller 540 converts the voice signal received through the voice signal receiver 510 into text (S1150).

Then, the controller 540 generates an event for selecting a menu item corresponding to the converted text (S1160). Specifically, when there is a specific text or a specific object corresponding to the converted text among the recognized texts and objects, the controller 540 may control a menu item corresponding to the specific text or specific object in the execution screen (that is, , an event to select an area corresponding to the location information of the menu item) is generated. According to an embodiment, even when only at least a part of the converted text is included in the recognized text and object, the controller 540 may include a menu item corresponding to the recognized text or object corresponding to the at least part of the text. (ie, an area corresponding to the location information of the menu item) may be generated.

According to an embodiment, the controller 540 may highlight a menu item (ie, an area corresponding to location information of the menu item) corresponding to the specific text or specific object in the execution screen of the specific application. Also, according to an embodiment, the controller 540 outputs a pointer corresponding to the movement of the remote controller to an arbitrary area on the execution screen of the specific application, and outputs a pointer corresponding to the converted text among the recognized texts and objects. When a specific text or a specific object exists, the pointer may be moved onto a menu item (ie, an area corresponding to location information of the menu item) corresponding to the specific text or specific object on the execution screen of the specific application. .

The control unit 540 may execute a function of a menu item corresponding to the area selected by the event among the functions of the specific application, and output a screen corresponding to the result of the function execution on the display unit 570 have.

According to an embodiment, when the specific application is downloaded from an external server, the controller 540 may also receive metadata of the specific application and store it in the memory 560 . The metadata of the specific application may include identification information for identifying each of the execution screens of the specific application and location information of each of a plurality of menu items included in each execution screen. Accordingly, the control unit 540 may omit the above-described step (S1120).

Hereinafter, examples of an embodiment of the present invention described above with reference to FIG. 11 will be described in detail with reference to FIGS. 12 to 15 .

12 to 15 are diagrams for explaining another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention. A detailed description of the content overlapping with the above in relation to FIGS. 6 to 11 will be omitted.

Referring to FIG. 12A , an execution screen 1200 of a specific application is displayed on the display unit 570 . The execution screen 1200 may include a plurality of menu items 1210 corresponding to a predetermined function of a specific application.

Referring to FIG. 12B , when a preset condition is satisfied, the controller 540 requests the specific application for location information on a plurality of menu items 1210 included in the execution screen 1200 . The control unit 540 controls the text 1221, “TV installation”, “TV operation ", "broadcast view", "content", "file sharing", "Smart TV", "environmental setting", "help") and an object in the image 1222 (web OS) can be recognized. In addition, the controller 540 stores the recognized text 1221 and the object 1222 in the memory 560 in response to each menu item 1210 included in the execution screen 1200 . Specifically, the controller 540 may correspond to the recognized text 1221 and the object 1222 with an area E corresponding to the location information of each menu item 1210 included in the execution screen 1200 . It may be stored in the memory 560 .

A user may receive a command (eg, "TV installation", "web OS", "TV operation", "view broadcast") corresponding to a desired menu item among a plurality of menu items 1210 included in the execution screen 1200 . ", "content", "file sharing", "Smart TV", "environment setting", or "help") may execute a function corresponding to the corresponding menu item. Execution of a function through voice recognition will be described in more detail with reference to FIGS. 14 and 15 .

According to an embodiment, the controller 540 controls the specific application on the execution screen 1200 through voice recognition after recognition and storage of texts/objects included in the execution screen 1200 are completed. A visual effect may be provided to notify the user that it is possible.

For example, referring to (a) of FIG. 13 , the controller 540 receives a message 1310 (eg, “to run Please ignite the menu)" may be output to the execution screen 1200 .

As another example, referring to FIG. 13B , the controller 540 may control the recognized text 1221 and the object 1222 corresponding to (eg, the recognized text 1221 and the object (eg) 1222 ) may be output in response to the requested location information of each menu item 1310 . This is to reduce errors in function control by voice recognition by guiding the user to more accurate utterance of commands. The indicator 1320 may be located in the area of the location information corresponding to each of the recognized text 1321 or the object 1322 in the execution screen 1300 or in the vicinity of the area. The indicator 1320 may include or point to the recognized text 1221 , or may include or point to the recognized object 1222 . The user executes the function of the menu item 1210 corresponding to the text 1221 or object 1222 by uttering a command matching the text 1221 or object 1222 corresponding to the indicator 1320 . / can be controlled.

According to an embodiment, the controller 540 highlights a menu item that the user wants to control from among the plurality of menu items 1210 included in the execution screen 1200 through voice recognition, or a pointer corresponding to the movement of the remote controller. can also be used to indicate.

Referring to FIG. 14A , the controller 540 outputs a pointer P corresponding to the movement of the remote controller to an arbitrary area on the execution screen 1200 . For example, the arbitrary area may be an area that the user last pointed to on the screen of the display unit 570 using a remote controller or a preset arbitrary area. In (a) of FIG. 14 , it is assumed that the user has uttered a command “manipulate TV”.

The controller 540 converts a voice signal corresponding to the “TV operation” received through the voice signal receiver 510 into text, and the converted text is applied to the recognized text 1221 and the object 1222 . to determine whether or not

Referring to (a) of FIG. 14 , the controller 540 controls the execution screen ( 1200), the pointer onto an area of the menu item 1210 corresponding to the specific text or specific object (eg, an area corresponding to location information of the menu item 1210 corresponding to the specific text or specific object). (P) can be moved. Although not shown, when there is a specific text and a specific object corresponding to the converted text among the recognized text 1221 and the object 1222 , the control unit 540 is configured to control the specific text in the execution screen 1200 . Alternatively, the region of the menu item 1210 corresponding to the specific object (eg, the region corresponding to the specific text or location information of the menu item 1210 corresponding to the specific object) may be highlighted. Also, the controller 540 controls a menu item corresponding to the specific text or specific object (eg, the specific text), similarly to the pointer P clicking a menu item corresponding to the specific text or specific object. Alternatively, an event for selecting an area corresponding to location information of the menu item 1210 corresponding to a specific object) may be generated. According to an embodiment, when the multimedia device is the mobile terminal 100 described above with reference to FIG. 1 , the controller 540 controls the specific text or the specific object, similarly to the user touching a menu item corresponding to the specific text or the specific object. An event for selecting a menu item corresponding to text or a specific object (eg, an area corresponding to location information of the menu item 1210 corresponding to the specific text or specific object) may be generated.

And, referring to FIG. 14C , the controller 540 executes a function of a menu item selected by the event among functions related to a plurality of menu items 1210 included in the execution screen 1200 . and output the screen 1400 corresponding to the result of the execution of the function to the display unit 570 . The user may also control a predetermined function of a specific application related to the screen 1400 through voice recognition. This will be described with reference to FIG. 15 .

Referring to (a) of FIG. 15 , the controller 540 requests location information for a plurality of menu items included in the screen 1400 to a specific application, and based on the requested location information, the screen 1400 At least one of the text included in 1400 and the object in the image may be recognized. As described above with reference to FIG. 7 , the controller 540 may recognize at least one of a text included in the screen 1400 and an object in an image only when the preset condition is satisfied, and Unless activation of the control function is released, at least one of a text included in the screen 1400 and an object in an image may be recognized when the screen 1400 is output.

For example, the controller 540 may control the text 1410 (“User Manual”, “Operating TV”, “Magic Remote Control ", "Search by voice", "Enter text by voice", "Screen remote control", "Integrated remote control", "Using keyboard/mouse"). In addition, the controller 540 may store the recognized text 1410 in the memory 560 in response to each menu item included in the screen 1400 . For example, the controller 540 may store the recognized text 1410 in the memory 560 in correspondence with location information of each menu item included in the screen 1400 . As another example, the controller 540 may store the recognized text 1410 in the memory 560 in correspondence with location information of each menu item included in the screen 1400 and identification information of a specific menu item corresponding to that of the screen 1400 . may be Although not shown, as described above with reference to FIG. 13 , the controller 540 may provide a visual effect to the screen 1400 to notify the user that the specific application can be controlled through voice recognition. .

A user may select a command corresponding to a desired menu item among a plurality of menu items included in the screen 1400 (eg, "Search by voice", "Enter text by voice", "Screen remote control", "Integration A function corresponding to the corresponding menu item can be executed by igniting the remote control” or “using the keyboard/mouse”).

Meanwhile, the controller 540 outputs the pointer P corresponding to the movement of the remote controller to an arbitrary area on the screen 1400 . For example, the arbitrary area may be an area that the user last pointed to on the screen of the display unit 570 using a remote controller or a preset arbitrary area. And, in FIG. 15A , it is assumed that the user has uttered a command "search by voice".

The controller 540 converts a voice signal corresponding to the “search by voice” received through the voice signal receiver 510 into text, and whether the converted text is included in the recognized text 1410 . to judge

Referring to (b) of FIG. 15 , when there is a specific text corresponding to the converted text among the recognized text 1410 , the controller 540 is configured to respond to the specific text in the screen 1400 . The pointer P may be moved over a menu item (eg, an area corresponding to location information of a menu item corresponding to the specific text). Also, when the pointer P is positioned on a menu item corresponding to the specific text on the screen 1400 , the controller 540 may control the menu item (eg, an area corresponding to location information of the menu item). ) can also be highlighted. Also, similarly to when the pointer P clicks on a menu item corresponding to the specific text, a menu item corresponding to the specific text (eg, an area corresponding to location information of a menu item corresponding to the specific text) ) can be triggered. According to an embodiment, when the multimedia device is the mobile terminal 100 described above with reference to FIG. 1 , the control unit 540 corresponds to the specific text as if the user touched a menu item corresponding to the specific text. An event for selecting a selected menu item (eg, an area corresponding to location information of a menu item corresponding to the specific text) may be generated. In addition, the controller 540 executes a function of a menu item corresponding to an area selected by the event among functions related to a plurality of menu items included in the screen 1400, and a screen corresponding to the result of executing the function. is output to the display unit 570 .

As described above with reference to FIGS. 11 to 15 , the user may continuously operate a function of a specific application. According to this embodiment, instead of recognizing the text/object in the entire execution screen, based on the requested location information, based on the area corresponding to the location information of each of the plurality of requested menu items in the execution screen Since text/object is recognized, resources and time consumed for voice recognition control can be significantly reduced, and the recognition rate of text/object can be improved.

16 is a flowchart illustrating another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention. Hereinafter, another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention will be described with reference to FIGS. 5 and 16 . A detailed description of the content overlapping with the above in relation to FIGS. 6 to 15 will be omitted.

The display unit 570 of the multimedia device displays an execution screen of a specific application (or a specific program) (S1610). The execution screen may include a plurality of menu items corresponding to functions of a specific application.

The controller 540 requests the specific application for location information on each of a plurality of menu items included in the execution screen and utterance information corresponding to each menu item (S1620). The request includes information necessary for identifying the execution screen among execution screens of a specific application, a signal requesting location information of a plurality of menu items included in the execution screen, and a signal requesting utterance information corresponding to each menu item may be included. The application module 550 stores the specific application and metadata related to the specific application. The metadata may include information necessary to identify each of the execution screens of the specific application, location information of a plurality of menu items included in each execution screen, and utterance information corresponding to each menu item. Here, the location information may be coordinate information indicating where each menu item is located on the execution screen, and the utterance information may be command information for controlling each menu item through voice recognition. The application module 550 may load location information of a plurality of menu items included in the execution screen and utterance information corresponding to each menu item into the memory 560 according to a request from the controller 540 .

According to an embodiment, after the location information and the utterance information are loaded into the memory 560 , the controller 540 operates the multimedia device through voice recognition (ie, the function of the multimedia device related to the execution screen of the specific application). ), a visual effect may be provided to the execution screen of the specific application so that the user can know that it can be controlled.

The controller 540 converts the voice signal received through the voice signal receiver 510 into text (S1630).

Then, the controller 540 generates an event for selecting a menu item corresponding to the converted text (S1640). Specifically, when there is specific utterance information corresponding to the converted text among the requested utterance information, the controller 540 is configured to control a menu item corresponding to the specific utterance information from among a plurality of menu items included in the execution screen. (ie, an area corresponding to the location information of the menu item) is generated. According to an embodiment, even when only at least a part of the converted text is included in the requested utterance information, a menu item corresponding to the utterance information corresponding to the at least part of the text (that is, the menu item corresponding to the location information of the menu item) An event to select an area) can be raised.

According to an embodiment, the control unit 540 may be, based on the location information of the requested menu item, a menu item corresponding to the specific utterance information in the execution screen of the specific application (ie, the menu item corresponding to the location information of the menu item). area) can also be highlighted. Also, according to an embodiment, the controller 540 outputs a pointer corresponding to the movement of the remote controller to an arbitrary area on the execution screen of the specific application, and outputs a pointer corresponding to the converted text among the requested utterance information. When specific utterance information exists, the pointer may be moved on a menu item corresponding to the utterance information (ie, an area corresponding to location information of the menu item) on the execution screen of the specific application.

The control unit 540 may execute a function of a menu item corresponding to the area selected by the event among the functions of the specific application, and output a screen corresponding to the result of the function execution on the display unit 570 have.

According to an embodiment, when the specific application is downloaded from an external server, the controller 540 may also receive metadata of the specific application and store it in the memory 560 . The metadata of the specific application may include identification information for identifying each of the execution screens of the specific application, location information of each of a plurality of menu items included in each execution screen, and utterance information corresponding to each menu item. have. Accordingly, the control unit 540 may omit the above-described step (S1620).

Hereinafter, examples of an embodiment of the present invention described above with reference to FIG. 16 with reference to FIG. 17 will be described in detail.

17 is a diagram for explaining another example of a method for controlling a multimedia device through voice recognition according to an embodiment of the present invention. A detailed description of the content overlapping with the above in relation to FIGS. 6 to 16 will be omitted.

Referring to FIG. 17A , an execution screen 1700 of a specific application is displayed on the display unit 570 . The execution screen 1700 may include a plurality of menu items 1710 corresponding to a predetermined function of a specific application.

When a preset condition is satisfied, the control unit 540 requests the specific application for location information for each of a plurality of menu items included in the execution screen 1700 and utterance information corresponding to each menu item, to the application module ( Location information for each of the plurality of menu items transmitted from 550 and utterance information corresponding to each menu item may be stored in the memory 560 .

When the text converted from the electrical signal received through the voice signal receiver 510 is included in the requested speech information, the controller 540 selects a menu item corresponding to specific speech information corresponding to the converted text. A selected event may be generated, and a function of the menu item may be executed according to the event.

According to an embodiment, the controller 540 may recognize at least one of a text and an object in an image included in the execution screen 1700 based on the location information of each requested menu item. In addition, the controller 540 may store the recognized text and object in the memory 560 in correspondence with each menu item (eg, location information of each menu item). Thereafter, when the text converted from the electrical signal received through the voice signal receiver 510 is included in the requested speech information, the controller 540 is configured to correspond to specific speech information corresponding to the converted text. When an event of selecting a menu item is generated, and text converted from an electrical signal received through the voice signal receiver 510 is not included in the requested utterance information but is included in the recognized text and object, the Generates an event for selecting a specific text corresponding to the converted text or a menu item corresponding to a specific object (eg, an area corresponding to location information of each menu item), and according to the event, the function of the menu item is changed can run

According to an embodiment, the controller 540 may provide a visual effect to the execution screen 1700 to notify the user that the specific application can be controlled through voice recognition.

For example, referring to FIG. 17B , the control unit 540 provides a message 1720 (eg, “to be executed”) for guiding the user that the specific application can be controlled through voice recognition. Please ignite the menu)" may be output to the execution screen 1700 .

As another example, referring to (c) of FIG. 17 , the controller 540 corresponds to each utterance information with an indicator 1730 including the requested utterance information based on the requested location information and utterance information. A menu item (eg, an area corresponding to location information of a menu item corresponding to each piece of utterance information) may be output. This is to reduce errors in function control by voice recognition by guiding the user to more accurate utterance of commands.

As another example, referring to (d) of FIG. 17 , the controller 540 is configured to provide the location information of each of the requested menu items in the execution screen 1700 based on the location information of the requested menu items. An object in text and an image may be recognized based on a corresponding area, and the recognized text and object may be stored in the memory 560 in correspondence with location information of each menu item. In addition, the controller 540 may output an indicator 1740 corresponding to the recognized text and object in response to each menu item (eg, an area corresponding to location information of each menu item). The control unit 540, when the text converted from the voice signal received through the voice signal receiver 510 corresponds to the requested utterance information, the recognized text, and/or the recognized object, the converted text An event for selecting a menu item corresponding to . According to the present embodiment, even when the user utters a command included in the execution screen 1700 but does not match at least part of the requested utterance information, the function of the menu item conforming to the user's intention is executed. There are advantages to being able to The embodiment of FIG. 17( d ) may be understood with reference to the contents described above in relation to FIGS. 11 to 14 .

In addition, since the execution of a function through voice recognition overlaps with that described above with reference to FIGS. 9, 10, 14 and 15 , a description thereof will be omitted.

According to the above-described embodiments of the present invention, a method of recognizing an object included in an image as well as text output on a screen of a multimedia device is proposed, thereby enabling a function corresponding to the image to be controlled through voice recognition. A device and a control method thereof can be provided. In addition, it is possible to provide a multimedia device capable of significantly reducing resources and time consumed for voice recognition control by receiving information about a menu item output on a screen of the multimedia device from an application, and a method for controlling the same.

The present invention described above can be implemented as computer-readable codes on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. In addition, the computer may include a control unit of the terminal. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

510: voice signal receiver
540: control unit

Claims (15)

a voice signal receiver for receiving a voice signal;
a display unit for displaying an execution screen of a specific application; and
control unit; including;
The execution screen of the specific application includes a plurality of menu items corresponding to the function of the specific application,
The control unit is
When a preset condition is satisfied, the specific application requests location information of each of the plurality of menu items and utterance information corresponding to each of the plurality of menu items on the execution screen of the specific application,
Recognizing at least one of an object in a text and an image around an area corresponding to the location information of each of the plurality of menu items,
converting the voice signal received through the voice signal receiver into text;
When specific speech information corresponding to the converted text exists among the requested speech information, generating a signal for selecting a menu item corresponding to the specific speech information from among the plurality of menu items. The method of claim 1,
The control unit is
Outputs a pointer corresponding to the movement of the remote controller to an arbitrary area on the execution screen of the specific application,
When specific speech information corresponding to the converted text exists among the requested speech information, the pointer is moved to a menu item corresponding to the specific speech information based on the location information of each of the plurality of requested menu items Let's say, multimedia device. The method of claim 1,
The control unit is
A multimedia device for outputting an indicator including the requested utterance information in response to a menu item corresponding to each of the requested utterance information. The method of claim 1,
The control unit is
Based on the location information and utterance information of each of the plurality of requested menu items, a visual effect for notifying that control is possible using a voice signal is provided to the plurality of menu items. The method of claim 1,
The preset condition is,
A multimedia device comprising at least one of a condition in which a preset signal is received from a remote controller and a condition in which text converted from a voice signal received through the voice signal receiver corresponds to a preset text. The method of claim 1,
The control unit is
Execute the function of the menu item selected by the event among the functions of the specific application,
Displaying a first screen corresponding to the result of the function execution on the display unit,
The multimedia device of claim 1, which requests the specific application for location information of each of the plurality of menu items included in the first screen and utterance information corresponding to each of the plurality of menu items included in the first screen. The method of claim 1,
Further comprising; a network interface unit for receiving the specific application and metadata for the specific application from an external server;
The metadata for the specific application includes information for identifying each execution screen of the specific application, location information of each of a plurality of menu items included in each execution screen, and each of a plurality of menu items included in each execution screen A multimedia device comprising utterance information corresponding to . displaying an execution screen of a specific application, wherein the execution screen includes a plurality of menu items corresponding to functions of the specific application;
when a preset condition is satisfied, requesting the specific application for location information of each of the plurality of menu items on the execution screen of the specific application;
recognizing at least one of a text included in an execution screen of the specific application and an object in an image included in an execution screen of the specific application, centering on an area corresponding to the location information of each of the plurality of requested menu items;
storing the recognized text and object corresponding to each menu item based on the location information of each of the plurality of requested menu items;
converting the voice signal received through the voice signal receiver into text; and
generating an event for selecting a menu item corresponding to the specific text or specific object from among the plurality of menu items when a specific text or a specific object corresponding to the converted text exists among the recognized texts and objects; Control method of a multimedia device comprising a. 9. The method of claim 8,
outputting a pointer corresponding to the movement of the remote controller to an arbitrary area on the execution screen of the specific application; and
When a specific text or a specific object corresponding to the converted text exists among the recognized texts and objects, the pointer is set to correspond to the specific text or specific object based on location information of each of the plurality of requested menu items. The method of controlling a multimedia device further comprising; moving on the menu item. 9. The method of claim 8,
outputting an indicator including the recognized text or the recognized object in response to a menu item corresponding to the recognized text or the recognized object based on the location information of each of the plurality of requested menu items; Control method of a multimedia device further comprising a. 9. The method of claim 8,
and providing a visual effect to the plurality of menu items to inform that they can be controlled using a voice signal, based on the location information of each of the plurality of requested menu items. 9. The method of claim 8,
The preset condition is,
A method of controlling a multimedia device, comprising at least one of a condition in which a preset signal is received from a remote controller and a condition in which text converted from the voice signal received through the voice signal receiver corresponds to the preset text. 9. The method of claim 8,
executing a function of a menu item selected by the event among functions of the specific application;
displaying a first screen corresponding to a result of executing the function; and
and requesting the specific application for location information of each of the plurality of menu items included in the first screen. 9. The method of claim 8,
Recognizing at least one of a text included in an execution screen of the specific application and an object in an image included in an execution screen of the specific application based on the location information of each of the plurality of requested menu items,
and recognizing an object in the text and the image in an area corresponding to the location information of each of the plurality of requested menu items among the displayed execution screen of the specific application. delete

Images