KR101730881B1 - Augmented Remote Controller and Method of Operating the Same - Google Patents

Abstract

The present invention relates to an augmented remote control device for providing a user interface using an augmented reality technology and an operation method thereof. A method of operating an augmented remote control device according to an embodiment of the present invention includes the steps of recognizing information on the external device and the external device, transmitting information on the external device to an image display device connected to a home network, Receiving control interface information for the external device from the video display device, and displaying a GUI (Graphic User Interface) related to the GUI using the control interface information. According to an embodiment of the present invention, a user can easily establish a connection with an external device, receive a manual for use and control of the external device, and receive a user interface corresponding to the usage method of the external device through the augmented remote control device So that the external device and the video display device can be controlled.

Description

[0001] The present invention relates to an Augmented Remote Controller and a method of operating the same,

The present invention relates to an augmented remote control device, a method of controlling an augmented remote control device, and a system thereof. The augmented remote control device can recognize a content or other kinds of objects provided by an image display device, an external device that can be connected to the image display device, an image display device or an external device, and display related information. The user can control surrounding objects and receive related information by using the augmented remote control device.

A video display device is a device having a function of displaying an image that a user can view. The user can view the broadcast through the video display device. The video display device can be connected to an external device. The user can view the content provided by the external device through the video display device connected to the external device. The video display device may be connected to a predetermined content provider through a network. The user can view contents provided by a predetermined content provider through an image display device connected to the network.

Information related to contents is increasing along with contents that a user can view through a video display device. The user wants to check information related to the content being watched in addition to viewing the content through the video display device. The user also wants to efficiently control the video display device and various kinds of external devices using a single device.

Accordingly, it is an object of the present invention to provide a remote control device capable of controlling an external device connected to an image display device or an image display device.

It is another object of the present invention to provide a remote control device that allows a user to efficiently use and manage content and content-related information reproduced or stored in an external device connected to an image display device or an image display device.

Also, in dealing with electronic devices including external devices, it is desired that the user can easily acquire manuals and provide information on proper usage.

According to another aspect of the present invention, there is provided a method of operating an augmented remote control device for providing a user interface using an augmented reality technology, the method comprising: recognizing information on the external device and the external device; Receiving a control interface information for the external device from the video display device, and transmitting a GUI-related GUI (Graphic User Interface) using the control interface information to the video display device connected to the home network .

According to another aspect of the present invention, there is provided a method of operating an augmented remote controller for providing a user interface using an augmented reality technology, the method comprising: recognizing information on the external device and the external device; Downloading the control interface information from a control interface database storing control interface information for the apparatus, and displaying a GUI (Graphic User Interface) related to the external apparatus using the control interface information.

According to another aspect of the present invention, there is provided an augmented remote control apparatus for providing a user interface using an augmented reality technology, including an identification unit for recognizing information on the external device and the external device, A wireless communication unit for transmitting information on the external device to a video display device connected to a home network, receiving control interface information for the external device from the video display device, and a GUI (Graphic User Interface) And a display unit for displaying the image.

According to another aspect of the present invention, there is provided an augmented remote control device for providing a user interface using an augmented reality technology, including an identification unit for recognizing information on the external device and the external device, A wireless communication unit for downloading the control interface information from a control interface database in which control interface information is stored and a control unit for generating a GUI related to the external device using the control interface information, User Interface).

According to the embodiment of the present invention, an image display apparatus or an external apparatus connected to the image display apparatus can be controlled by using one remote control apparatus. In addition, when controlling the video display device or the external device, the user can receive the manual in real time, and can operate the devices in accordance with the manual. In particular, it is possible to easily provide a countermeasure method in case of failure and a device operation method which is difficult to find by an augmented remote control device.

1 is a block diagram illustrating an augmented remote control system system according to an embodiment of the present invention;
2 is a block diagram illustrating an augmented remote control system system according to an embodiment of the present invention;
3 is an internal block diagram of a video display device according to an embodiment of the present invention.
4 is an internal block diagram of a video / audio processing unit of a video display device according to an embodiment of the present invention;
5 is an internal block diagram of an augmented remote control device according to an embodiment of the present invention;
6 is a flowchart illustrating a flow of an operation method of an augmented remote controller according to an embodiment of the present invention.
7 is a view schematically showing an appearance of an augmented remote control apparatus according to an embodiment of the present invention;
8 is a diagram illustrating an augmented remote control apparatus and an operation method thereof according to an embodiment of the present invention.
9 is a flow chart illustrating an augmented remote control device and a method of operation thereof according to another embodiment of the present invention.
10 is a flow chart illustrating an operation method of an augmented remote control apparatus according to another embodiment of the present invention.
FIGS. 11 to 13D are views showing a GUI associated with a manual according to an operation method of the augmented remote control device according to the embodiment of the present invention.
FIGS. 14A to 14C are diagrams illustrating a GUI associated with a manual according to an operation method of the augmented remote control apparatus according to another embodiment of the present invention; FIG.
FIGS. 15A to 15C are diagrams illustrating a GUI associated with a manual according to an operation method of the augmented remote control apparatus according to another embodiment of the present invention; FIG.

Hereinafter, the present invention will be described in detail with reference to the drawings.

The suffix "module" and " part "for components used in the following description are given merely for convenience of description, and do not give special significance or role in themselves. Accordingly, the terms "module" and "part" may be used interchangeably.

The augmented remote control apparatus according to an embodiment of the present invention is capable of providing an augmented reality environment to a user by providing information related to an object recognized and recognized by the user to the user . The object existing around the user may be an image display device which can be controlled using the augmented remote control device, an external device connected to the image display device, a content provided by the image display device or an external device, or another kind of object .

The augmented remote control device can collect the user related information and recognize the objects existing around the user. For example, the augmented remote control device can acquire information about the user's position or the user's direction using a GPS or a compass. In addition, the augmented remote control device can photograph a user's surrounding environment using a camera and recognize an object existing in the surroundings. In addition, the augmented remote control device can recognize an object existing around the user by using an RFID (Radio Frequency Identification) reader.

The augmented remote control device recognizes an object existing around the user and can retrieve and display information related to the recognized object. The type of information searched through the augmented remote control device may correspond to the type of the recognized object.

For example, if the augmented remote control device recognizes an image display device or an external device existing in the vicinity of the user, the augmented remote control device can search information about the content list that can be provided through the image display device or the external device . Further, the augmented remote control device can retrieve information related to the user interface that can control the video display device or the external device. The augmented remote control device can display the retrieved information to the user with respect to the video display device or the external device existing around the user.

In addition, the augmented remote control device can recognize contents provided by an image display device or an external device existing around the user. In this case, the augmented remote control device can retrieve information about the content. The augmented remote control device can display to the user information about the content provided by the video display device or the external device. In addition, the augmented remote control device can display to the user a user interface that can edit, reproduce, or transmit contents provided by the video display device or the external device.

Also, the augmented remote control device can recognize other kinds of objects existing around the user. For example, a user can take pictures of furniture existing in the vicinity by using a camera mounted on the augmented remote control device. The augmented remote control device can recognize that the object corresponding to the image photographed through the camera is a household by referring to a database storing information about images of various objects. The augmented remote control device can retrieve information about the name or manufacturer of the furniture and display it to the user.

The augmented remote control device can enhance and display the retrieved information about the object photographed through the camera on the actual image photographed through the camera. For example, the augmented remote control device can display an actual image photographed through a camera on a display unit. The augmented remote control device can retrieve information about an object included in the actual image displayed on the display unit. The augmented remote control device can display information about the searched object on a display unit using a pop-up window or an icon. Further, the augmented remote control device can display information about the retrieved object on the display unit as an image or text.

The user can confirm the augmented real image of the information retrieved from the augmented remote control device through the augmented remote control device. The user can confirm information about a predetermined object included in an actual image or an actual image through augmented information displayed so as to overlap with an actual image photographed through a camera.

Also, when the display unit on which the enhanced actual image is displayed is a touch screen, the user can select a pop-up window, an icon, an image or text representing the augmentation information, and interact with the augmented remote control device. For example, if the user selects a pop-up window showing first enhancement information, the augmented remote control device can display detailed information related to the first enhancement information on the display unit. Further, when the user selects a pop-up window indicating first enhancement information, the augmented remote control device can execute an application related to the first enhancement information. In this case, the application executed by the augmented remote control device may be an application capable of controlling an object such as an image display device or an external device included in the augmented reality image.

In the case of the augmented remote control apparatus using the transparent display unit, the augmented remote control apparatus can display the augmentation information so that the augmented information overlaps with the actual image projected through the transparent display unit. The augmented remote control apparatus searches for information about a predetermined object included in an actual image projected through the transparent display unit, and displays the retrieved information on the transparent display unit as reinforcement information.

The augmented remote control device can wirelessly transmit and receive signals to and from an external device that can be connected to the image display device or the image display device. The user can control the video display device or the external device using the augmented remote control device. Further, the augmented remote control apparatus can receive information on the operation state of the image display apparatus or the external apparatus and display the information on the display unit.

 The augmented remote control device may be connected to a network including the Internet network. The augmented remote control device can search information on the recognized object through the network. The augmented remote control device can display the information retrieved through the network on the display unit.

1 is a block diagram illustrating an augmented remote control system according to an embodiment of the present invention.

The augmented remote control device 200 can transmit and receive signals to and from the video display device 100, the external device 30 that can be connected to the video display device 100, and the network server 300. The video display device 100 is a device capable of reproducing various kinds of contents. The image display apparatus 100 can receive a signal including a video signal corresponding to the content from the outside. The image display apparatus 100 can extract the image signal included in the received signal and display an image corresponding to the extracted image signal.

The external device 30 is a device capable of reproducing contents stored in conformity with a predetermined standard. The external device 30 may include a display unit. In this case, the external device 30 can display an image corresponding to the content to be reproduced on the display unit of the external device 30. Also, the external device 30 may be connected to the image display device 100. In this case, the external device 30 can transmit a signal including the video signal corresponding to the content to the video display device 100. [ The image display apparatus 100 may display an image corresponding to a video signal included in a signal received through the external apparatus 30. [

The video display apparatus 100 can receive a broadcast signal through the broadcast station 10. [ The image display apparatus 100 can display an image corresponding to a video signal included in a broadcast signal. The image display apparatus 100 may receive a signal including a video signal from the network server 300 in the network including the Internet. The image display apparatus 100 may display an image corresponding to a video signal included in the signal received by the network server 300. [

In the case of the video display device 100 connected to the Internet, a signal including a video signal corresponding to a predetermined content can be received through a content provider 20 provided in the Internet network and providing content. The image display apparatus 100 can receive a signal transmitted from the content provider 20 and display an image corresponding thereto.

The augmented remote control device 200 can recognize the image display device 100 or the external device 30. [ The augmented remote control apparatus 200 can photograph the user's surrounding environment through the camera and recognize the image display apparatus 100 or the external apparatus 30 through the image discrimination. The augmented remote control device 200 also receives a signal transmitted from the image display device 100 or the external device 30 to which the RFID is attached through the RFID reader and transmits the signal to the image display device 100 or the external device 30) can be recognized.

The augmented remote control apparatus 200 can also recognize the corresponding image display apparatus 100 or the external apparatus 30 by transmitting and receiving different kinds of signals to and from the image display apparatus 100 or the external apparatus 30. [ For example, the augmented remote control device 200 can transmit a signal according to the IR or RF communication standard to the image display device 100 or the external device 30. The augmented remote control apparatus 200 can be paired with the image display apparatus 100 or the external apparatus 30 that transmits and receives a signal according to the IR or RF communication standard. The image display apparatus 100 or the external apparatus 30 can recognize a signal transmitted from the augmented remote control apparatus 200 that has been paired.

The augmented remote control apparatus 200 can display a menu for controlling the recognized image display apparatus 100 or the external apparatus 30 on the display unit of the augmented remote control apparatus 200. [ The user can select the menu displayed on the display unit of the augmented remote control apparatus 200 or control the video display apparatus 100 or the external apparatus 30 by operating buttons or keys provided in the augmented remote control apparatus 200 You can enter the command that you have. The augmented remote control apparatus 200 can transmit a signal corresponding to a command input through the user to the image display apparatus 100 or the external apparatus 30. [ The image display device 100 or the external device 30 can be controlled according to a signal transmitted through the augmented remote controller 200. [

The augmented remote control device 200 can recognize the image display device 100 or the external device 30 using a camera, an RFID reader, or the like. The augmented remote control apparatus 200 determines whether the content related information provided from the recognized image display apparatus 100 or the external apparatus 30 is provided through the metadata provided by the image display apparatus 100 or the external apparatus 30 can do. Also, the augmented remote control device 200 can search the network server 300 for the content-related information provided by the recognized image display device 100 or the external device 30.

The augmented remote control device 200 can display the content related information provided by the video display device 100 or the external device 30 on the display unit of the augmented remote control device 200. [ The content-related information may correspond to the type of content recognized by the augmented remote control device 200. [

For example, if the currently provided content is related to shopping, the augmented remote control device 200 can retrieve information related to the price of the item, the name of the product, the place of sale, and the online selling site related to the content. If the presently provided content is content related to a famous sightseeing spot, the augmented remote control device 200 can search for information related to the name of the sightseeing spot, a tourism product, a related photograph or a moving image related to the content. Also, if the currently provided content is a movie, the augmented remote control device 200 can retrieve information related to the movie producer, the producer, the actor, or the related movie. The user can set the type of information to be searched by the augmented remote control device 200 for each content type.

2 is a block diagram illustrating an augmented remote control system system according to an embodiment of the present invention.

The augmented remote control apparatus 200 of FIG. 2 transmits and receives signals directly to and from the image display apparatus 100 and the external apparatus 30 or the home network 40 to which the image display apparatus 100 or the external apparatus 30 is connected, Lt; / RTI > and home server 50 of FIG. The home network 40 is a network in which a predetermined number of video display devices 100 or external devices 30 can transmit and receive signals according to a predetermined network communication standard. The home network 40 may be an independent network that is distinguished from a network in which the network server 300 including the content provider 20 is constructed. The home network 40 may be built in-house or at home.

The home server 50 may store information related to the video display device 100 and the external device 30 connected to the home network 40. [ For example, the home server 50 may store information related to the product name, model name, usage method, provided content, etc. of the video display device 100 or the external device 30 connected to the home network 40. In addition, the home server 50 can control signal transmission / reception of the home network 40. [

The augmented remote control device 200 can recognize the type of the image display device 100 or the external device 30 connected to the home network 40 through the camera and the RFID reader. The augmented remote control apparatus 200 transmits the information about contents provided by the recognized image display apparatus 100, the external apparatus 30, the image display apparatus 100 or the external apparatus 30 directly to the image display apparatus 100. [ Or through the external device 30. Also, the remote controller 200 can receive the related information through the home server 50. Further, the augmented remote control device 200 can search for and receive related information through the network server 300. [

The augmented remote control device 200 can display the searched information on the display unit of the augmented remote control device 200. [ The augmentation information displayed on the augmented remote controller 200 may overlap an image photographed through a camera or an actual image projected on a transparent display unit. The augmenting remote controller 200 can display the augmented actual image in which the augmenting information is overlapped on the display unit.

3 is an internal block diagram of an image display apparatus according to an embodiment of the present invention.

3, an image display apparatus 100 according to an exemplary embodiment of the present invention includes a broadcast signal receiving unit 110, a network interface unit 120, an external device input / output unit 130, a remote control device interface unit 140 A control unit 150, a video / audio processing unit 170, a display unit 180, and an audio output unit 185.

The broadcast signal receiving unit 110 may receive an RF broadcast signal corresponding to a channel selected by the user or all pre-stored channels among RF (Radio Frequency) broadcast signals received through the antenna. The broadcast signal receiving unit 110 may convert the received RF broadcast signal into an intermediate frequency signal, a baseband image, or a voice signal, and output the converted signal to the video / audio processing unit 170.

In addition, the broadcast signal receiver 110 may receive an RF broadcast signal of a single carrier according to an Advanced Television System Committee (ATSC) scheme or an RF broadcast signal of a plurality of carriers according to a DVB (Digital Video Broadcasting) scheme. The broadcast signal receiving unit 110 sequentially selects RF broadcast signals of all broadcast channels stored through the channel memory function among the received RF broadcast signals, and converts the RF broadcast signals into an intermediate frequency signal, a baseband image, or a voice signal. This is to show a thumbnail list including a plurality of thumbnail images corresponding to broadcast channels on the display unit 180. [ Therefore, the broadcast signal receiving unit 110 can receive the RF broadcast signals of the selected channel or all the stored channels sequentially / periodically.

The network interface unit 120 provides an interface for connecting the video display device 100 to a wired / wireless network including the Internet network.

The network interface unit 120 may include a wireless communication unit capable of wirelessly connecting the video display device 100 to the Internet. For wireless Internet access, WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access)

The network interface unit 120 may receive content or data provided by a content provider or a network operator through a network. That is, it is possible to receive contents such as broadcasts, games, VOD, broadcasting signals, and the related information provided from a contents provider through a network. In addition, the update information and the update file of the firmware provided by the network operator can be received.

The external device input / output unit 130 can connect the external device 30 and the image display device 100. [ To this end, the external device input / output unit 130 may include an A / V input / output unit or a wireless communication unit.

The external device input / output unit 130 is connected to an external device 30 such as a digital versatile disk (DVD), a Blu ray, a game device, a camera, a camcorder, a computer (notebook) The external device input / output unit 130 transmits video, audio, or data signals input from the outside through the connected external device 30 to the video / audio processing unit 170 of the video display device 100. Also, the video / audio processing unit 170 can output the processed video, audio, or data signals to the connected external device 30.

The A / V input / output unit includes an Ethernet terminal, a USB terminal, a CVBS (Composite Video Banking Sync) terminal, a component terminal, and a video terminal so that the video and audio signals of the external device 30 can be input to the video display device 100. [ Video terminal (analog), a DVI (Digital Visual Interface) terminal, an HDMI (High Definition Multimedia Interface) terminal, an RGB terminal, a D-SUB terminal, and the like.

Further, the wireless communication unit can perform wireless communication with another external device. The video display device 100 is connected to other electronic devices in accordance with communication standards such as Bluetooth, Radio Frequency Identification (RFID), IrDA (Infrared Data Association), UWB (Ultra Wideband), ZigBee, Can be connected.

Also, the external device input / output unit 130 may be connected to various set-top boxes via at least one of the various terminals described above to perform input / output operations with the set-top box.

For example, when the set-top box is a set-top box for an IP (Internet Protocol) TV, the external device input / output unit 130 transmits video, audio, or data signals processed in the set- To the voice processing unit 170. Also, the video / audio processing unit 170 can transmit the processed signals to the IP TV set-top box.

Meanwhile, the IPTV may include ADSL-TV, VDSL-TV, FTTH-TV and the like depending on the type of the transmission network. The IPTV may include a TV over DSL, a video over DSL, a TV over IP BTV), and the like. In addition, IPTV may also mean an Internet TV capable of accessing the Internet, or a full browsing TV.

Also, the external device input / output unit 130 may be connected to a communication network capable of video or voice communication. The communication network may mean a broadcasting communication network, a public telephone network, a mobile communication network, etc. connected through a LAN.

The augmented remote control device interface unit 140 includes a wireless communication unit capable of wirelessly transmitting and receiving signals with the augmented remote control device 200 and a wireless communication unit capable of calculating the coordinates of a pointer corresponding to the operation of the augmented remote control device 200 And a coordinate calculation unit. The augmented remote control device interface unit 140 can wirelessly transmit and receive signals to and from the augmented remote control device 200 through the RF module. Also, the augmented remote control device 200 can receive a signal transmitted in accordance with the IR communication standard through the IR module.

The coordinate calculation unit of the augmented remote control unit interface unit 140 can correct the camera shake or error from the signal corresponding to the motion of the augmented remote control device 200 received through the wireless communication unit of the augmented remote control device interface unit 140 . The coordinate calculation unit may calculate the coordinates (x, y) of the pointer to be displayed on the display unit 180 of the image display device 100 after the camera shake or error is corrected.

The transmission signal of the augmented remote control device 200 inputted to the image display device 100 through the augmented remote control device interface unit 140 is outputted to the control unit 150 of the image display device 100. [ The control unit 150 can determine the information on the motion and the key operation of the augmented remote control apparatus 200 from the signals transmitted from the augmented remote control apparatus 200 and control the image display apparatus 100 in response to the information .

As another example, the augmented remote control device 200 may calculate pointer coordinates corresponding to the motion of the augmented remote control device 200 and output the coordinates to the augmented remote control device interface unit 140. [ In this case, the augmented remote control device interface unit 140 can transmit information about the coordinates of the received pointer to the control unit without any additional camera shake or error correction process.

The control unit 150 controls the overall operation of the image display apparatus 100. The control unit 150 can receive signals transmitted from the augmented remote control device 200 through the augmented remote control device interface unit 140. [ In addition, a command can be input through the local key provided in the image display apparatus 100. The controller 150 determines a command included in the received signal or a command corresponding to a local key operation, and controls the image display apparatus 100 to correspond to the command.

For example, when the user inputs a predetermined channel selection command, the control unit 150 controls the broadcast signal receiving unit 110 to receive the broadcast signal provided from the selected channel through the broadcast signal receiving unit 110. In addition, the video / audio processing unit 170 is controlled so that the video signal and the audio signal of the selected channel are processed. In addition, the control unit 170 controls the video / audio processing unit 170 so that channel information selected by the user can be output through the display unit 180 or the audio output unit 185 together with the video signal and the audio signal.

As another example, the user can input another kind of video or audio output command through the augmented remote control device 200. [ That is, the user may desire to watch a camera or camcorder image signal inputted through the external device input / output unit 130 instead of the broadcast signal. In this case, the control unit 150 controls the video / audio processing unit 170 to output the video or audio signal input through the USB input unit of the external device input / output unit 130 or the like through the display unit 180 or the audio output unit 185 Can be controlled.

The image display apparatus 100 may include a user interface control unit for generating a graphic user interface associated with the image display apparatus 100. The function of the user interface control unit can be handled by the control unit 150. [ In the present embodiment, the user interface control unit will be described as a separate component for the sake of convenience.

The graphic user interface generated through the user interface control unit is output to the display unit 180 or the sound output unit 185 through the video / audio processing unit 170. [ The graphic user interface generated by the user interface control unit includes a command included in the signal received through the augmented remote controller 200, a command input through the local key included in the image display apparatus 100, As shown in FIG.

For example, when a signal is received through the augmenting remote controller 200, the user interface controller generates a pointer image signal corresponding to the motion of the augmenting remote controller 200, 170). The controller 150 can output information on the pointer coordinates calculated from the signal received through the augmented remote controller 200 to the user interface controller. The user interface control unit can generate a pointer image signal based on the information about the pointer coordinates output from the control unit 150. [ The video / audio processing unit 170 processes the video signal output to the display unit 180 so that a pointer corresponding to the pointer video signal generated by the user interface control unit is displayed on the display unit 180. The pointer displayed on the display unit 180 corresponds to the movement of the augmented remote controller 200. [

As another example, the user interface control unit may include a command included in the signal received through the augmented remote controller 200, a command input through the local key, or a user interface composed of objects corresponding to the operation of the image display apparatus 100 And outputs the generated video signal to the video / audio processing unit 170. [

The object includes various kinds of widgets displayed on the display unit 180 so as to be able to input commands to the image display apparatus 100 or display information related to the image display apparatus 100. [ The widget can be displayed as an OSD (On Screen Display).

The object includes an image or text representing information about the image displayed on the image display apparatus 100 or the image display apparatus 100, such as a sound output (volume) level of the image display apparatus 100, channel information, do. The object may be displayed in the image display apparatus 100 or may be implemented in a form (ex) movie according to the kind of information to be displayed), and the object of the present embodiment does not limit the scope of rights of the present invention.

A widget is a graphical user interface (GUI) that allows a user to directly change specific data. For example, the widget may be one of a volume control button, a channel control button, a menu, an icon, a navigation tab, a scroll bar, a progress bar, a text box, and a window displayed on the display unit 180 of the image display apparatus 100 . The type of the widget implemented in the image display apparatus 100 may be changed in accordance with the specific specification of the GUI that can be implemented or implemented in the image display apparatus 100 and the widget of the present embodiment does not limit the scope of rights of the present invention No.

The video / audio processing unit 170 receives video signals or audio signals included in signals received through the broadcast signal receiving unit 110, the network interface unit 120, and the external device input / output unit 130, . The video / audio processing unit 170 may process the video signal or the audio signal based on the information included in the data signal received together with the video signal or the audio signal.

The video / audio processing unit 170 processes the video signal or the audio signal inputted through the user interface control unit so that the video signal or the audio signal can be output from the display unit 180 or the sound output unit 185. The user can grasp the operation state of the image display apparatus 100 using the graphic user interface displayed on the display unit 180 based on the image signal or the voice signal generated by the user interface control unit, You can enter commands.

The video / audio processing unit 170 can select a video signal or a voice signal to be processed in response to a user command input through the control unit 150. The video signal or audio signal processed by the video / audio processing unit 170 is a video or audio signal output through the display unit 180 or the audio output unit 185. The command input through the control unit 150 may include a broadcast channel selection command, a content selection command to be reproduced among the contents input to the image display apparatus 100, and the like.

The video / audio processing unit 170 of the present embodiment can process a video signal so that a 2D or 3D video signal input from the outside can be displayed on the display unit 180. [ The video / audio processing unit 170 can process the video signal so that the user interface generated through the user interface control unit is three-dimensionally displayed on the display unit 180. A detailed description of the video / audio processing unit 170 will be described later with reference to FIG.

The display unit 180 displays a video signal, a data signal, an OSD signal processed by the video / audio processing unit 170, a video signal and a data signal received through the external device input / output unit 130 as R, G, B Signal to generate a driving signal. The display unit 180 may be a PDP, an LCD, an OLED, or a flexible display. In this embodiment, the display unit 180 preferably has a 3D display.

The three-dimensional display can be divided into an additional display method and a single display method according to a method of recognizing a three-dimensional image by a user.

In the single display method, a 3D image can be implemented independently on a display without a separate auxiliary device. A user who views a display using a single display method can view a three-dimensional image even without the additional apparatus (ex) flat glasses). The single display method may be a lenticular method, a parallax barrier, or the like.

The additional display method is a method of implementing a 3D image using an auxiliary device. Additional display methods include a head-mounted display (HMD) system, a glasses system, and the like. Further, polarized glasses, shutter glasses, spectral filters, and the like can be applied to the glasses used in the glasses system.

Meanwhile, the display unit 180 may be configured as a touch screen and used as an input device in addition to the output device.

The audio output unit 185 receives a signal processed by the video / audio processing unit 170, for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs it as a voice. The sound output unit 185 may be implemented by various types of speakers.

4 is an internal block diagram of the video / audio processing unit 170 of the video display device according to the embodiment of the present invention.

The video / audio processing unit 170 may include a demodulator 171, a demultiplexer 172, a decoder 173, and a formatter 175.

The demodulator 171 may demodulate the broadcast signal received by the broadcast signal receiver 110. For example, the demodulator 171 may receive the digital IF signal DIF converted by the broadcast signal receiver 110 and perform a demodulation operation. The demodulator 171 may perform channel decoding. For this, the demodulator 171 may include a convolution decoder, a deinterleaver, and a reed-solomon decoder to perform convolutional decoding, deinterleaving, and reed solomon decoding.

The demodulator 171 can demodulate and decode the channel and output the stream signal TS. The stream signal may be a signal in which a video signal, a voice signal, or a data signal is multiplexed. For example, the stream signal may be an MPEG-2 TS (Transport Stream) multiplexed with an MPEG-2 standard video signal, a Dolby AC-3 standard audio signal, or the like. Specifically, the MPEG-2 TS may include a header of 4 bytes and a payload of 184 bytes.

The demodulator 171 may be separately provided according to the ATSC scheme and the DVB scheme.

The stream signal output from the demodulator 171 may be input to the demultiplexer 172.

The demultiplexer 172 demultiplexes the received stream signal, for example, the MPEG-2 TS into a video signal, a voice signal, and a data signal. The stream signal input to the demultiplexer 172 may be a stream signal output from the demodulator 171, the network interface unit 120, and the external device input / output unit 130.

The demultiplexed data signal may be an encoded data signal. The encoded data signal may include EPG (Electronic Program Guide) information including broadcast information such as a start time and an end time of a broadcast program broadcasted on each channel. For example, the EPG information may be ATSC-PSIP (ATSC-Program and System Information Protocol) information in the case of the ATSC scheme and DVB-Service Information (DVB-SI) information in case of the DVB scheme .

The decoder unit 173 can decode the demultiplexed signal. The decoder unit 173 of the present embodiment may include a video decoder for decoding the demultiplexed video signal and a scaler for adjusting the resolution of the decoded video signal to be output from the video display device 100.

The video / audio processing unit 170 according to an exemplary embodiment of the present invention may include a mixer for mixing a video signal input from the outside to the video display apparatus 100 and a video signal generated from the user interface control unit 160 have. The formatter 175 may be responsible for the function of the mixer. In the present embodiment, the mixer will be described separately from the formatter 175 for the sake of convenience. The display unit 180 displays an image based on the mixed video signal. The video signal mixed in the mixer is output to the formatter 175.

The formatter 175 can determine the format of the mixed video signal by referring to the data signal related to the video signal. The formatter 175 may convert the mixed video signal into a format suitable for the display unit 180 and output the formatted video signal to the display unit 180.

The video display device 100 of this embodiment can display a 3D image on the display unit 180. [ The formatter 175 may generate a 3D image signal according to a predetermined format by separating the mixed image signal into a plurality of view image. The formatter 175 outputs the generated 3D video signal to the display unit 180. The display unit 180 displays a 3D image based on the generated 3D image signal.

The 3D image is composed of a plurality of viewpoint images. The user can view the multiple view image through the left eye and the right eye. The user can feel the stereoscopic effect of the 3D image through the difference of the image viewed through the left eye and the right eye. A multi-view image for realizing a 3D image is generally composed of a left eye image that the user can recognize through the left eye and a right eye image that can be recognized through the right eye.

The 3D image signal format may be determined according to a method of arranging the left eye image and the right eye image generated to implement the 3D image. The manner in which the left eye image and the right eye image are arranged left and right is called a side by side format. The method of arranging the left eye image and the right eye image up and down is called a top / down format. The method of arranging the left eye image and the right eye image in a time-division manner is called a frame sequential format. The method of mixing the left eye image and the right eye image line by line is referred to as an interlaced format. A method of mixing the left eye image and the right eye image on a box basis is called a Checker Box format.

A video signal included in a signal input from the outside to the image display apparatus 100 may be a 3D image signal capable of implementing a 3D image. Also, the graphic user interface video signal generated by the user interface control unit 160 may be a 3D video signal capable of implementing 3D video. The mixer can mix the 3D image signal included in the signal input from the outside to the image display apparatus 100 and the 3D image signal generated by the user interface control unit, and output the mixed image to the formatter 175.

The formatter 175 can determine the format of the mixed 3D video signal by referring to the related data signal. The formatter 175 may process the 3D video signal to fit the determined format and output the processed 3D video signal to the display unit 180. When the 3D image signal format that can be output from the display unit 180 is limited, the formatter 175 converts the received 3D image signal into a 3D image signal format that can be output from the display unit 180 And output it to the display unit 180.

If the format can not be determined from the data signal of the mixing 3D video signal input to the formatter 175, the formatter 175 can determine the format of the 3D video signal input using a predetermined algorithm. For example, the formatter 175 may determine the format of a 3D video signal by analyzing the edge of an image generated based on the input 3D video signal.

In addition, when the mixing video signal input to the formatter 175 is a 2D video signal capable of implementing a 2D video, the formatter 175 can generate a 3D video signal through an algorithm for converting the 2D video to the 3D video.

5 is an internal block diagram of an augmented remote controller 200 according to an embodiment of the present invention. The augmenting remote controller 200 includes a wireless communication unit 210, a storage unit 220, a user interface unit 230, a sensor unit 240, an identification unit 250, a display unit 260, ), And a control unit 280.

The wireless communication unit 210 can transmit and receive signals to and from an electronic device such as the video display device 100, the external device 30, the home server 50, or the network server 300. In the present embodiment, the augmented remote control device 200 may include an RF module capable of transmitting and receiving a signal with a peripheral device according to an RF communication standard. In addition, the augmented remote control device 200 may include an IR module capable of transmitting and receiving a signal with a peripheral device according to an IR communication standard.

In addition to the wireless communication unit 210, the wireless communication unit 210 has a module capable of transmitting / receiving signals to / from an electronic device according to various communication standards and transmitting / receiving signals according to the communication standard. Examples of wireless communication standards for signals that can be transmitted and received through the wireless communication unit 210 of the augmented remote control device 200 include Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), UWB (Ultra Wideband), and ZigBee (ZigBee).

The wireless communication unit 210 can transmit and receive signals to and from the Internet network in accordance with various types of wireless Internet standards and has a module suitable for transmitting and receiving signals according to the wireless Internet standard. The wireless Internet standard that can be used in the augmented remote control apparatus 200 includes a WLAN (Wireless LAN) (Wi-Fi), a Wibro (Wireless broadband), a Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) Can be.

In the present embodiment, the augmented remote control device 200 transmits a signal containing information on the operation of the augmented remote control device 200 to the electronic device through the wireless communication unit 210. [ Also, the augmented remote control device 200 can receive the signal transmitted by the electronic device through the RF module. In addition, the augmented remote control device 200 can transmit commands to the electronic device via the IR module, such as power on / off, channel change, volume change, and the like, as needed.

The storage unit 220 may store various kinds of programs, application data, and the like necessary for control or operation of the augmented remote control device 200. [ If the augmented remote control device 200 wirelessly transmits and receives a signal through an electronic device and an RF module, the augmented remote control device 200 and the electronic device transmit and receive signals through a predetermined frequency band. The control unit 280 of the augmented remote control device 200 can store information about a frequency band and the like capable of wirelessly transmitting and receiving a signal with a peripheral device paired with the augmented remote control device 200 in the storage unit 220, have.

The user interface unit 230 may include a keypad or a button. The user can input a command related to the image display apparatus 100 to the augmented remote control apparatus 200 by operating the user interface unit 230. [ When the user interface unit 230 includes a hard key button, the user can input a command related to the electronic device to the augmented remote control device 200 through the push operation of the hard key button. When the user interface unit 230 includes a touch screen, the user can input a command related to the electronic device to the augmenting remote controller 200 by touching the soft key of the touch screen. Also, the user interface unit 230 may include various types of input means such as a scroll key, a jog key, and the like that can be operated by the user.

The sensor unit 240 includes a sensor that can be used to collect user-related information using the augmented remote control device 200. [ The sensor unit 240 may include a GPS, a compass, a gyro sensor, an acceleration sensor, an infrared sensor, and the like. GPS can be used to determine the location of the user. The compass can be used to determine the direction of the user.

The gyro sensor may be used to determine information about the motion of the augmented remote control device 200 operated by the user or the user. For example, the gyro sensor may sense information about the motion of the augmented remote control device 200 based on x, y, and z axes. The acceleration sensor 243 can sense information on the moving speed and the like of the spatial remote controller.

The augmented remote control device 200 can refer to the user related information collected through the sensor unit 240 and use it to identify an object existing in the user's surrounding environment. Also, the augmented remote control device 200 can determine the user gesture by referring to the user-related information collected through the sensor unit 240. [ The augmented remote control apparatus 200 can be controlled according to a command corresponding to the user gesture. Also, if the command corresponding to the user gesture is an electronic device control command, the augmented remote control device 200 can transmit a signal including the corresponding control command to the electronic device.

The identification unit 250 may identify a user peripheral object including the electronic device. The identification unit 250 may include a camera, an RFID reader, an infrared sensor, or the like. The identification unit 250 can photograph a user's peripheral seam using a camera. The image of the object photographed through the identification unit 250 may be compared with images of various objects stored in the storage unit 220, the home server 50, or the network server 300. [ The identifying unit 250 may analyze the pattern of the image and recognize information about the object corresponding to the image having the same pattern as the pattern of the captured image to recognize the photographed object.

In addition, the identification unit 250 can read the RFID-related information attached to the object through the RFID reader and recognize the object. In addition, the identification unit 250 can determine whether an object exists around the user through the infrared sensor. The augmented remote control device 200 can refer to information about an object that exists for each user location or direction. Information about an object existing according to a user location or direction may be stored in the storage unit 220, the home server 50, or the network server 300.

The identification unit 250 determines the location and direction of the current user on the basis of the user-related information collected through the sensor unit 240, identifies the position of the object detected through the infrared sensor among the information on the stored object And direction of the object, and recognize the object existing around the user. For example, the augmented remote control device 200 may refer to map information stored with information on a building located by a predetermined user location and direction. In this case, the augmented remote control device 200 can recognize the actual user's neighboring building by referring to the map information on the information of the surrounding objects corresponding to the user's location and direction possessing the augmented remote control device 200. [

Also, the identification unit 250 can recognize the face, fingerprint, iris, and the like of a person who has photographed through the camera. The identification unit 250 can recognize a person who has photographed through the augmented remote control device 200 by comparing the recognized face, fingerprint, and iris patterns with stored face images, fingerprints, and iris patterns. The control unit 280 can retrieve and output the content related information recognized through the identification unit 250. [ For example, when a person is recognized through the identification unit 250, the user can retrieve and output the recognized person-related information (ex) name, age, individual preferred content, etc.).

The display unit 260 and the sound output unit 270 may correspond to the operation of the user interface unit 230 or may correspond to the operations of the image display apparatus 100, the external apparatus 30, the home server 50, the network server 300, It is possible to output video or audio corresponding to the signal transmitted from the device. The user can recognize whether the user interface unit 230 is operated or whether the electronic device is controlled through the display unit 260 and the sound output unit 270.

The display unit 260 may display object related information included in the image captured through the camera. The display unit 260 may display an augmented real image that overlaps the augmented information searched on an image photographed through the camera. As another example, when the display unit 260 is a transparent display unit including a transparent panel, the user can view the actual image of the surrounding environment through the transparent display unit. The augmented remote control apparatus 200 can retrieve object related information included in the actual image being projected on the transparent display unit. The augmented remote control device 200 can display information about the object searched in the transparent display unit.

The control unit 280 may overlap the enhancement information with an image photographed through the camera and output the enhancement information to the display unit 260. The control unit 280 outputs the image signal related to the enhancement information to the display unit 260 so that the enhancement information related to the object included in the image projected on the transparent display unit overlaps with the actual image currently projected on the transparent display unit . In addition, the control unit 280 controls various matters related to the control of the augmented remote control device 200.

The control unit 280 transmits a signal corresponding to the predetermined key operation of the user interface unit 230 or a signal corresponding to the operation of the augmented remote control device 200 sensed by the sensor unit 240 through the wireless communication unit 210 It can be transmitted to an electronic device.

The block diagrams of the image display apparatus 100 and the augmented remote control apparatus 200 shown in Figs. 3 to 5 are block diagrams for an embodiment of the present invention. Each component of the block diagram can be integrated, added, or omitted according to specifications of the video display apparatus 100 and the augmented remote control apparatus 200 actually implemented. That is, two or more constituent elements may be combined into one constituent element, or one constituent element may be constituted by two or more constituent elements, if necessary. In addition, the functions performed in each block are intended to illustrate the embodiments of the present invention, and the specific operations and apparatuses do not limit the scope of the present invention.

6 is a flowchart illustrating a flow of an operation method of the augmented remote controller 200 according to an embodiment of the present invention.

The augmented remote controller 200 of the present embodiment can recognize a user's peripheral object (S10). The augmented remote control device 200 can retrieve the recognized object related information (S20). The augmented remote controller 200 can display the retrieved object related information (S30).

There are various kinds of user peripheral objects that the augmented remote control device 200 can recognize. The user can select the type of object to be recognized through the augmented remote control device 200. [ For example, the user may select to recognize the electronic device in the augmented remote control device 200. [ In this case, the augmented remote control device 200 can recognize the electronic devices existing around the user by using the RFID attached to the electronic device.

The user can also select to recognize the content provided by the video display device 100 or the external device 30. [ In this case, the augmented remote control apparatus 200 can control the current image display apparatus 100 or the external apparatus 30 using the metadata provided from the image display apparatus 100, the external apparatus 30 or the home server 50, It is possible to recognize the contents provided by the user. That is, when the user is positioned such that the identification means such as a camera or an RFID reader mounted on the augmented remote control device 200 faces the video display device 100 in which the content is currently being reproduced, the augmented- The content that the current user wants to identify can be recognized by the current user and the content being played back in the video display device 100 can be recognized using the relevant electronic device related metadata.

The user can also choose to recognize a particular person. In this case, the augmented remote controller 200 can read the face, fingerprint, iris, and the like of a specific person through the camera, the fingerprint reader, and the iris reader in the identification unit 250. The augmented remote control device 200 can recognize the identity of a specific person photographed through the augmented remote control device 200 by comparing the information about the person's face, fingerprint, iris, etc. stored in the database with the read information.

The user can also choose to recognize nearby objects (ex) buildings, furniture, etc.). In this case, the augmented remote control device 200 can collect user related information related to the user's location or direction using a GPS or a compass. In addition, the augmented remote control device 200 can photograph an image of a surrounding object using a camera. The augmented remote control apparatus 200 can recognize the captured peripheral object by referring to the image information of the peripheral object corresponding to the current user position or direction among the databases storing the object information per user position and direction.

The information referred to by the augmented remote control device 200 in the process of recognizing the object may be transmitted to the image display device 100, the external device 30, the storage unit 220 of the augmented remote control device 200, the home server 50, Or may be stored in network server 300. The augmented remote control apparatus 200 may transmit the recognized object related information to the image display apparatus 100, the external apparatus 30, the storage unit 220 of the augmented remote control apparatus 200, the home server 50, And can be retrieved from the server 300.

For example, when the augmenting remote controller 200 recognizes the video display device 100 or the external device 30, the augmenting remote controller 200 can search the list of contents being played back in the corresponding electronic device. The content list stored in the video display device 100 or the external device 30 can also be searched. In addition, it is possible to retrieve broadcast signal related information received by the video display device 100. [ It is also possible to search for information including a menu for controlling the electronic apparatus such as the video display apparatus 100 or the external apparatus 30. [

In addition, the augmented remote control apparatus 200 can retrieve content related information when the image display apparatus 100 or the external apparatus 30 recognizes the content being reproduced or stored. The content-related information may include title information, shopping information, and the like of the content.

Further, when the augmenting remote controller 200 recognizes a specific person, the augmenting remote controller 200 can retrieve information related to the specific person. Information related to a specific person may include information about a specific person's name, age, occupation, contact information, or the content preferred by a specific person or the history of a content viewed by a specific person.

In addition, the augmented remote control device 200 can retrieve information related to an actual object when recognizing a real object (ex) building, furniture, artwork, etc. of the user's surroundings). Information related to an actual object may include the name of an actual object, a maker, a selling price, a selling place, a usage method, and the like.

The augmented remote control device 200 can display the searched information on the display unit 260. [ The augmented remote control device 200 can display the searched information by using a pop-up window or an icon. Also, an image or text corresponding to the retrieved information can be displayed. The augmented remote control apparatus 200 can display the enhancement information searched on the image photographed through the camera on the display unit 260 in an overlapping manner.

At this time, the control unit 280 may configure a screen displayed on the display unit 260 so that the object corresponding to the information retrieved from objects included in the surrounding environment does not overlap with the retrieved information. The controller 280 may configure a screen to be displayed on the display unit 260 so that the augmenting information is displayed in an area not overlapping with the object corresponding to the searched information.

7 is a view schematically showing an appearance of an augmented remote control device 200 according to an embodiment of the present invention.

The screen 261 displayed on the display unit 260 of the augmenting remote controller 200 displays the object 1001 recognized by the augmented remote controller 200 and the information 1002 and 1003 related to the recognized object 1001 .

The object 1001 recognized by the augmented remote control device 200 in the present embodiment is an image of an object actually photographed through the camera of the augmented remote control device 200. [ The related information 1002 and 1003 are enhancement information composed of information retrieved by the augmented remote control device 200. [ As shown in the figure, the augmenting remote controller 200 displays the augmenting information on the screen 261 together with the image of the object to be actually photographed. The user can confirm the information about the object to be actually photographed through the augmentation information included in the screen 261. [

The object-related information may include a menu for controlling the object. In this embodiment, the related information 2 1003 may be a menu capable of inputting related commands to the video display device 100 and the external device 30. Also, in this embodiment, the related information 2 1003 may be a menu capable of interrupting the reproduction of the currently reproduced content or transmitting it to another electronic apparatus.

When the screen 261 is displayed on the touch screen, the user can touch the touch screen to display the information related to the object 1001 on the screen 261. Also, a predetermined icon among the menus included in the related information 2 (1003) displayed on the screen (261) can be selected and an instruction corresponding to the selected icon can be inputted to the augmented remote control device (200).

The augmented remote control apparatus 200 may include a keypad 231. [ The user can input a predetermined command to the augmented remote controller 200 by operating a predetermined key constituting the keypad 231. [

8 is a diagram illustrating an augmented remote control apparatus and an operation method thereof according to an embodiment of the present invention.

According to the embodiment described with reference to FIG. 8, the augmented remote control device recognizes one or more external devices connected to the image display device or the home network through the identification part 250 (S810). The process of recognizing an external device is usually performed in conjunction with a process in which the augmented remote control device shoots the external device and generates a preview image. Accordingly, the augmented remote control device includes an identification unit 250 such as a module having a function of recognizing a subject and generating a preview image thereon and an image capturing and storing function, a display unit 250 capable of displaying and storing the preview image or the stored image in real time, A portion 260 is mounted.

As described above, the identification unit 250 of the augmented remote control device can recognize the external device through the RFID or the like. The augmented remote control device reads or acquires external device identification information such as a serial number or a product code of the external device through various methods. Alternatively, the image data generated by photographing the external device may be used for external device identification. Here, the image data generated by photographing an external device may be at least one of preview image data that is streamed or image data that is stored after capture.

The augmented remote control device transmits image or image data of the external device through the camera included in the identification part 250 and all information including various identification information of the external device read or received from the external device to the external device Quot; information " The wireless communication unit 210 of the augmented remote control device transmits information on the recognized external device to the video display device through the above-described process (S820).

For example, in the case of the augmented remote control apparatus of the present embodiment, after the identification unit 250 photographs an external device to which a manual is to be downloaded, the external device corresponding to the image photographed by the wireless communication unit 210 or the control unit 280 You can search directly. In this case, the wireless communication unit 210 connects to a network server or a home server, searches for an external device, and transmits information about the external device to the video display device. An image photographed in the search process can be used. Thereafter, the image display apparatus downloads control interface information or manual information corresponding to the external apparatus using information on the external apparatus received from the augmented remote control apparatus. The video display device may display the GUI related to the manual directly on the display unit 180 of the video display device using the control interface information or the manual information, or may transmit the control interface information or the manual information to the augmented remote control device.

As another example, when the RFID remote control device reads the serial number, the manufacturer name, the product name, the product code, the model name, etc. of the external device to which the RFID is attached using the RFID reader of the identification unit 250, Device related information can be determined. The augmented remote control device recognizes an external device to download a manual based on the information about the external device, and can transmit information on the recognized external device to the video display device. The identification unit 250 of the augmented remote control device can identify the information on the external device and recognize the corresponding electronic device using a code according to a communication standard other than the RFID reader.

As another example, the augmented remote control device can check the list of external devices connected to the home network by connecting to the home server of the home network to which the external device is connected or connectable through the wireless communication unit 210. [ If an external device to receive a manual is included in the list of external devices through the wireless communication unit 210, the control unit 280 of the augmented remote control device downloads information on the external device, Device. Then, the video display device uses the information about the external device in searching the control interface information or the manual information of the external device in the control interface database.

Upon receiving the information about the external device, the video display device inquires whether the external device has a history of connection with the augmented remote control device or the video display device. If there is a connection history, control interface information or manual information for the external device is already stored, so there is no need to connect to the web server and download the control interface information or the manual information newly.

Here, the control interface information is information necessary for a user to create a user interface to be provided in controlling an external device using the augmented remote control device. In particular, the control interface information may include manual information of the external device. The manual information is information corresponding to the operation method necessary for using a specific external device, the instruction manual of the external device, or a method of handling a failure. That is, the image display apparatus or the augmented remote control apparatus generates the GUI related to the manual using the control interface information and the manual information included therein.

The control unit 280 of the augmented remote control apparatus or the control unit 150 or the video / audio processing unit 170 of the video display apparatus can generate the GUI related to the manual using the control information or the manual information included in the control interface information . The GUI related to the manual is a user interface image provided so that a user can input a user signal for control when the user intends to control the external device using the augmented remote control device. Manual-related GUIs can be displayed in the form of an OSD. Accordingly, the GUI associated with the manual can display, for example, text and image data on the operation method of each function of the external device, a virtual keypad to be pressed for the operation, and the like.

If it is determined that the external device has not been connected before, the video display device downloads the control interface information corresponding to the external device. In order to download necessary control interface information, the network interface unit 120 of the video display device connects to the web server, and the control unit 150 of the video display device searches the control interface database for control interface information corresponding to the external device And can be selected. If there is a connection history with the external device and the control interface information is stored in the video display device or the home server, the video display device transmits the previously stored control interface information to the augmented remote control device, You can create a manual GUI.

When the video display device searches the control interface information or the manual information, the controller 150 uses the information about the external device received from the augmented remote control device for searching. The control interface database is a database storing control interfaces of external devices connectable to the video display device and / or the augmented remote control device.

The control interface database may store manual related information of each external device. When the video display device downloads the control interface information from the control interface database, the augmented remote control device receives the control interface information and the like from the video display device through the wireless communication unit 210 (S830). Of course, the control interface information may include manual information on the external device.

According to another embodiment, before the augmented remote control device transmits information on the external device to the video display device, the control unit 280 of the augmented remote control device inquires whether there is a previous connection history with the external device, The information on the external device may be transmitted to the image display device only when there is no history. In the embodiment described with reference to FIG. 8, when the augmented remote control device transmits information on the external device to the video display device, the video display device determines whether or not the video display device has the previous connection history, To the Web server. That is, there is a difference in whether or not the connection history inquiry is performed in an augmented remote control device or an image display device.

The augmented remote controller displays the GUI associated with the manual on the display unit 260 of the augmented remote controller using the received control interface information (S840). Or the augmented remote controller may display the control interface information on the display unit 180 of the image display apparatus.

The user can receive other information such as a method of using the external device, a product specification, a countermeasure in case of failure, etc., through the GUI related GUI displayed on the display unit 260 of the image display device or the augmented remote control device .

9 is a flowchart illustrating an augmented remote control apparatus and an operation method thereof according to another embodiment of the present invention.

The augmented remote control device recognizes one or more external devices (S910). The identification unit 250 of the augmented remote control device acquires information that can identify the external device in the recognition process of the external device. As described above, the information that can identify the external device is information that enables identification of an external device such as an image of an external device or an external device such as a serial number, a manufacturer name, a product name, a product code, and a model name.

The augmented remote control device can recognize an external device by shooting an external device through a camera and comparing the captured image with an image related to an external device stored in the database. Also, in case of an external device with an RFID, the augmented remote control device can recognize the external device by reading the RFID attached to the external device through the RFID reader. Also, the augmented remote control device may recognize the external device through the home server.

The control unit 280 of the augmented remote control apparatus determines whether there is a previously connected history with the external device using information on the external device (S920). As a result, if there is a connection history, the control unit 280 of the augmented remote control device generates a GUI related to the manual using the previously stored control interface information or previously stored manual information in the augmented remote control device, (Step S950).

In the case of an external device having no previously connected history, the augmented remote control device accesses the network server through the wireless communication unit 210 and searches the control interface database (S930). If there is control interface information or manual information corresponding to the external device among the information stored in the control interface database, the augmenting remote controller downloads the corresponding control interface information or manual information (S940). Then, the control unit 280 generates a GUI related to the manual using the downloaded information and displays it on the display unit 260 (S950).

The augmented remote control device can reproduce the manual related GUI from a specific electronic device irrespective of the place where the control interface information or the manual information of the recognized external device is stored. Here, the electronic device to reproduce and display the GUI associated with the manual can be designated in advance by the user. Accordingly, when the augmented remote control device can be connected to a plurality of image display devices, the augmented remote control device may cause the first external device related manual stored in the first image display device to be reproduced in the second image display device , And may be displayed through the display unit 260 of the augmented remote control device.

In this case, the user can photograph the first external device through the camera provided in the identification unit 250 of the augmented remote control device. The augmented remote control device can recognize the first external device photographed through the camera. The augmented remote control device can search the recognized first external device related manual. The augmented remote control device which confirms that the first external device related manual is stored in the first image display device can display a thumbnail corresponding to the manual stored in the first image display device on the display unit of the augmented remote control device . The user may touch the thumbnail and drag the thumbnail to the second image display device so that the manual corresponding to the thumbnail is displayed on the second image display device.

10 is a flowchart illustrating an operation method of the augmented remote control apparatus according to another embodiment of the present invention.

The augmented remote controller recognizes the external device through the identification unit 250 (S1010). As described above, an RFID reader or a camera may be used for recognition of an external device. The identification unit 250 recognizes or acquires information on the external device while recognizing the external device. The information on the external device may be image data generated by photographing the external device as described above, or information such as a serial number and product code that can identify the external device as information read from the external device.

Then, the control unit 280 of the augmented remote control device inquires or determines whether there is a previous history of connection with the external device (S1020). If there is a connection history with the external device, the control unit 280 inquires whether there is previously stored control interface information with respect to the external device (S1030). If there is no previous connection with the external device, the control unit 280 of the augmented remote control device transmits information about the external device to the image display device through the wireless communication unit 210 (50). The control interface information for the external device must be newly obtained through the video display device.

The video display device downloads control interface information for an external device by connecting to a network server or connecting to a home server and searching the control interface database. If the control interface information is already stored in the storage unit of the image display apparatus, the server connection and / or control interface information search process may be omitted. The image display apparatus transmits control interface information for the external device to the augmented remote controller through the augmented remote controller interface unit 140 and receives the control interface information from the augmented remote controller through the wireless communication unit 210 of the augmented remote controller (S1070) .

If there is no pre-stored control interface information associated with the external device, the augmented remote control device may request control interface information from the augmented remote control device (S1060). The control unit 280 generates a control signal for requesting the control interface information and the wireless communication unit 210 may transmit the control signal to the augmented remote control unit interface unit 140 of the image display device. Alternatively, the control interface information request may be replaced by transmitting information (video data or identification information) about the external device. Also in this case, the augmented remote control device receives the control interface information from the image display device (S1070).

The control unit 280 of the augmented remote control device generates a GUI associated with the corresponding external device using the received control interface information (S1080). When there is data in the video display device capable of displaying a GUI related to the external device, the augmented remote control device can receive and use the data.

If there is pre-stored control interface information in the augmented remote control device, the control unit 280 may also determine whether there is a GUI related to the manual previously created and stored using the control interface information (S1040). If the GUI related to the manual is already generated and stored in the external device, the process of transmitting the information about the external device to the video display device separately or requesting the control interface information and generating the GUI related to the manual is omitted . Accordingly, if the GUI related to the external device is previously stored, but the GUI related to the manual is not generated, the control unit 280 of the augmented remote control device generates a GUI related to the GUI (S1080)

The display unit 260 of the augmented remote controller displays the created GUI related to the manual (S1090). The GUI associated with the manual can be displayed together with the external device, and is not necessarily displayed on the display unit 260 of the augmented remote control device, but may be displayed through an image display device or other electronic device having a video display function You may.

FIGS. 11 to 13D are views showing a GUI associated with a manual according to an operation method of the augmented remote control apparatus according to an embodiment of the present invention.

11, an external device 30, an image display device 100, and an augmented remote control device 200 are shown. The augmented remote control device tries to display the GUI related to the external device 30 by the augmented reality technology.

11, the GUI 1110 related to the manual is displayed on the display unit 260 of the augmented remote controller 200 together with the external device image 1101. The user may obtain more detailed information related to the manual by entering the desired number through the keypad. That is, if the user selects 2, information on the use method of the external device 30 is selected through the manual-related GUI 1110, and information on the method of diagnosing the failure of the external device 30 is selected, If you select 4, you will be provided with information about the repair method and A / S center in case of trouble.

Referring to FIG. 12A, the user has selected two 'usage methods'. That is, if the user selects the number 2 and number 1 in turn, the manual related GUI can display the manual about the reproduction method among the usage methods of the external device 30 in more detail. In another embodiment, a user interface image capable of actually controlling the external device 30 may be provided in the manual-related GUI. That is, the manual-related GUI 1110 not only provides information on the playback method among the methods of using the external device 30, but also provides an interface with which the external device 30 can play the content immediately when a specific button is pressed It is possible.

12A, the user selects the 'use method' twice. In the next step, a manual related GUI 1111 is displayed to guide the user in more detail about the usage method as shown in FIG. 12B. Referring to FIG. 12B, the user selects an item regarding 'subtitle setting' in the 'usage method'. Accordingly, referring to FIG. 12C, a manual-related GUI 1112 related to the subtitle setting is displayed. The manual related GUI 1112 related to the caption setting includes a method for setting a caption, a command to be displayed or not to display a caption, a language of a caption provided, a method for selecting a caption language, and the like Can be displayed. Referring to FIG. 12C, in the manual-related GUI 1112 related to the subtitle setting, an option (eg, a subtitle setting page) for enabling selection of whether to directly move to a screen for setting a subtitle or to continue reading a usage description including a subtitle setting Quot; 1113 ", and " no usage explanatory view " 1114) are provided. If the user immediately reproduces the content through the external device 30 and desires to set the caption for the content, the user can select 'Yes, to the caption setting page' 1113. Alternatively, if it is not necessary to immediately set the subtitles, or if further manual information other than the subtitle setting is to be obtained, the user can select 'No, Continue to use explanation' 1114.

As with the manual information on how to use it, manual information on troubleshooting and troubleshooting can also be provided in the same way. Referring to FIGS. 13A to 13C, manual related information related to the failure coping method is provided through the display unit 260 of the augmented remote remover 200. Referring to FIG. 13A, the user selects 'trouble diagnosis' 3 of the manual related GUI 1110 displayed on the display unit 260.

13B, a manual-related GUI 1115 for the failure diagnosis method is displayed on the display unit 260 of the augmented remote control device 260. As shown in Fig. The menu for 'fault diagnosis' shows various fault conditions that can occur when using the external device 30, such as (1) playback error, (2) sound error, (3) screen error, (4) The user can select necessary information from the information on each failure situation.

For example, when the user selects '(3) screen abnormality' as shown in FIG. 13C, a manual-related GUI 1116 that provides more detailed information about coping or repairing a screen abnormality may be displayed in the next step . In the manual related GUI 1116 for the screen abnormality, the user can select whether to immediately move to the repair method page. 'Do you want to go to the repair method page for "screen error"? A user who wants to receive a repair method immediately and a screen abnormality occurs at present, selects 'Yes' 1117. Alternatively, if the user does not need to be immediately provided with a repair method for screen anomalies, that is, the user wishes to keep the current page or see more manual related information, the user can select 'No' 1118.

If the user selects "Yes" 1117, the next step is to go to the repair method page for "screen error". Referring to FIG. 13D, a manual related GUI 1119 is provided to specifically guide a repair method for a screen abnormality. That is, when a failure occurs in which the screen does not normally appear during the reproduction of the content, the user inputs troubleshooting information corresponding to the corresponding situation through the GUI 1119 related to the external device 30 displayed on the augmented remote control device 200 Will be provided. Therefore, even if there is no booklet on which the manual-related information about the external device 30 is written, the manual-related information can be immediately provided. At the same time, the user can control or repair the external device 30 or the image display device 100 through the augmented remote control device 200 according to the provided manual related information.

FIGS. 14A to 14C are diagrams illustrating a GUI associated with a manual according to an operation method of the augmented remote control apparatus according to another embodiment of the present invention.

14A to 14C, while the external device image 1101 is displayed on the display unit 260 of the augmented remote control device 200, the GUI 1210 related to the manual is displayed on the display unit 100 of the image display device 100, (180). The external device image 1101 and the product name of the external device may be displayed together on the display unit 260 of the augmented remote control device 200 together. In this case, the user inputs a user signal through the augmented remote control device 200 according to the guidance of the GUI 1210 related to the manual displayed on the display unit 180 of the video display device 100, The operation method, and the like, and at the same time, the external device 30 can be controlled according to the manual.

Referring to FIG. 14B, a user is provided with manual related information through the display unit 180 of the image display apparatus 100 by operating the user interface unit 230 of the augmented remote control apparatus 200. FIG. 14B, when the user presses the button 2 of the UI unit 230 of the augmented remote control device 200, the user presses the button 2 of the GUI related GUI displayed on the display unit 180 of the image display apparatus 100 2) The item for the 'recording' function (1211) is activated.

In the next step, detailed manual information on the recording function of the external device 30 is displayed on the display unit 180 of the video display device 100 through the manual related GUI 1210 as shown in FIG. 14C.

FIGS. 15A to 15C are diagrams illustrating a GUI associated with a manual according to an operation method of the augmented remote control apparatus according to another embodiment of the present invention. 11 to 14C are referred to as a first image display device 100, the second image display device 102 is also shown in FIGS. 15A to 15C. 15A, an external device image 1101 is displayed together with the product name on the display unit 260 of the augmented remote control device 200, and a manual-related GUI 1310 for the external device is displayed 2 video display device 102. The video display device 102 shown in FIG.

15B, when the user plays back the content stored in the external device 30 through the first video display device 100, the second video display device displays a manual related GUI 1310 . Accordingly, the user can use the external device 30 and the first video display device 100, and can continuously receive the manual-related information on the external device 30 without being disturbed by the content viewing. Even when control interface information or manual information for the external device 30 is required while viewing the content, such information can be obtained through the second video display device 102. [ Therefore, it is not necessary to stop or switch the screen displayed on the display unit 180 of the first video display device 100, and the augmented remote control device 200 can be used in the same manner as before. By operating all of the four devices (the external device 30, the first video display device 100, the second video display device 102, and the augmented remote control device 200) connected through the home network, So that it can be provided and used more conveniently.

In particular, the GUI-related GUI 1311 associated with content reproduction is displayed on the second video display device 102. The user can control the external device 30 and / or the first image display device 100 by receiving the manual information related to the content reproduction through the button operation of the user interface unit 230 of the augmented remote control device 200 .

15C shows a state in which no image is displayed on the display unit of the image display apparatus 100 due to failure of the external device 30 or the first image display apparatus 100. FIG. In this case, information related to the manual is required to cope with a failure situation such as a screen error. However, the GUI 1312 relating to the manual can not be displayed through the first video display device 100. [ In this case, a manual-related GUI 1312 related to screen failure is provided through the second video display device 200 located in the vicinity. The user can immediately receive the manual related information and repair the malfunction even if a malfunction occurs in any one of the external device 30 or the first video display device 100. [ At a minimum, the user can identify a fault in which device, whether repair is possible, or the like, through the manual-related information, and deal with the failure.

 It is to be understood that the embodiments and the constructions of the embodiments of the present invention are not limited to the above embodiments and various modifications may be made without departing from the spirit and scope of the invention. Some of which may be selectively combined.

Meanwhile, the present invention can be embodied as a code readable by a processor in a processor-readable recording medium provided in an image display apparatus. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium readable by the processor include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also a carrier wave such as transmission over the Internet. In addition, the processor readable recording medium may be distributed over networked computer systems so that code readable by the processor in a distributed manner can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

10: Broadcast station 30: External device
40: home network 50: home server
100: Image display device 200: Augmented remote control device
300: Network server

Claims (20)

A method of operating an augmented remote control device that provides a user interface using an augmented reality technology,
Recognizing information on a peripheral external device and the external device using a camera;
Transmitting information on the external device to an image display device connected to the external device through a home network;
Receiving control interface information for the external device from the video display device; And
Displaying a GUI (Graphic User Interface) related to a manual using the control interface information so as to overlap with a real image photographed through the camera;
Wherein the information on the external device includes at least one of a serial number, a manufacturer name, a product name, a product code, and a model name of the external device. The method according to claim 1,
Wherein the GUI related to the manual displays information including at least one of product information, use description, or operation method of the external device. The method according to claim 1,
Wherein the information on the external device is image data generated by photographing the external device. delete The method according to claim 1,
Wherein the control interface information includes manual information on the external device. The method according to claim 1,
Wherein the video display device is connected to a network server and receives the control interface information corresponding to the external device when receiving information on the external device. A method of operating an augmented remote control device for providing a user interface using an augmented reality technology,
Recognizing information on a peripheral external device and the external device using a camera;
Receiving the control interface information from a control interface database in which control interface information for the external device is stored; And
And displaying a GUI (Graphic User Interface) related to the external device on the actual image photographed through the camera using the control interface information,
Wherein the information on the external device includes at least one of image data generated by photographing the external device or a serial number, a manufacturer name, a product name, a product code, and a model name of the external device. How it works. 8. The method of claim 7,
Wherein the control interface database is provided in a home server of a home network to which the external device and the augmented remote control device are connected. 8. The method of claim 7,
Wherein the augmented remote control device accesses the control interface database through an IP (Internet Protocol) network to download the control interface information. delete 1. An augmented remote control device for providing a user interface using an augmented reality technology,
An identification unit for recognizing information on a peripheral external device and the external device using a camera;
A wireless communication unit for transmitting information on the external device to a video display device connected to a home network and receiving control interface information for the external device from the video display device; And
And a display unit for displaying a GUI (Graphic User Interface) related to a manual using the control interface information so as to overlap with an actual image photographed through the camera,
Wherein the information on the external device includes at least one of a serial number, a manufacturer name, a product name, a product code, and a model name of the external device. 12. The method of claim 11,
Wherein the GUI related to the manual includes information including at least one of product information, use description, or operation method of the external device. 12. The method of claim 11,
Wherein the identification unit photographs the external device to generate image data, and the information on the external device is the image data generated by photographing the external device. delete 12. The method of claim 11,
Wherein the control interface information includes manual information on the external device. 12. The method of claim 11,
Wherein the image display device is connected to a network server and receives the control interface information corresponding to the external device when receiving information on the external device. 1. An augmented remote control device for providing a user interface using an augmented reality technology,
An identification unit for recognizing information on a peripheral external device and the external device using a camera;
A wireless communication unit for receiving the control interface information from a control interface database in which control interface information for the external device is stored;
A controller for generating a graphical user interface (GUI) related to the external device using the control interface information; And
And a display unit for displaying the manual-related GUI (Graphic User Interface) so as to overlap with the actual image photographed through the camera,
Wherein the information on the external device includes at least one of image data generated by photographing the external device by the camera included in the identification portion or a serial number, a manufacturer name, a product name, a product code, and a model name of the external device Characterized by a remote control device. 18. The method of claim 17,
Wherein the control interface database is provided in a home server of a home network to which the external device is connected. 18. The method of claim 17,
Wherein the wireless communication unit accesses the control interface database through an IP (Internet Protocol) network to download the control interface information.



delete

Images