KR20110117492A - Augmented remote controller and method of operating the same - Google Patents

Abstract

The present invention relates to an augmented remote control apparatus for providing a user interface using augmented reality technology and a method of operation thereof. A method of operating an augmented remote control apparatus according to an embodiment of the present invention includes the steps of recognizing information on the external device and the external device, transmitting the information on the external device to a video display device connected to a home network, Receiving control interface information for the external device from an image display device and displaying a manual GUI (Graphic User Interface) using the control interface information. According to an embodiment of the present invention, a user can easily establish a connection with an external device, and can be provided with a manual for using and controlling the external device. It can be provided to control the external device and the image display device.

Description

Augmented Remote Controller and Method of Operating the Same

The present invention relates to an augmented remote control device, an augmented remote control device control method and a system thereof. The augmented remote control apparatus may recognize the image display device, an external device that may be connected to the image display device, the content provided from the image display device or the external device, or another kind of object and display related information. The user may use the augmented remote control device to control an existing object and receive related information.

The image display device is a device having a function of displaying an image that a user can watch. The user can watch the broadcast through the image display device. The image display device may be connected to an external device. The user can watch the content provided by the external device through the image display device connected to the external device. The video display device may be connected to a predetermined content provider through a network. The user may watch content provided by a predetermined content provider through an image display device connected to a network.

In addition to the content that a user can watch through the image display device, information related to the content is increasing. The user wants to check the information related to the content being viewed in addition to viewing the content through the image display apparatus. In addition, a user wants to efficiently control an image display device and various types of external devices using a single device.

Accordingly, one object of the present invention is to provide a remote control device capable of controlling an image display device or an external device connected to the image display device.

In addition, one object of the present invention is to provide a remote control device that allows a user to efficiently use and manage content and content related information played back or stored in an image display device or an external device connected to the image display device.

In addition, in dealing with electronic devices including external devices, the user can easily obtain a manual and be provided with information on correct usage.

A method of operating an augmented remote control apparatus for providing a user interface using an augmented reality technology according to an embodiment of the present invention for achieving the above object, the step of recognizing information on the external device and the external device, the external device Transmitting GUI information to a video display device connected to a home network, receiving control interface information on the external device from the video display device, and using a control interface information, a GUI related graphical user interface (GUI). Displaying.

A method of operating an augmented remote control apparatus for providing a user interface using augmented reality technology according to another embodiment of the present invention for achieving the above object, the step of recognizing information on the external device and the external device, the external Downloading the control interface information from a control interface database in which the control interface information for the device is stored, and displaying a graphical user interface (GUI) for the external device using the control interface information.

An augmented remote control apparatus for providing a user interface using an augmented reality technology according to another embodiment of the present invention for achieving the above object is an identification unit for recognizing information on the external device and the external device, the external device Graphical User Interface (GUI) using a wireless communication unit and the control interface information for transmitting the information about the information to the image display device connected to the home network and receiving control interface information for the external device from the image display device. It includes a display unit for displaying.

An augmented remote control apparatus for providing a user interface using an augmented reality technology according to an embodiment of the present invention for achieving the above object is an identification unit for recognizing information on the external device and the external device, for the external device A control unit for generating a manual GUI (Graphic User Interface) for the external device using the wireless communication unit for downloading the control interface information from the control interface database in which control interface information is stored and the control interface information, and the GUI related graphic (Graphic) And a display unit for displaying a user interface.

According to an embodiment of the present invention, an image display device or an external device connected to the image display device can be controlled using a single remote control device. In addition, in controlling the image display device or an external device, the user may receive the manual in real time and operate the corresponding devices according to the manual. In particular, it can be easily provided by the augmented remote control device to deal with failures, and difficult to find equipment operation methods.

1 is a block diagram showing an augmented remote control system according to an embodiment of the present invention.
Figure 2 is a block diagram showing an augmented remote control system according to an embodiment of the present invention.
3 is an internal block diagram of an image display apparatus according to an embodiment of the present invention.
4 is an internal block diagram of an image / audio processor of an image display device according to an embodiment of the present invention;
5 is an internal block diagram of an augmented remote control apparatus according to an embodiment of the present invention.
Figure 6 is a flow chart illustrating the flow of the operation method of the augmented remote control apparatus according to an embodiment of the present invention.
7 is a view briefly showing the appearance of the augmented remote control apparatus according to an embodiment of the present invention.
8 is a diagram illustrating an augmented remote control apparatus and its operation method according to an embodiment of the present invention.
9 is a flowchart illustrating a method of operating an augmented remote control apparatus and its system according to another embodiment of the present invention.
10 is a flowchart illustrating a method of operating an augmented remote control apparatus according to another embodiment of the present invention.
11 to 13d is a view showing a manual-related GUI is displayed according to the operation method of the augmented remote control device according to an embodiment of the present invention.
14a to 14c is a view showing a manual related GUI is displayed according to the operation method of the augmented remote control device according to another embodiment of the present invention.
15a to 15c is a view showing a manual-related GUI is displayed according to the operation method of the augmented remote control device according to another embodiment of the present invention.

Hereinafter, with reference to the drawings will be described the present invention in more detail.

The suffixes "module" and "unit" for components used in the following description are merely given in consideration of ease of preparation of the present specification, and do not impart any particular meaning or role by themselves. Therefore, the "module" and "unit" may be used interchangeably.

An augmented remote control apparatus according to an embodiment of the present invention is an apparatus capable of providing an augmented reality environment to a user by recognizing an object existing around the user and providing information related to the recognized object to the user. . The object around the user may be an image display device that can be controlled using an augmented remote control device, an external device connected to the image display device, content provided by the image display device or an external device, or another kind of object. .

The augmented remote control device may collect information related to a user and recognize an object existing around the user. As an example, the augmented remote controller may collect information about a user's location or the user's direction using a GPS or a compass. In addition, the augmented remote control apparatus may photograph the environment around the user by using a camera and recognize an object present in the vicinity. In addition, the augmented remote control device may recognize an object around the user by using a radio frequency identification (RFID) reader.

The augmented remote control apparatus may recognize an object present around a user and search and display information related to the recognized object. The type of information retrieved through the augmented remote control device may correspond to the type of the recognized object.

For example, when the augmented remote control apparatus recognizes an image display apparatus or an external apparatus that exists near a user, the augmented remote control apparatus may search for information on a content list that may be provided through the image display apparatus or the external apparatus. . In addition, the augmented remote controller may retrieve information about a user interface that can control the image display device or an external device. The augmented remote control device may display the searched information about an image display device or an external device existing around the user to the user.

In addition, the augmented remote control device may recognize content provided from an image display device or an external device that exists around the user. In this case, the augmented remote control device may search for information about the content. The augmented remote controller may display information on content provided by the video display device or an external device to the user. In addition, the augmented remote controller may display a user interface for editing, playing, or transmitting content provided from an image display device or an external device.

The augmented remote control may also recognize other types of objects that exist around the user. For example, a user may photograph furniture existing in the vicinity of a camera mounted on the augmented remote control device. The augmented remote control apparatus may recognize that an object corresponding to an image photographed through a camera is a furniture by referring to a database in which information about images of various objects is stored. The augmented remote control device may search for and display information on a name or a manufacturer of furniture.

The augmented remote controller may augment and display the information retrieved about the object photographed by the camera on the actual image photographed by the camera. For example, the augmented remote controller may display an actual image photographed through a camera on a display unit. The augmented remote controller may retrieve information about an object included in an actual image displayed on the display unit. The augmented remote controller may display information on the searched object on the display unit using a popup window or an icon. In addition, the augmented remote control device may display information on the searched object as an image or text on the display unit.

The user may check the Augmented Real Image, which is augmented with information retrieved from the augmented remote control, through the augmented remote control. The user may check information on a real image or a predetermined object included in the real image through augmented information displayed to overlap the real image captured by the camera.

In addition, when the display unit displaying the augmented real image is a touch screen, the user may interact with the augmented remote controller by selecting a pop-up window, an icon, an image, or a text indicating the augmented information. For example, when the user selects a pop-up window indicating the first augmentation information, the augmented remote control apparatus may display detailed information related to the first augmentation information on the display unit. In addition, when the user selects a pop-up window indicating the first augmentation information, the augmented remote control device may execute an application related to the first augmentation information. In this case, the application executed by the augmented remote control device may be an application that can control an object such as an image display device or an external device included in the augmented real image.

In the case of the augmented remote control apparatus using the transparent display unit, the augmented remote control apparatus may display the augmentation information so that the augmentation information overlaps the actual image projected through the transparent display unit. The augmented remote control apparatus may search for information about a predetermined object included in an actual image projected through the transparent display unit and display the retrieved information as augmented information on the transparent display unit.

The augmented remote control device may wirelessly transmit and receive signals with an image display device or an external device that may be connected to the image display device. The user may control the image display device or an external device using the augmented remote control device. In addition, the augmented remote control device may receive information on the operation state of the image display device or the external device and display it on the display unit.

 The augmented remote control device may be connected to a network including an internet network. The augmented remote control apparatus may retrieve information about the recognized object through a network. The augmented remote control device may display the information retrieved through the network to the display unit.

1 is a block diagram showing an augmented remote control system according to an embodiment of the present invention.

The augmented remote control apparatus 200 may transmit / receive a signal with the image display apparatus 100, the external device 30 that may be connected to the image display apparatus 100, and the network server 300. The video display device 100 is a device capable of playing various types of content. The image display apparatus 100 may receive a signal including an image signal corresponding to the content from the outside. The image display apparatus 100 may extract an image signal included in the received signal and display an image corresponding to the extracted image signal.

The external device 30 is a device capable of playing content stored in accordance with a predetermined standard. The external device 30 may include a display unit. In this case, the external device 30 may display an image corresponding to the reproduced content on the display unit of the external device 30. In addition, the external device 30 may be connected to the image display device 100. In this case, the external device 30 may transmit a signal including a video signal corresponding to the content to the video display device 100. The image display apparatus 100 may display an image corresponding to an image signal included in a signal received through the external device 30.

The video display device 100 may receive a broadcast signal through the broadcast station 10. The image display apparatus 100 may display an image corresponding to the image signal included in the broadcast signal. The image display apparatus 100 may receive a signal including an image signal from a network server 300 existing in a network including an internet network. The image display apparatus 100 may display an image corresponding to an image signal included in a signal received from the network server 300.

In the case of the image display apparatus 100 connected to the Internet network, a signal including a video signal corresponding to the predetermined content may be received through the content provider 20 that is built in the Internet network and provides the content. The image display apparatus 100 may receive a signal transmitted from the content provider 20 and display an image corresponding thereto.

The augmented remote control apparatus 200 may recognize the image display apparatus 100 or the external device 30. The augmented remote control apparatus 200 may photograph a user's surrounding environment through a camera and recognize the image display apparatus 100 or the external device 30 through the photographed image discrimination. In addition, the augmented remote control apparatus 200 receives a signal transmitted from the image display device 100 or the external device 30 with an RFID attached through the RFID reader, and the image display device 100 or an external device with an RFID ( 30) can be recognized.

In addition, the augmented remote control apparatus 200 may recognize the image display apparatus 100 or the external device 30 by transmitting and receiving different types of signals with the image display apparatus 100 or the external device 30. For example, the augmented remote control apparatus 200 may transmit a signal based on an IR or RF communication standard with the image display apparatus 100 or the external device 30. The augmented remote control apparatus 200 may be paired with an image display apparatus 100 or an external device 30 that transmits and receives a signal according to an IR or RF communication standard. The image display device 100 or the external device 30 may recognize a signal transmitted from the augmented remote control device 200 in which pairing is completed.

The augmented remote control apparatus 200 may display a menu for controlling the recognized image display apparatus 100 or the external device 30 on a display unit of the augmented remote control apparatus 200. The user may control the image display device 100 or the external device 30 by selecting a menu displayed on the display unit of the augmented remote control apparatus 200 or by operating a button or a key provided in the augmented remote control apparatus 200. You can enter a command. The augmented remote control apparatus 200 may transmit a signal corresponding to a command input through a user to the image display apparatus 100 or the external device 30. The image display device 100 or the external device 30 may be controlled according to a signal transmitted through the augmented remote control device 200.

The augmented remote control apparatus 200 may recognize the image display apparatus 100 or the external device 30 using a camera, an RFID reader, or the like. The augmented remote control apparatus 200 determines the content related information provided from the recognized image display apparatus 100 or the external device 30 through metadata provided from the image display apparatus 100 or the external device 30. can do. Also, the augmented remote control apparatus 200 may search the network server 300 for content related information provided from the recognized image display apparatus 100 or the external device 30.

The augmented remote control apparatus 200 may display content related information provided by the image display apparatus 100 or the external device 30 on a display unit of the augmented remote control apparatus 200. The content related information may correspond to the type of content recognized by the augmented remote control apparatus 200.

For example, when the currently provided content is content related to shopping, the augmented remote control apparatus 200 may search for information related to a price, a product name, a sales place, an online sales site, etc. of an item related to the corresponding content. In addition, when the currently provided content is content related to a famous tourist destination, the augmented remote control apparatus 200 may search for information related to a name of a tourist destination, a tourism product, a related picture or a video, etc. related to the corresponding content. In addition, when the currently provided content is a movie, the augmented remote control apparatus 200 may search for information related to the movie maker, the production company, the actor, or the like, or related movie. The user may set the type of information searched by the augmented remote control apparatus 200 for each content type.

Figure 2 is a block diagram showing an augmented remote control system according to an embodiment of the present invention.

The augmented remote control apparatus 200 of FIG. 2 transmits and receives a signal directly with the image display apparatus 100 and the external device 30, or the home network 40 to which the image display device 100 or the external device 30 is connected. Signals can be transmitted and received with the home server 50. The home network 40 is a network through which a predetermined number of video display devices 100 or external devices 30 can transmit and receive signals according to a predetermined network communication standard. The home network 40 may be an independent network distinct from the network on which the network server 300 including the content provider 20 is built. Home network 40 may be built in-house or at home.

The home server 50 may store information about the image display device 100 and the external device 30 connected to the home network 40. For example, the home server 50 may store information about a product name, a model name, a usage method, content provided, etc. of the image display device 100 or the external device 30 connected to the home network 40. In addition, the home server 50 may control signal transmission and reception of the home network 40.

The augmented remote control device 200 may recognize the type of the image display device 100 or the external device 30 connected to the home network 40 through a camera or an RFID reader. The augmented remote control apparatus 200 directly displays the information on the content provided by the recognized image display apparatus 100, the external apparatus 30, the image display apparatus 100, or the external apparatus 30. Or it may be provided through the external device 30. In addition, the interim remote control device 200 may be provided with the relevant information through the home server (50). In addition, the augmented remote control apparatus 200 may be provided by searching the relevant information through the network server 300.

The augmented remote control apparatus 200 may display the retrieved information on the display unit of the augmented remote control apparatus 200. The augmentation information displayed on the augmented remote control apparatus 200 may overlap an image captured by a camera or an actual image projected on the transparent display unit. The augmented remote control apparatus 200 may display an augmented real image having the augmentation information overlapped on the display unit.

3 is an internal block diagram of an image display apparatus according to an embodiment of the present invention.

Referring to FIG. 3, the image display apparatus 100 according to an embodiment of the present invention may include a broadcast signal receiver 110, a network interface unit 120, an external device input / output unit 130, and a remote control device interface unit 140. ), The controller 150, the image / audio processor 170, the display 180, and the sound output unit 185 may be included.

The broadcast signal receiver 110 may receive an RF broadcast signal corresponding to a channel selected by a user or all pre-stored channels among RF (Radio Frequency) broadcast signals received through an antenna. The broadcast signal receiver 110 may convert the received RF broadcast signal into an intermediate frequency signal or a baseband video or audio signal and output the converted RF signal to the video / audio processor 170.

In addition, the broadcast signal receiver 110 may receive an RF broadcast signal of a single carrier according to an Advanced Television System Committee (ATSC) scheme or an RF broadcast signal of multiple carriers according to a digital video broadcasting (DVB) scheme. The broadcast signal receiving unit 110 may sequentially select the RF broadcast signals of all broadcast channels stored through the channel memory function among the received RF broadcast signals and convert them into intermediate frequency signals or baseband video or audio signals. This is for illustrating a thumbnail list including a plurality of thumbnail images corresponding to a broadcast channel on the display unit 180. Accordingly, the broadcast signal receiver 110 may receive RF broadcast signals of a selected channel or all prestored channels sequentially / periodically.

The network interface unit 120 provides an interface for connecting the image display apparatus 100 to a wired / wireless network including an internet network.

The network interface unit 120 may include a wireless communication unit capable of wirelessly connecting the image display apparatus 100 to the Internet. For wireless Internet access, wireless LAN (Wi-Fi), wireless broadband (Wibro), world interoperability for microwave access (Wimax), high speed downlink packet access (HSDPA) communication standards, and the like may be used.

The network interface unit 120 may receive content or data provided by a content provider or a network operator through a network. That is, content such as broadcasts, games, VODs, broadcast signals, and the like, which are provided from a content provider, may be received through a network. In addition, the update information and the update file of the firmware provided by the network operator can be received.

The external device input / output unit 130 may connect the external device 30 and the image display device 100. To this end, the external device input / output unit 130 may include an A / V input / output unit or a wireless communication unit.

The external device input / output unit 130 is connected to the external device 30 such as a DVD (Digital Versatile Disk), Blu-ray (Blu-ray), a game device, a camera, a camcorder, a computer (laptop), or the like by wire / wireless. The external device input / output unit 130 transmits an image, audio or data signal input from the outside through the connected external device 30 to the video / audio processing unit 170 of the video display device 100. In addition, the video, audio or data signal processed by the video / audio processor 170 may be output to the connected external device 30.

The A / V input / output unit includes an Ethernet terminal, a USB terminal, a Composite Video Banking Sync (CVBS) terminal, a component terminal, so that video and audio signals of the external device 30 can be input to the video display device 100. S-video terminal (analog), DVI (Digital Visual Interface) terminal, HDMI (High Definition Multimedia Interface) terminal, RGB terminal, D-SUB terminal and the like.

In addition, the wireless communication unit may perform wireless communication with another external device. The image display device 100 may be connected to other electronic devices and networks according to communication standards such as Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, etc. Can be connected.

In addition, the external device input / output unit 130 may be connected through at least one of the various set top boxes and the various terminals described above to perform input / output operations with the set top box.

For example, when the set-top box is an IP (internet protocol) TV set-top box, the external device input / output unit 130 outputs a video / audio or data signal processed by the set-top box for the IP TV to enable bidirectional communication. The voice processing unit 170 may transmit the voice. In addition, the signals processed by the video / audio processor 170 may be transmitted to the set-top box for the IP TV.

Meanwhile, the above-described IPTV may mean ADSL-TV, VDSL-TV, FTTH-TV, etc. according to the type of transmission network, and include TV over DSL, Video over DSL, TV overIP (TVIP), and Broadband TV ( BTV) and the like. In addition, IPTV may also mean an Internet TV capable of accessing the Internet, or a full browsing TV.

In addition, the external device input / output unit 130 may be connected to a communication network capable of video or voice calls. The communication network may mean a broadcast type communication network, a public telephone network, a mobile communication network, or the like connected through a LAN.

The augmented remote control device interface unit 140 may calculate a coordinate of a wireless communication unit capable of transmitting and receiving a signal wirelessly with the augmented remote control device 200 and a pointer corresponding to an operation of the augmented remote control device 200. Coordinate calculation unit may be provided. The augmented remote control device interface unit 140 may transmit and receive a signal wirelessly with the augmented remote control device 200 through the RF module. In addition, the augmented remote control apparatus 200 may receive a signal transmitted according to the IR communication standard through the IR module.

The coordinate calculation unit of the augmented remote controller interface unit 140 may correct a hand shake or an error from a signal corresponding to the movement of the augmented remote controller 200 received through the wireless communication unit of the augmented remote controller interface unit 140. . The coordinate calculation unit may calculate the coordinates (x, y) of the pointer to be displayed on the display unit 180 of the image display apparatus 100 after correcting the shaking or error.

The augmented remote control apparatus 200 transmission signal input to the image display apparatus 100 through the augmented remote controller interface unit 140 is output to the controller 150 of the image display apparatus 100. The controller 150 may determine the information on the movement and the key manipulation of the augmented remote controller 200 from the signal transmitted from the augmented remote controller 200 and control the image display apparatus 100 accordingly. .

As another example, the augmented remote controller 200 may calculate a pointer coordinate corresponding to the movement of the augmented remote controller 200 and output the pointer coordinates to the augmented remote controller interface unit 140. In this case, the augmented remote control interface unit 140 may transmit the information about the received pointer coordinates to the controller without any additional shaking or error correction process.

The controller 150 controls overall operations of the image display apparatus 100. The controller 150 may receive a signal transmitted from the augmented remote controller 200 through the augmented remote controller interface unit 140. In addition, a command may be input through a local key provided in the image display apparatus 100. The controller 150 determines a command included in the received signal or a command corresponding to a local key manipulation and controls the image display apparatus 100 to correspond thereto.

For example, when a user inputs a predetermined channel selection command, the controller 150 controls the broadcast signal receiver 110 to input a broadcast signal provided through the selected channel through the broadcast signal receiver 110. In addition, the video / audio processor 170 is controlled to process the video signal and the audio signal of the selected channel. In addition, the video / audio processing unit 170 is controlled so that the channel information selected by the user can be output through the display unit 180 or the audio output unit 185 together with the image signal and the audio signal.

As another example, the user may input another type of video or audio output command through the augmented remote control apparatus 200. That is, the user may want to watch a camera or camcorder video signal input through the external device input / output unit 130 instead of the broadcast signal. In this case, the controller 150 controls the video / audio processor 170 to output the video signal or the audio signal input through the USB input unit or the like of the external device input / output unit 130 through the display unit 180 or the audio output unit 185. ) Can be controlled.

The image display apparatus 100 may include a user interface controller for generating a graphic user interface associated with the image display apparatus 100. The function of the user interface controller may be in charge of the controller 150. In the present embodiment, the user interface controller is described as a separate component for convenience.

The graphic user interface generated through the user interface controller is output to the display unit 180 or the sound output unit 185 through the image / audio processor 170. The graphical user interface generated by the user interface controller may include a command included in a signal received through the augmented remote control apparatus 200, a command input through a local key included in the image display apparatus 100, or an image display apparatus 100. May be changed based on the operation of the "

For example, when a signal is received through the augmented remote control apparatus 200 or 200, the user interface controller generates a pointer image signal corresponding to the movement of the augmented remote control apparatus 200 or 200 to generate an image / audio processor. 170). The controller 150 may output the information about the pointer coordinates calculated from the signal received through the augmented remote controller 200 to the user interface controller. The user interface controller may generate a pointer image signal based on the information about the pointer coordinates output from the controller 150. The image / audio processor 170 processes the image signal output to the display unit 180 such that a pointer corresponding to the pointer image signal generated by the user interface controller is displayed on the display unit 180. The pointer displayed on the display unit 180 corresponds to the movement of the augmented remote control apparatus 200.

As another example, the user interface controller may include a command included in a signal received through the augmented remote control apparatus 200, a command input through a local key, or an object corresponding to an operation of the image display apparatus 100. The video signal may be generated and output to the video / audio processor 170.

The object may include various types of widgets displayed on the display unit 180 to input a command to the image display apparatus 100 or to display information related to the image display apparatus 100. The widget may be displayed by an OSD (On Screen Display).

The object may include an image or text indicating information on an image displayed by the image display apparatus 100 or the image display apparatus 100, such as an audio output (volume) level, channel information, and current time of the image display apparatus 100. do. The object may be displayed in the image display apparatus 100 or may be implemented in another form (ex) according to the type of information to be displayed. The object of the present embodiment does not limit the scope of the present invention.

Widget refers to an element that allows a user to directly change specific data in a graphical user interface (GUI). For example, the widget may be one of a volume control button, a channel control button, a menu, an icon, a navigation tab, a scroll bar, a progress bar, a text box, and a window displayed on the display unit 180 of the image display apparatus 100. . The shape of the widget implemented in the image display apparatus 100 may be changed according to a specific specification of a GUI that may or may be implemented in the image display apparatus 100 and the widget of this embodiment does not limit the scope of the present invention. No.

The video / audio processor 170 is suitable for the display unit 180 such that a video signal or an audio signal included in a signal received through the broadcast signal receiver 110, the network interface unit 120, and the external device input / output unit 130 is provided. Do it. The video / audio processor 170 may process the video signal or the audio signal based on information included in the data signal received together with the video signal or the audio signal.

The video / audio processor 170 processes the video signal or the audio signal input through the user interface controller to be output from the display unit 180 or the audio output unit 185. The user may grasp the operation state of the image display apparatus 100 or relate to the image display apparatus 100 using a graphic user interface displayed on the display unit 180 based on the image signal or the audio signal generated by the user interface controller. You can enter a command.

The video / audio processor 170 may select a video signal or an audio signal to be processed in response to a user command input through the controller 150. The video signal or audio signal processed by the video / audio processor 170 is a video or audio signal output through the display unit 180 or the audio output unit 185. The command input through the controller 150 may include a broadcast channel selection command and a content selection command to be played among contents input to the image display apparatus 100.

The image / audio processor 170 of the present exemplary embodiment may process the image signal so that a 2D or 3D image signal input from the outside may be displayed on the display unit 180. The video / audio processor 170 may process the video signal such that the user interface generated through the user interface controller is displayed in three dimensions on the display 180. A detailed description of the video / audio processor 170 will be described later with reference to FIG. 4.

The display unit 180 displays the video signal, the data signal, the OSD signal processed by the video / audio processor 170, or the video signal, data signal, etc. received through the external device input / output unit 130, respectively. Convert to a signal to generate a drive signal. The display unit 180 may be a PDP, an LCD, an OLED, a flexible display, or the like. In the present embodiment, the display unit 180 may be a 3D display.

The 3D display may be divided into an additional display method and a single display method according to a method of recognizing a 3D image by a user.

The independent display method is a method in which a 3D image may be implemented on a display alone without a separate auxiliary device. A user viewing a display using a single display method can view a 3D image without an additional device (ex) glasses. The independent display may include a lenticular method and a parallax barrier.

The additional display method is a method of implementing a 3D image using an auxiliary device. Additional display methods may include a head mounted display (HMD) method, glasses method, and the like. In addition, polarizing glasses, shutter glasses, spectral filters, and the like may be applied to the glasses used in the spectacles.

The display unit 180 may be configured as a touch screen and used as an input device in addition to the output device.

The audio output unit 185 receives a signal processed by the video / audio processor 170, for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs the audio. The sound output unit 185 may be implemented with various types of speakers.

4 is an internal block diagram of an image / audio processor 170 of an image display apparatus according to an exemplary embodiment.

The video / audio processor 170 may include a demodulator 171, a demultiplexer 172, a decoder 173, and a formatter 175.

The demodulator 171 may demodulate the broadcast signal received by the broadcast signal receiver 110. For example, the demodulator 171 may perform a demodulation operation by receiving the digital IF signal DIF converted by the broadcast signal receiver 110. In addition, the demodulator 171 may perform channel decoding. To this end, the demodulator 171 may include a convolution decoder, a deinterleaver, a reed-soloman decoder, and the like to perform convolutional decoding, deinterleaving, and reed soloman decoding.

The demodulator 171 may output a stream signal TS after performing demodulation and channel decoding. The stream signal may be a signal multiplexed with a video signal, audio signal or data signal. For example, the stream signal may be an MPEG-2 Transport Stream (TS) multiplexed with an MPEG-2 standard video signal, a Dolby AC-3 standard audio signal, and the like. Specifically, the MPEG-2 TS may include a header of 4 bytes and a payload of 184 bytes.

The demodulator 171 may be provided separately according to the ATSC method and the DVB method.

The stream signal output from the demodulator 171 may be input to the demultiplexer 172.

The demultiplexer 172 may demultiplex the received stream signal, for example, the MPEG-2 TS, and divide the received stream signal into a video signal, an audio signal, and a data signal. The stream signal input to the demultiplexer 172 may be a stream signal output from the demodulator 171, the network interface 120, and the external device input / output unit 130.

The demultiplexed data signal may be an encoded data signal. The encoded data signal may include EPG (Electronic Progtam Guide) information including broadcast information such as a start time and an end time of a broadcast program broadcasted in each channel. For example, the EPG information may be TSC-PSIP (ATSC-Program and System Information Protocol) information in the ATSC scheme, and may include DVB-Service Information (DVB-SI) in the DVB scheme. .

The decoder 173 may decode the demultiplexed signal. The decoder 173 of the present exemplary embodiment may include a video decoder for decoding the demultiplexed video signal and a scaler for adjusting the resolution of the decoded video signal to be output from the video display device 100.

The image / audio processor 170 according to an embodiment of the present invention may further include a mixer that mixes an image signal input from the outside to the image display device 100 and an image signal generated by the user interface controller 160. have. The function of the mixer may be in charge of the formatter 175. In the present embodiment, a mixer is described separately from the formatter 175 for convenience. The display unit 180 displays an image based on the mixed image signal. The video signal mixed by the mixer is output to the formatter 175.

The formatter 175 may determine the format of the mixed video signal by referring to the data signal associated with the video signal. The formatter 175 may convert the mixed video signal into a format suitable for the display unit 180 and output the converted image signal to the display unit 180.

The image display apparatus 100 of the present exemplary embodiment may display a 3D image on the display unit 180. The formatter 175 may generate a 3D image signal according to a predetermined format by separating the mixed image signal into a plurality of viewpoint images. The formatter 175 outputs the generated 3D image signal to the display unit 180. The display unit 180 displays a 3D image based on the generated 3D image signal.

The 3D image is composed of a multi-view image. The user may watch the multi-view image through the left eye and the right eye. The user may feel a three-dimensional effect of the 3D image through the difference in the image viewed through the left eye and the right eye. A multi-view image for realizing a 3D image generally includes a left eye image that a user can recognize through the left eye and a right eye image that can be recognized through the right eye.

The 3D video signal format may be determined according to a method of arranging a left eye image and a right eye image generated to implement a 3D image. The left eye image and the right eye image are arranged in a left and right format as a side by side format. The method of arranging the left eye image and the right eye image up and down is called a top / down format. The method of arranging the left eye image and the right eye image in time division is called a frame sequential format. The method of mixing the left eye image and the right eye image for each line is called an interlaced format. The method of mixing the left eye image and the right eye image for each box is called a checker box format.

The image signal included in the signal input to the image display apparatus 100 from the outside may be a 3D image signal capable of realizing a 3D image. Also, the graphic user interface video signal generated by the user interface controller 160 may be a 3D video signal capable of implementing 3D video. The mixer may mix the 3D image signal included in the signal input to the image display apparatus 100 from the outside and the 3D image signal generated by the user interface controller to output the output to the formatter 175.

The formatter 175 may determine the format of the mixed 3D video signal by referring to a related data signal. The formatter 175 may process and output the 3D image signal to the display unit 180 so as to conform to the determined format. In addition, when the 3D video signal format that can be output from the display unit 180 is limited, the formatter 175 may adapt the received 3D video signal to the 3D video signal format that can be output from the display unit 180. The conversion may be output to the display unit 180.

If the format cannot be determined from the data signal of the mixed 3D video signal input to the formatter 175, the formatter 175 may determine the format of the input 3D video signal using a predetermined algorithm. For example, the formatter 175 may determine the format of the 3D video signal by analyzing an edge of the image generated based on the input 3D video signal.

In addition, when the mixed video signal input to the formatter 175 is a 2D video signal capable of realizing 2D video, the formatter 175 may generate a 3D video signal through an algorithm for converting the 2D video into a 3D video.

5 is an internal block diagram of the augmented remote control apparatus 200 according to an embodiment of the present invention. The augmented remote controller 200 includes a wireless communication unit 210, a storage unit 220, a user interface unit 230, a sensor unit 240, an identification unit 250, a display unit 260, and an audio output unit 270. ), And a controller 280.

The wireless communication unit 210 may transmit and receive signals to and from electronic devices such as the image display device 100, the external device 30, the home server 50, or the network server 300. In this embodiment, the augmented remote control apparatus 200 may be provided with an RF module that can transmit and receive signals with the peripheral device according to the RF communication standard. In addition, the enhanced remote control device 200 may be provided with an IR module that can transmit and receive signals with the peripheral device according to the IR communication standard.

In addition to the wireless communication unit 210 may transmit and receive signals to and from the electronic device according to various types of communication standards, and includes a module suitable for transmitting and receiving signals according to the corresponding communication standards. Examples of wireless communication standards for signals that can be transmitted and received through the wireless communication unit 210 of the augmented remote control device 200 include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), and UWB. (Ultra Wideband), ZigBee (ZigBee) may be.

The wireless communication unit 210 may transmit and receive signals to and from the internet network according to various types of wireless internet standards, and includes a module suitable for transmitting and receiving signals according to the corresponding wireless internet standards. Wireless Internet standards that can be used in the augmented remote control device 200 include Wireless LAN (WLAN) (Wi-Fi), Wibro (Wireless broadband), World Interoperability for Microwave Access (Wimax), and High Speed Downlink Packet Access (HSDPA). There may be.

In the present embodiment, the augmented remote control apparatus 200 transmits a signal containing information on the operation of the augmented remote control apparatus 200 to the electronic device through the wireless communication unit 210. In addition, the augmented remote control device 200 may receive a signal transmitted by the electronic device through the RF module. In addition, the augmented remote control apparatus 200 may transmit a command regarding power on / off, channel change, volume change, etc. to the electronic device through the IR module as necessary.

The storage unit 220 may store various types of programs, application data, and the like required for controlling or operating the augmented remote control apparatus 200. If the augmented remote control device 200 transmits and receives a signal wirelessly through the electronic device and the RF module, the augmented remote control device 200 and the electronic device transmit and receive a signal through a predetermined frequency band. The controller 280 of the augmented remote control apparatus 200 may store and refer to the storage 220 for information about a frequency band for wirelessly transmitting and receiving a signal with a peripheral device paired with the augmented remote control apparatus 200. have.

The user interface 230 may be configured as a keypad or a button. The user may input a command related to the image display apparatus 100 to the augmented remote control apparatus 200 by manipulating the user interface 230. When the user interface 230 includes a hard key button, a user may input a command related to an electronic device to the augmented remote control apparatus 200 through a push operation of the hard key button. When the user interface 230 includes a touch screen, a user may input a command related to an electronic device to the augmented remote control apparatus 200 by touching a soft key of the touch screen. In addition, the user interface 230 may include various kinds of input means that a user can operate, such as a scroll key or a jog key.

The sensor unit 240 includes a sensor that can be used to collect user related information using the augmented remote control apparatus 200. The sensor unit 240 may include a GPS, a compass (compass), a gyro sensor, an acceleration sensor, an infrared sensor, and the like. GPS can be used to determine a user's location. The compass can be used to determine the direction of the user.

The gyro sensor may be used to determine information about the movement of the user or the augmented remote control apparatus 200 that the user is operating. For example, the gyro sensor may sense information about the movement of the augmented remote control apparatus 200 based on the x, y, and z axes. The acceleration sensor 243 may sense information about a moving speed of the spatial remote control.

The augmented remote control apparatus 200 may be used to determine an object existing in a user's surrounding environment with reference to user related information collected through the sensor unit 240. In addition, the augmented remote control apparatus 200 may determine the user gesture by referring to the user related information collected through the sensor unit 240. The augmented remote control apparatus 200 may be controlled according to a command corresponding to a user gesture. In addition, when the command corresponding to the user gesture is an electronic device control command, the augmented remote control apparatus 200 may transmit a signal including the control command to the corresponding electronic device.

The identification unit 250 may identify a user surrounding object including an electronic device. The identification unit 250 may include a camera, an RFID reader, an infrared sensor, or the like. The identification unit 250 may photograph a user surrounding body using a camera. Images of objects photographed through the identification unit 250 may be compared with images of various objects stored in the storage unit 220, the home server 50, or the network server 300. The identification unit 250 may recognize the photographed object by analyzing the pattern of the image and extracting information about an object corresponding to the image having the same pattern as the photographed image pattern.

In addition, the identification unit 250 may read the RFID related information mounted on the object through the RFID reader and recognize the object. In addition, the identification unit 250 may determine whether an object exists around the user through the infrared sensor. The augmented remote control apparatus 200 may refer to information about an object existing for each user location or direction. Information about an object existing for each user location or direction may be stored in the storage 220, the home server 50, or the network server 300.

The identification unit 250 determines the location and direction of the current user based on the user-related information collected through the sensor unit 240, and the location of the object whose presence is detected through the infrared sensor among the stored information about the object. And extract information about an object corresponding to a direction to recognize an object present around a user. For example, the augmented remote control apparatus 200 may refer to map information in which information about a building located at a predetermined user location and direction is stored. In this case, the augmented remote control apparatus 200 may recognize an actual building around a user by referring to map information of the surrounding object corresponding to the user location and direction of the user who has the augmented remote control apparatus 200.

In addition, the identification unit 250 may recognize the face, fingerprint, iris and the like of the person photographed through the camera. The identification unit 250 may recognize the person photographed through the augmented remote control apparatus 200 by comparing the recognized face, fingerprint, and iris patterns with the stored face, fingerprint, and iris patterns for each person. The controller 280 may search for and output the content related information recognized through the identification unit 250. For example, when a person is recognized through the identification unit 250, the recognized person related information (eg, name, age, personal preference content, etc.) may be searched and output.

The display unit 260 and the audio output unit 270 may correspond to the operation of the user interface unit 230 or may be provided with an electronic device such as an image display device 100, an external device 30, a home server 50, and a network server 300. Video or audio corresponding to the signal transmitted from the device can be output. The user may recognize whether the user interface 230 is manipulated or whether the electronic device is controlled through the display 260 and the sound output unit 270.

The display unit 260 may display object related information included in an image captured by the camera. The display unit 260 may display an augmented real image overlapping the augmented information searched on the image photographed by the camera. As another example, when the display unit 260 is a transparent display unit formed of a transparent panel, the user may view an actual image of the surrounding environment through the transparent display unit. The augmented remote control apparatus 200 may search for object related information included in an actual image projected on the transparent display unit. The augmented remote control apparatus 200 may display information about the searched object on the transparent display unit.

The controller 280 may overlap the augmentation information on the image photographed by the camera and output the overlapping information to the display unit 260. Also, the controller 280 may output an image signal related to the augmentation information to the display unit 260 such that the augmentation information related to the object included in the image projected on the transparent display unit overlaps with the actual image currently being projected on the transparent display unit. Can be. In addition, the controller 280 controls various aspects related to the control of the augmented remote controller 200.

The control unit 280 corresponds to a signal corresponding to a predetermined key operation of the user interface unit 230 or a signal corresponding to an operation of the augmented remote control apparatus 200 sensed by the sensor unit 240 through the wireless communication unit 210. Can be sent to the electronics.

3 to 5 are block diagrams of the image display apparatus 100 and the augmented remote control apparatus 200 according to an embodiment of the present invention. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the image display apparatus 100 and the augmented remote control apparatus 200 that are actually implemented. That is, two or more constituent elements may be combined into one constituent element, or one constituent element may be constituted by two or more constituent elements, if necessary. In addition, the functions performed in each block are intended to illustrate the embodiments of the present invention, and the specific operations and apparatuses do not limit the scope of the present invention.

Figure 6 is a flow chart illustrating the flow of the operation method of the augmented remote control apparatus 200 according to an embodiment of the present invention.

The augmented remote control apparatus 200 of the present embodiment may recognize an object around a user (S10). The augmented remote control apparatus 200 may search for the recognized object related information (S20). The augmented remote control apparatus 200 may display the searched object related information (S30).

There may be various types of objects around the user that the augmented remote control apparatus 200 may recognize. The user may select the type of object that the user wants to recognize through the augmented remote control apparatus 200. For example, the user may select to recognize the electronic device in the augmented remote control apparatus 200. In this case, the augmented remote control apparatus 200 may recognize an electronic device present around a user by using an RFID attached to the electronic device.

In addition, the user may select to recognize the content provided by the image display apparatus 100 or the external device 30. In this case, the augmented remote control apparatus 200 may display the current image display apparatus 100 or the external apparatus 30 using metadata provided from the image display apparatus 100, the external apparatus 30, or the home server 50. You can recognize the content provided by. That is, when the user is positioned so that the identification means such as a camera or an RFID reader mounted on the augmented remote control apparatus 200 faces the image display apparatus 100 that is currently playing the content, the augmented remote control apparatus 200 identifies the identification means. Through the present invention, the content that the user wants to identify can be recognized by the electronic device that is being played, and the content being played by the video display device 100 can be recognized by using the metadata related to the electronic device.

The user can also choose to recognize a particular person. In this case, the augmented remote control apparatus 200 may read a face, a fingerprint, an iris, etc. of a specific person through a camera, a fingerprint reader, an iris reader, etc. of the identification unit 250. The augmented remote control apparatus 200 may recognize the identity of a specific person photographed through the augmented remote control apparatus 200 by comparing the information on the face, fingerprint, iris and the like stored in the database with the read information.

The user may also choose to recognize nearby objects (ex buildings, furniture, etc.). In this case, the augmented remote control apparatus 200 may collect user related information related to a user's location or direction using a GPS or a compass. In addition, the augmented remote control apparatus 200 may capture an image of the surrounding object using a camera. The augmented remote control apparatus 200 may recognize the surrounding object photographed by referring to the image information of the surrounding object corresponding to the current user position or direction in a database in which object information for each user position and direction is stored.

The information referred to by the augmented remote control apparatus 200 in the process of recognizing an object may include an image display apparatus 100, an external device 30, a storage unit 220 of the augmented remote control apparatus 200, and a home server 50. Or it may be stored in the network server 300. Also, the augmented remote control apparatus 200 may display the recognized object-related information on the image display apparatus 100, the external device 30, the storage 220 of the augmented remote control apparatus 200, the home server 50, or the network. The server 300 may search.

For example, when the augmented remote control apparatus 200 recognizes the image display apparatus 100 or the external apparatus 30, the augmented remote control apparatus 200 may search a list of contents being played by the corresponding electronic device. In addition, the content list stored in the image display apparatus 100 or the external device 30 may be searched. In addition, the broadcast signal related information received by the image display apparatus 100 may be searched. In addition, it is possible to search for information including a menu for controlling the corresponding electronic device, such as the image display device 100 or the external device 30.

The augmented remote control apparatus 200 may search for content related information when the image display apparatus 100 or the external apparatus 30 recognizes the content being played or stored. The content related information may include title information and shopping information of the corresponding content.

In addition, when the augmented remote control apparatus 200 recognizes a specific person, the augmented remote control apparatus 200 may search for information related to the specific person. The information related to a specific person may include information about a specific person's name, age, occupation, contact information, or the like, or a content history preferred by the specific person or a content history watched by the specific person.

In addition, when the augmented remote control apparatus 200 recognizes a real object (ex) building, furniture, artwork, etc. around the user, the augmented remote control apparatus 200 may search for information related to the real object. The information related to the real object may include a name of the real object, a manufacturer, a selling price, a place of sale, and a method of use.

The augmented remote control apparatus 200 may display the retrieved information on the display unit 260. The augmented remote control apparatus 200 may display the searched information using a popup window or an icon. In addition, an image or text corresponding to the retrieved information may be displayed. The augmented remote control apparatus 200 may overlap the augmented information searched on the image photographed by the camera and display it on the display unit 260.

In this case, the controller 280 may configure a screen displayed on the display unit 260 so that the object corresponding to the searched information among the objects included in the surrounding environment and the searched information do not overlap. Even when the augmentation information is displayed on the transparent display unit, the controller 280 may configure a screen displayed on the display unit 260 so that the augmentation information is displayed in an area that does not overlap with the object corresponding to the searched information.

Figure 7 is a simplified view showing the appearance of the augmented remote control apparatus 200 according to an embodiment of the present invention.

The screen 261 displayed on the display unit 260 of the augmented remote control apparatus 200 displays the object 1001 recognized by the augmented remote control apparatus 200 and information related to the recognized object 1001 1002 and 1003. It may include.

In the present embodiment, the object 1001 recognized by the augmented remote control apparatus 200 is an image of an object actually photographed through a camera of the augmented remote control apparatus 200. Also, the related information 1002 and 1003 are augmented information including information retrieved by the augmented remote control apparatus 200. As shown, the augmented remote control apparatus 200 displays augmented information along with an image of an object to be photographed on the screen 261. The user may check the information about the object to be photographed through the augmentation information included in the screen 261.

The object related information may include a menu for controlling the object. In this embodiment, the related information 2 1003 may be a menu for inputting a related command to the image display device 100 and the external device 30. In addition, in the present embodiment, the related information 2 1003 may be a menu for stopping playback of content currently being played or transmitting it to another electronic device.

When the screen 261 is displayed on the touch screen, the user may touch the touch screen to display information related to the object 1001 on the screen 261. In addition, a command corresponding to the selected icon may be input to the augmented remote controller 200 by selecting a predetermined icon from a menu included in the related information 2 1003 displayed on the screen 261.

The enhanced remote control device 200 may include a keypad 231. The user may input a predetermined command to the augmented remote control apparatus 200 by operating a predetermined key constituting the keypad 231.

8 is a diagram illustrating an augmented remote control apparatus and its operation method according to an embodiment of the present invention.

According to the embodiment described with reference to FIG. 8, the augmented remote control apparatus first recognizes one or more external devices connected to the image display device or the home network through the identification unit 250 (S810). The recognition process of the external device is usually performed along with a process in which the augmented remote control device photographs the external device to generate a preview image. Therefore, the augmented remote controller has a module capable of recognizing a subject, generating a preview image, and capturing and storing an image of the module 250 such as a camera and a display capable of displaying and storing the preview image or the stored image in real time. The unit 260 is mounted.

As described above, the identification unit 250 of the augmented remote control device may recognize the external device through the RFID. The augmented remote control device reads or acquires external device identification information such as a serial number or a product code of the corresponding external device through various methods. Alternatively, the image data generated by photographing the external device may be used for identifying the external device. The image data generated by photographing the external device may be one or more of preview image data streamed or image data stored after capture.

The augmented remote control apparatus, through the camera included in the identification unit 250, any information including the image or image data of the external device and various identification information of the external device read or received from the external device will be described below. This is called information about the system. The wireless communication unit 210 of the augmented remote control device transmits the information about the external device recognized through the above process to the image display device (S820).

For example, in the case of the augmented remote control apparatus of the present embodiment, the identification unit 250 photographs an external device to download a manual, and then the external device corresponding to the image captured by the wireless communication unit 210 or the control unit 280 is photographed. You can search directly. In this case, the wireless communication unit 210 accesses a network server or a home server, searches for an external device, and transmits information about the external device to the image display device. Images captured in the search process may be used. Thereafter, the image display device downloads control interface information or manual information corresponding to the external device by using information about the external device received from the augmented remote control device. In addition, the image display apparatus may directly display a GUI related to the manual on the display unit 180 of the image display apparatus using control interface information or manual information, or may transmit control interface information or manual information to the augmented remote control apparatus.

As another example, the augmented remote control device reads the serial number, manufacturing company name, product name, product code, model name, etc. of the RFID-attached external device using the RFID reader of the identification unit 250, the controller 280 is external Device related information can be determined. The augmented remote control device may recognize an external device to download a manual based on the information on the external device, and transmit the information on the recognized external device to the image display device. The identification unit 250 of the augmented remote control device may determine information on the external device by using a code according to a communication standard other than the RFID reader and recognize the corresponding electronic device.

As another example, the augmented remote control apparatus may check a list of external devices connected to the home network by accessing a home server of a home network to which an external device is connected or connectable through the wireless communication unit 210. The controller 280 of the augmented remote control device downloads information on the external device and displays the downloaded information when the external device to be provided with a manual is included in the list of external devices through the wireless communication unit 210. Can be sent to the device. Then, the image display device uses the information on the external device in searching the control interface information or manual information of the corresponding external device in the control interface database.

The image display device receiving the information on the external device inquires whether the corresponding external device has a history of being previously connected to the augmented remote control device or the image display device. If there is a history of access, since the control interface information or the manual information for the external device is already stored, it is not necessary to download the control interface information or the manual information newly by connecting to the web server.

In this case, the control interface information is information necessary for generating a user interface to be provided when the user controls the external device using the augmented remote control device. In particular, the control interface information may include manual information of an external device. The manual information is information corresponding to an operation method necessary for using a specific external device, a manual of the external device, or a countermeasure in case of a failure. That is, the image display apparatus or the augmented remote control apparatus then generates a GUI related to the manual using the control interface information or the manual information included therein.

The controller 280 of the augmented remote control apparatus, the controller 150 of the image display apparatus, or the image / audio processor 170 may generate a manual-related GUI by using control interface information or manual information included in the control interface information. . The manual related GUI is a user interface image provided to input a user signal for control when a user wants to control an external device using an augmented remote control device. The manual related GUI may be displayed in the form of an OSD. Therefore, the manual-related GUI may display, for example, text and image data on an operation method for each function of an external device, a virtual keypad to be pressed for operation, and the like.

If it is confirmed that the external device has never been connected before, the image display device downloads control interface information corresponding to the external device. In order to download the necessary control interface information, the network interface unit 120 of the image display apparatus connects to a web server, and the controller 150 of the image display apparatus retrieves the control interface information corresponding to the external device from the control interface database. And can be selected. If there is a history of access to the external device and the control interface information is stored in the video display device or the home server, the video display device transmits the previously stored control interface information to the augmented remote control device or uses the control interface information. You can create a manual GUI.

When the image display device retrieves the control interface information or the manual information, the controller 150 uses the information on the external device received from the augmented remote control device in the search. The control interface database is a database in which control interfaces of external devices connectable to the image display device and / or the augmented remote control device are stored.

In the control interface database, manual related information of each external device may be stored together. When the image display apparatus downloads control interface information and the like from the control interface database, the augmented remote control apparatus receives control interface information and the like from the image display apparatus through the wireless communication unit 210 (S830). Of course, the control interface information may include manual information on the corresponding external device.

According to another exemplary embodiment, before the augmented remote control apparatus transmits information about an external device to the image display apparatus, the controller 280 of the augmented remote control apparatus inquires whether there is a history of previously connected with the corresponding external device, Only when there is no history, information on an external device may be transmitted to the image display device. In the embodiment described with reference to FIG. 8, when the augmented remote control apparatus transmits information about an external device to the image display apparatus, the image display apparatus determines whether the image display apparatus has previously connected, and there is no connection history. Only connect to the web server. That is, there is a difference in whether the access history inquiry is performed in the augmented remote control apparatus or the image display apparatus.

The augmented remote control apparatus displays the manual related GUI on the display unit 260 of the augmented remote control apparatus using the received control interface information (S840). Alternatively, the augmented remote control apparatus may allow the control interface information to be displayed on the display unit 180 of the image display apparatus.

The user may be provided with other information such as usage of the corresponding external device, product specification, and countermeasures in case of failure through the GUI related manual displayed on the display unit 260 of the image display device or the AR controller. .

9 is a flowchart illustrating a method of operating an augmented remote control apparatus and its system according to another embodiment of the present invention.

The augmented remote control device recognizes one or more external devices (S910). The identification unit 250 of the augmented remote control device acquires information for identifying the external device in the recognition process of the external device. As described above, the information for identifying the external device is information for identifying the external device, such as an image photographing the external device or a serial number of the external device, a manufacturing company name, a product name, a product code, a model name, and the like.

The augmented remote control apparatus may recognize an external device by capturing an external device through a camera and comparing the captured image with an image of the external device stored in a database. In addition, in the case of an RFID-attached external device, the augmented remote control device may recognize the external device by reading the RFID attached to the external device through the RFID reader. In addition, the augmented remote control device may recognize the external device through the home server.

The controller 280 of the augmented remote control device determines whether there is a history of previously connected to the external device by using the information on the external device (S920). As a result, if there is a history of access, the controller 280 of the augmented remote control device generates a GUI related to the manual using previously stored control interface information or pre-stored manual information, and displays the generated manual related GUI. 260 may display (S950).

If the external device does not have a previously connected history, the augmented remote control device accesses the network server through the wireless communication unit 210 to search the control interface database (S930). If there is control interface information or manual information corresponding to the external device among the information stored in the control interface database, the augmented remote control device downloads the corresponding control interface information or manual information (S940). In addition, a GUI related to the manual is generated using the information downloaded by the controller 280 and displayed on the display unit 260 (S950).

The augmented remote control device may allow a manual related GUI to be played on a specific electronic device regardless of the location where the recognized external device control interface information or manual information is stored. In this case, the electronic device for reproducing and displaying the manual-related GUI may be predefined by the user. Accordingly, when the augmented remote control apparatus is connectable with a plurality of image display apparatuses, the augmented remote control apparatus may allow a manual related to the first external apparatus stored in the first image display apparatus to be played on the second image display apparatus. The display unit 260 of the augmented remote control device may be displayed.

In this case, the user may photograph the first external device through a camera provided in the identification unit 250 of the augmented remote control device. The augmented remote controller may recognize the first external device photographed through the camera. The augmented remote control device may search for a recognized first external device related manual. In the present exemplary embodiment, the augmented remote control apparatus confirming that the first external device related manual is stored in the first image display apparatus may display a thumbnail corresponding to the manual stored in the first image display apparatus on the display unit of the augmented remote control apparatus. . The user may touch the thumbnail and drag it to the second image display device so that a manual corresponding to the thumbnail is displayed on the second image display device.

10 is a flowchart illustrating a method of operating an augmented remote control apparatus according to another embodiment of the present invention.

The augmented remote control device recognizes the external device through the identification unit 250 (S1010). As described above, an RFID reader or a camera may be used to recognize the external device. The identification unit 250 recognizes the external device and recognizes or acquires information on the external device together. The information on the external device may be image data generated by photographing the external device as described above, or may be information such as a serial number and a product code for identifying the external device as information read from the external device.

In addition, the control unit 280 of the augmented remote control device inquires or determines whether there is a history of the previously connected to the external device (S1020). If there is a history of access to the corresponding external device, the controller 280 inquires whether there is previously stored control interface information in relation to the corresponding external device (S1030). If there is no previous connection with the corresponding external device, the controller 280 of the augmented remote control device transmits information about the external device to the image display device through the wireless communication unit 210 (50). This is because the control interface information for the external device must be newly acquired through the image display device.

The video display device downloads the control interface information for the external device by accessing a network server or a home server and searching a control interface database. If the control interface information is already stored in the storage unit of the image display device, the server connection and / or the control interface information retrieval process may be omitted. In addition, the image display device transmits control interface information about the external device to the augmented remote control apparatus through the augmented remote control interface 140, and the wireless communication unit 210 of the augmented remote control apparatus receives it (S1070). .

In the above, when the augmented remote control apparatus does not have previously stored control interface information with respect to the corresponding external device, the augmented remote control apparatus may request control interface information from the image display apparatus (S1060). The controller 280 may generate a control signal for requesting control interface information, and the wireless communication unit 210 may transmit the control signal to the augmented remote control interface unit 140 of the image display device. Alternatively, the control interface information request may be replaced by transmitting information (image data or identification information) for the corresponding external device. In this case, the augmented remote control apparatus receives control interface information from the image display apparatus (S1070).

The controller 280 of the augmented remote controller generates a manual related GUI corresponding to the corresponding external device using the received control interface information (S1080). If there is data that can display a GUI related to an external device in the image display device, the augmented remote control device may receive and use it.

If there is previously stored control interface information in the augmented remote control apparatus, the controller 280 may also determine whether there is a manual-related GUI previously generated and stored using the control interface information (S1040). If a GUI related to the manual has already been generated and stored for the corresponding external device, the process of separately transmitting information on the external device to the video display device or requesting control interface information and using the same to generate the manual related GUI will be omitted. Can be. Therefore, the controller 280 of the augmented remote control device generates a manual related GUI when there is already stored control interface information related to the external device but a manual related GUI is not generated (S1080).

The display unit 260 of the augmented remote control device displays the generated manual-related GUI (S1090). The GUI related manual may be displayed together with the corresponding external device, and is not necessarily displayed on the display unit 260 of the augmented remote control device, but may be displayed through an image display device or an electronic device having other image display functions. You may.

11 to 13D are diagrams illustrating a manual related GUI displayed according to an operation method of an augmented remote control apparatus according to an exemplary embodiment of the present invention.

First, referring to FIG. 11, an external device 30, an image display device 100, and an augmented remote control device 200 are illustrated. The augmented remote control device is intended to display a manual-related GUI for the external device 30 by augmented reality technology.

Referring to FIG. 11, the manual-related GUI 1110 is displayed on the display unit 260 of the augmented remote control apparatus 200 together with the external device image 1101. The user can also obtain more detailed information related to the manual by entering the desired number via the keypad. That is, when the user selects No. 2, information on how to use the external device 30 through the manual-related GUI 1110, and when No. 3 is selected, information on how to diagnose the failure of the external device 30, If you select No. 4, you will be provided with information on repair or after-sales center in case of failure.

Referring to FIG. 12A, the user has selected two 'use methods'. That is, if the second and the first is selected in sequence, the manual related GUI may display the manual related to the playback method in detail in the use method of the external device 30. In another embodiment, a user interface image for actually controlling the external device 30 may be included in the manual related GUI. That is, the manual-related GUI 1110 may provide not only information about a playback method among the usage methods of the external device 30, but also an interface for allowing the external device 30 to play content immediately after pressing a specific button. It may be.

As the user selects two 'use methods' in FIG. 12A, in the next step, a manual-related GUI 1111 is displayed to guide further details on the use methods as shown in FIG. 12B. Referring to FIG. 12B, a user selects an item related to 'caption setting' from 'use method'. Accordingly, referring to FIG. 12C, a manual-related GUI 1112 related to caption setting is displayed. The manual-related GUI 1112 related to subtitle setting includes a method for setting subtitles, what command to enter or not to display subtitles, what is the language of the subtitles provided, how to select the subtitle language, etc. Information may be displayed. 12C, the manual-related GUI 1112 related to the subtitle setting includes an option for selecting whether to go directly to a screen for setting a subtitle, or to continue reading the instruction including the subtitle setting (eg, the subtitle setting page). Low '1113 and' No, continue to use instructions' (1114). If the user wants to immediately play the content through the external device 30 and set the caption for the content, the user may select 'Yes, to the caption setting page' 1113. Alternatively, if the subtitle setting does not need to be performed immediately or if the user wants to obtain more manual information other than the subtitle setting, the user may select 'No, continue to use the instruction' 1114.

Similar to the manual information on how to use, manual information on troubleshooting or how to deal with a failure can be provided in the same way. 13A to 13C, manual related information related to a failure handling method is provided through the display unit 260 of the augmented remote removal apparatus 200. Referring to FIG. 13A, the user selects No. 3 'Fault Diagnosis' from the manual-related GUI 1110 displayed on the display unit 260.

Accordingly, referring to FIG. 13B, a manual-related GUI 1115 for a fault diagnosis method is displayed on the display unit 260 of the augmented remote control device 260. The 'Troubleshooting' menu shows various failure situations that can occur when the external device 30 is used, such as (1) playback error, (2) sound error, (3) screen error, and (4) recording error. The user can select the necessary information from the information on each failure situation.

For example, as illustrated in FIG. 13C, when the user selects '(3) screen abnormality', a manual-related GUI 1116 that provides more detailed information about handling or repairing a screen abnormality may be displayed in the next step. . The manual-related GUI 1116 for screen anomaly allows the user to select whether to go immediately to the repair method page. 'Would you like to go to the repair page for' screen anomalies '?' In response to the question, a user who wants to be immediately provided with a repair method due to a screen abnormality currently selects 'YES' 1117. Alternatively, if the user does not need to be immediately provided with a repair method for the screen abnormality, that is, the user wants to maintain the current page or to view other manual related information, the user may select 'No' 1118.

If the user selects 'YES' 1117, the next step goes to the repair method page for "screen abnormality". Referring to FIG. 13D, a manual-related GUI 1119 for guiding a repair method for a screen abnormality is provided. That is, when a failure situation in which the screen does not appear normally when the content is played back, the user is provided with troubleshooting information corresponding to the situation through the GUI 1119 related to the external device 30 displayed on the augmented remote control apparatus 200. You will be provided. Therefore, even if there is no booklet describing the manual-related information on the external device 30, the manual-related information can be immediately provided. At the same time, the user can control or repair the external device 30 or the image display device 100 through the augmented remote control device 200 according to the provided manual related information.

14A to 14C are diagrams illustrating a manual related GUI displayed according to a method of operating an augmented remote control apparatus according to another embodiment of the present invention.

14A to 14C, while the external device image 1101 is displayed on the display unit 260 of the augmented remote control apparatus 200, the manual related GUI 1210 may be displayed on the display unit of the image display apparatus 100. It is displayed at 180. The display unit 260 of the augmented remote control apparatus 200 may briefly display the external device image 1101 and the product name of the external device. In this case, the user inputs a user signal through the augmented remote control apparatus 200 according to the guidance of the manual-related GUI 1210 displayed on the display unit 180 of the image display apparatus 100 to use the external device 30. , The method of operation and the like can be learned, and at the same time, the external device 30 can be controlled according to the manual.

Referring to FIG. 14B, it is shown that a user is provided with manual-related information through the display unit 180 of the image display apparatus 100 by manipulating the user interface unit 230 of the augmented remote control apparatus 200. For example, as shown in FIG. 14B, when the user presses the button 2 of the user interface 230 of the augmented remote control apparatus 200, the user may view a '(') in the GUI related to the manual displayed on the display unit 180 of the image display apparatus 100. 2) The item for the 'recording' function 1211 is activated.

In the next step, as shown in FIG. 14C, detailed manual information on the recording function of the external device 30 is displayed on the display unit 180 of the image display apparatus 100 through the manual related GUI 1210.

15A to 15C are diagrams illustrating a manual related GUI displayed according to a method of operating an augmented remote control apparatus according to another embodiment of the present invention. When the image display apparatus 100 illustrated in FIGS. 11 to 14C is referred to as the first image display apparatus 100, the second image display apparatus 102 is illustrated together with FIGS. 15A to 15C. As shown in FIG. 15A, an external device image 1101 is displayed on the display unit 260 of the augmented remote control apparatus 200 together with its product name, and a GUI 1310 related to the manual for the external device is provided. 2 is displayed via the image display device 102.

In addition, as illustrated in FIG. 15B, when a user is playing and watching content stored in the external device 30 through the first image display apparatus 100, a manual-related GUI 1310 is displayed on the second image display apparatus. Can be. Accordingly, the user may use the external device 30 and the first image display device 100 and at the same time, the manual related information on the external device 30 may be continuously provided without being disturbed from viewing the content. Even when control interface information or manual information on the external device 30 is required while viewing the content, the information may be obtained through the second image display device 102. Therefore, it is not necessary to stop or switch the screen displayed on the display unit 180 of the first image display apparatus 100, and the augmented remote control apparatus 200 may also be used in the same manner as before. By operating all four devices (external device 30, first image display device 100, second image display device 102, augmented remote control device 200) connected through a home network, a user can obtain desired information. It can be provided and used more conveniently.

In particular, a manual-related GUI 1311 related to content playback is displayed on the second video display device 102. The user may control the external device 30 and / or the first image display apparatus 100 by receiving manual information related to content playback through a button manipulation of the user interface 230 of the augmented remote control apparatus 200. .

FIG. 15C illustrates a situation in which no image is displayed on the display unit of the image display apparatus 100 due to a failure of the external device 30 or the first image display apparatus 100. In this case, manual related information is required to cope with the failure situation such as screen abnormality. However, the manual related GUI 1312 may not be displayed through the first image display apparatus 100. In this case, a manual related GUI 1312 related to a screen failure is provided through the second image display apparatus 200 that is present in the vicinity. The user may immediately receive manual information and repair the failure even when a failure occurs in any one of the external device 30 or the first image display device 100. At least, through the manual-related information, the user can grasp the situation of which device has failed or can be repaired, and can cope with the failure situation.

 Augmented remote control apparatus and its operation method according to the present invention is not limited to the configuration and method of the embodiments described as described above, the embodiments are all or all of the embodiments so that various modifications can be made Some may be optionally combined.

Meanwhile, the present invention can be embodied as processor readable codes on a processor readable recording medium included in the image display device. The processor-readable recording medium includes all kinds of recording devices that store data that can be read by the processor. Examples of the processor-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like, and also include a carrier wave such as transmission through the Internet. The processor-readable recording medium can also be distributed over network coupled computer systems so that the processor-readable code is stored and executed in a distributed fashion.

In addition, while the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

10: broadcasting station 30: external device
40: home network 50: home server
100: video display device 200: augmented remote control device
300: network server

Claims (20)

In the operating method of the augmented remote control device providing a user interface using augmented reality technology,
Recognizing an external device and information on the external device;
Transmitting information about the external device to an image display device connected to the external device through a home network;
Receiving control interface information on the external device from the image display device; And
And displaying a graphic related user interface (GUI) using the control interface information. The method of claim 1,
And the manual-related GUI displays information including one or more of product information, usage description, or operation method of the external device. The method of claim 1,
The information on the external device is an operation method of the augmented remote control device, characterized in that the image data generated by photographing the external device. The method of claim 1,
The information on the external device includes at least one of a serial number, a manufacturing company name, a product name, a product code, and a model name of the external device. The method of claim 1,
And the control interface information includes manual information about the external device. The method of claim 1,
When the image display device receives the information on the external device, the image display device accesses a network server and downloads the control interface information corresponding to the external device. In the operating method of the augmented remote control device providing a user interface using augmented reality technology,
Recognizing an external device and information on the external device;
Receiving the control interface information from a control interface database in which control interface information for the external device is stored; And
And displaying a graphic related user interface (GUI) for the external device using the control interface information. The method of claim 7, wherein
And the control interface database is provided in a home server of a home network to which the external device and the augmented remote control device are connected. The method of claim 7, wherein
The augmented remote control device operating method of the augmented remote control device, characterized in that to download the control interface information by accessing the control interface database through an Internet Protocol (IP) network. The method of claim 7, wherein
The information on the external device includes at least one of image data generated by photographing the external device or a serial number of the external device, a manufacturing company name, a product name, a product code, and a model name. How it works. In the augmented remote control device providing a user interface using augmented reality technology,
An identification unit for recognizing an external device and information about the external device;
A wireless communication unit which transmits information about the external device to an image display device connected through a home network, and receives control interface information on the external device from the image display device; And
Augmented remote control device including a display unit for displaying a graphical user interface (GUI) related to the manual by using the control interface information. The method of claim 11,
The manual-related GUI is augmented remote control device, characterized in that it comprises information including one or more of the product information, instructions for use or operation of the external device. The method of claim 11,
And the identification unit generates image data by photographing the external device, and the information on the external device is the image data generated by the identification unit photographing the external device. The method of claim 11,
And the information on the external device includes at least one of a serial number, a manufacturing company name, a product name, a product code, and a model name of the external device. The method of claim 11,
The control interface information is augmented remote control device, characterized in that it comprises a manual information for the external device. The method of claim 11,
And the image display device receives the information on the external device, connects to a network server, and downloads the control interface information corresponding to the external device. In the augmented remote control device providing a user interface using augmented reality technology,
An identification unit for recognizing an external device and information about the external device;
A wireless communication unit which receives the control interface information from a control interface database in which control interface information for the external device is stored;
A controller configured to generate a GUI related graphical user interface (GUI) for the external device using the control interface information; And
Augmented remote control device including a display for displaying the graphical user interface (GUI) associated with the manual. The method of claim 17,
The control interface database is provided in the home server of the home network to which the external device is connected. The method of claim 17,
And the wireless communication unit downloads the control interface information by accessing the control interface database through an Internet Protocol (IP) network. The method of claim 17,
The information on the external device may include at least one of image data generated by the camera included in the identification unit by photographing the external device or a serial number of the external device, a manufacturing company name, a product name, a product code, and a model name. Augmented remote control device characterized in that.

Images