KR101694166B1 - Augmented Remote Controller and Method of Operating the Same - Google Patents

Abstract

The present invention relates to an augmented remote control device for providing a user interface using an augmented reality technology and an operation method thereof. The method for operating the augmented remote control device according to an embodiment of the present invention includes the steps of recognizing an image display device displaying an image according to the reproduction of the content, recognizing the content being reproduced through the image display device, Receiving a bookmark setting signal for editing a portion of the content to generate a bookmark, and storing bookmark setting information for the content corresponding to the bookmark setting signal. The user can utilize the bookmark function for the content being played back in the electronic device through the augmented remote control device, and also can edit and store a desired portion of the content and play it through the electronic device.

Description

[0001] The present invention relates to an Augmented Remote Controller and a method of operating the same,

The present invention relates to an augmented remote control device, a method of controlling an augmented remote control device, and a system thereof. The augmented remote control device can recognize a content or other kinds of objects provided by an image display device, an external device that can be connected to the image display device, an image display device or an external device, and display related information. The user can control surrounding objects and receive related information by using the augmented remote control device.

A video display device is a device having a function of displaying an image that a user can view. The user can view the broadcast through the video display device. The video display device can be connected to an external device. The user can view the content provided by the external device through the video display device connected to the external device. The video display device may be connected to a predetermined content provider through a network. The user can view contents provided by a predetermined content provider through an image display device connected to the network.

Information related to contents is increasing along with contents that a user can view through a video display device. The user wants to receive information related to the content being watched in addition to watching the content through the video display device. In addition, when the image display apparatus and various types of external apparatuses are efficiently controlled by using one remote control apparatus, the convenience of the user is increased.

Accordingly, it is an object of the present invention to provide a remote control device capable of controlling an external device connected to an image display device or an image display device.

It is another object of the present invention to provide a remote control device that allows a user to efficiently use and manage content and content-related information reproduced or stored in an external device connected to an image display device or an image display device.

In particular, the present invention provides an augmented remote control device capable of generating a bookmark of content reproduced through a video display device or an external device, and controlling an image display device or the like to reproduce edited content according to the generated bookmark, and an operation method thereof do.

According to an aspect of the present invention, there is provided an operation method of an augmented remote control device capable of providing information on a recognized content by using augmented reality technology, Recognizing a content displayed on the video display device, receiving a bookmark setting signal for editing a part of the content to generate a bookmark, and receiving a bookmark setting signal corresponding to the bookmark setting signal, And storing bookmark setting information for the content to be stored in the storage unit.

According to another aspect of the present invention, there is provided an augmented remote control device capable of providing information on content recognized using augmented reality technology, A user interface unit for receiving a bookmark setting signal for generating a bookmark by editing a part of the content, a display unit for displaying the content on the display unit according to the bookmark setting signal, And a storage unit for storing bookmark setting information for the contents corresponding to the bookmark setting signal.

According to the embodiment of the present invention, an image display apparatus or an external apparatus connected to the image display apparatus can be controlled by using one remote control apparatus. In addition, according to the present invention, a user can efficiently use and manage content and content related information reproduced or stored in an external device connected to a video display device or an image display device. Particularly, through the bookmark function, a user can edit or store information on a specific part or a part of contents being played back on the electronic device through the augmented remote control device. In addition, a desired portion of the content can be edited and stored, and only the edited or stored portion can be reproduced through the electronic device, thereby reducing the inconvenience of the user playing the entire content to search for the desired portion. Such a bookmark function can be applied not only to content stored in electronic devices but also to contents received from a broadcasting station or a network, so that utilization and management of contents can be diversified and facilitated.

1 is a block diagram illustrating an augmented remote control system system according to an embodiment of the present invention;
2 is a block diagram illustrating an augmented remote control system system according to an embodiment of the present invention;
3 is an internal block diagram of a video display device according to an embodiment of the present invention.
4 is an internal block diagram of a video / audio processing unit of a video display device according to an embodiment of the present invention;
5 is an internal block diagram of an augmented remote control device according to an embodiment of the present invention;
6 is a flowchart illustrating a flow of an operation method of an augmented remote controller according to an embodiment of the present invention.
7 is a view schematically showing an appearance of an augmented remote control apparatus according to an embodiment of the present invention;
8 is a flowchart illustrating an operation method of the augmented remote control apparatus according to an embodiment of the present invention.
9 is a diagram illustrating an operation method of the augmented remote control apparatus according to another embodiment of the present invention.
FIGS. 10 to 13 illustrate how the augmented remote control device displays bookmark-related information according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

The suffix "module" and " part "for components used in the following description are given merely for convenience of description, and do not give special significance or role in themselves. Accordingly, the terms "module" and "part" may be used interchangeably.

The augmented remote control apparatus according to an embodiment of the present invention is capable of providing an augmented reality environment to a user by providing information related to an object recognized and recognized by the user to the user . The object existing around the user may be an image display device which can be controlled using the augmented remote control device, an external device connected to the image display device, a content provided by the image display device or an external device, or another kind of object .

The augmented remote control device can collect the user related information and recognize the objects existing around the user. For example, the augmented remote control device can acquire information about the user's position or the user's direction using a GPS or a compass. In addition, the augmented remote control device can photograph a user's surrounding environment using a camera and recognize an object existing in the surroundings. Also, the augmented remote control device can recognize an object existing around the user by using an RFID reader.

The augmented remote control device recognizes an object existing around the user and can retrieve and display information related to the recognized object. The type of information searched through the augmented remote control device may correspond to the type of the recognized object.

For example, if the augmented remote control device recognizes an image display device or an external device existing in the vicinity of the user, the augmented remote control device can search information about the content list that can be provided through the image display device or the external device . Further, the augmented remote control device can retrieve information related to the user interface that can control the video display device or the external device. The augmented remote control device can display the retrieved information to the user with respect to the video display device or the external device existing around the user.

In addition, the augmented remote control device can recognize contents provided by an image display device or an external device existing around the user. In this case, the augmented remote control device can retrieve information about the content. The augmented remote control device can display to the user information about the content provided by the video display device or the external device. In addition, the augmented remote control device can display to the user a user interface that can edit, reproduce, or transmit contents provided by the video display device or the external device.

Also, the augmented remote control device can recognize other kinds of objects existing around the user. For example, a user can take pictures of furniture existing in the vicinity by using a camera mounted on the augmented remote control device. The augmented remote control device can recognize that the object corresponding to the image photographed through the camera is a household by referring to a database storing information about images of various objects. The augmented remote control device can retrieve information about the name or manufacturer of the furniture and display it to the user.

The augmented remote control device can enhance and display the retrieved information about the object photographed through the camera on the actual image photographed through the camera. For example, the augmented remote control device can display an actual image photographed through a camera on a display unit. The augmented remote control device can retrieve information about an object included in the actual image displayed on the display unit. The augmented remote control device can display information about the searched object on a display unit using a pop-up window or an icon. Further, the augmented remote control device can display information about the retrieved object on the display unit as an image or text.

The user can confirm the augmented real image of the information retrieved from the augmented remote control device through the augmented remote control device. The user can confirm information about a predetermined object included in an actual image or an actual image through augmented information displayed so as to overlap with an actual image photographed through a camera.

Also, when the display unit on which the enhanced actual image is displayed is a touch screen, the user can select a pop-up window, an icon, an image or text representing the augmentation information, and interact with the augmented remote control device. For example, if the user selects a pop-up window showing first enhancement information, the augmented remote control device can display detailed information related to the first enhancement information on the display unit. Further, when the user selects a pop-up window indicating first enhancement information, the augmented remote control device can execute an application related to the first enhancement information. In this case, the application executed by the augmented remote control device may be an application capable of controlling an object such as an image display device or an external device included in the augmented reality image.

In the case of the augmented remote control apparatus using the transparent display unit, the augmented remote control apparatus can display the augmentation information so that the augmented information overlaps with the actual image projected through the transparent display unit. The augmented remote control apparatus searches for information about a predetermined object included in an actual image projected through the transparent display unit, and displays the retrieved information on the transparent display unit as reinforcement information.

The augmented remote control device can wirelessly transmit and receive signals to and from an external device that can be connected to the image display device or the image display device. The user can control the video display device or the external device using the augmented remote control device. Further, the augmented remote control apparatus can receive information on the operation state of the image display apparatus or the external apparatus and display the information on the display unit.

 The augmented remote control device may be connected to a network including the Internet network. The augmented remote control device can search information on the recognized object through the network. The augmented remote control device can display the information retrieved through the network on the display unit.

1 is a block diagram illustrating an augmented remote control system according to an embodiment of the present invention.

The augmented remote control device 200 can transmit and receive signals to and from the video display device 100, the external device 30 that can be connected to the video display device 100, and the network server 300. The video display device 100 is a device capable of reproducing various kinds of contents. The image display apparatus 100 can receive a signal including a video signal corresponding to the content from the outside. The image display apparatus 100 can extract the image signal included in the received signal and display an image corresponding to the extracted image signal.

The external device 30 is a device capable of reproducing contents stored in conformity with a predetermined standard. The external device 30 may include a display unit. In this case, the external device 30 can display an image corresponding to the content to be reproduced on the display unit of the external device 30. Also, the external device 30 may be connected to the image display device 100. In this case, the external device 30 can transmit a signal including the video signal corresponding to the content to the video display device 100. [ The image display apparatus 100 may display an image corresponding to a video signal included in a signal received through the external apparatus 30. [

The video display apparatus 100 can receive a broadcast signal through the broadcast station 10. [ The image display apparatus 100 can display an image corresponding to a video signal included in a broadcast signal. The image display apparatus 100 may receive a signal including a video signal from the network server 300 in the network including the Internet. The image display apparatus 100 may display an image corresponding to a video signal included in the signal received by the network server 300. [

In the case of the video display device 100 connected to the Internet, a signal including a video signal corresponding to a predetermined content can be received through a content provider 20 provided in the Internet network and providing content. The image display apparatus 100 can receive a signal transmitted from the content provider 20 and display an image corresponding thereto.

The augmented remote control device 200 can recognize the image display device 100 or the external device 30. [ The augmented remote control apparatus 200 can photograph the user's surrounding environment through the camera and recognize the image display apparatus 100 or the external apparatus 30 through the image discrimination. The augmented remote control device 200 also receives a signal transmitted from the image display device 100 or the external device 30 to which the RFID is attached through the RFID reader and transmits the signal to the image display device 100 or the external device 30) can be recognized.

The augmented remote control apparatus 200 can also recognize the corresponding image display apparatus 100 or the external apparatus 30 by transmitting and receiving different kinds of signals to and from the image display apparatus 100 or the external apparatus 30. [ For example, the augmented remote control device 200 can transmit a signal according to the IR or RF communication standard to the image display device 100 or the external device 30. The augmented remote control apparatus 200 can be paired with the image display apparatus 100 or the external apparatus 30 that transmits and receives a signal according to the IR or RF communication standard. The image display apparatus 100 or the external apparatus 30 can recognize a signal transmitted from the augmented remote control apparatus 200 that has been paired.

The augmented remote control apparatus 200 can display a menu for controlling the recognized image display apparatus 100 or the external apparatus 30 on the display unit of the augmented remote control apparatus 200. [ The user can select the menu displayed on the display unit of the augmented remote control apparatus 200 or control the video display apparatus 100 or the external apparatus 30 by operating buttons or keys provided in the augmented remote control apparatus 200 You can enter the command that you have. The augmented remote control apparatus 200 can transmit a signal corresponding to a command input through the user to the image display apparatus 100 or the external apparatus 30. [ The image display device 100 or the external device 30 can be controlled according to a signal transmitted through the augmented remote controller 200. [

The augmented remote control device 200 can recognize the image display device 100 or the external device 30 using a camera, an RFID reader, or the like. The augmented remote control apparatus 200 determines whether the content related information provided from the recognized image display apparatus 100 or the external apparatus 30 is provided through the metadata provided by the image display apparatus 100 or the external apparatus 30 can do. Also, the augmented remote control device 200 can search the network server 300 for the content-related information provided by the recognized image display device 100 or the external device 30.

The augmented remote control device 200 can display the content related information provided by the video display device 100 or the external device 30 on the display unit of the augmented remote control device 200. [ The content-related information may correspond to the type of content recognized by the augmented remote control device 200. [

For example, if the currently provided content is related to shopping, the augmented remote control device 200 can retrieve information related to the price of the item, the name of the product, the place of sale, and the online selling site related to the content. If the presently provided content is content related to a famous sightseeing spot, the augmented remote control device 200 can search for information related to the name of the sightseeing spot, a tourism product, a related photograph or a moving image related to the content. Also, if the currently provided content is a movie, the augmented remote control device 200 can retrieve information related to the movie producer, the producer, the actor, or the related movie. The user can set the type of information to be searched by the augmented remote control device 200 for each content type.

2 is a block diagram illustrating an augmented remote control system system according to an embodiment of the present invention.

The augmented remote control apparatus 200 of FIG. 2 transmits and receives signals directly to and from the image display apparatus 100 and the external apparatus 30 or the home network 40 to which the image display apparatus 100 or the external apparatus 30 is connected, Lt; / RTI > and home server 50 of FIG. The home network 40 is a network in which a predetermined number of video display devices 100 or external devices 30 can transmit and receive signals according to a predetermined network communication standard. The home network 40 may be an independent network that is distinguished from a network in which the network server 300 including the content provider 20 is constructed. The home network 40 may be built in-house or at home.

The home server 50 may store information related to the video display device 100 and the external device 30 connected to the home network 40. [ For example, the home server 50 may store information related to the product name, model name, usage method, provided content, etc. of the video display device 100 or the external device 30 connected to the home network 40. In addition, the home server 50 can control signal transmission / reception of the home network 40. [

The augmented remote control device 200 can recognize the type of the image display device 100 or the external device 30 connected to the home network 40 through the camera and the RFID reader. The augmented remote control apparatus 200 transmits the information about contents provided by the recognized image display apparatus 100, the external apparatus 30, the image display apparatus 100 or the external apparatus 30 directly to the image display apparatus 100. [ Or through the external device 30. Also, the remote controller 200 can receive the related information through the home server 50. Further, the augmented remote control device 200 can search for and receive related information through the network server 300. [

The augmented remote control device 200 can display the searched information on the display unit of the augmented remote control device 200. [ The augmentation information displayed on the augmented remote controller 200 may overlap an image photographed through a camera or an actual image projected on a transparent display unit. The augmenting remote controller 200 can display the augmented actual image in which the augmenting information is overlapped on the display unit.

3 is an internal block diagram of an image display apparatus according to an embodiment of the present invention.

3, an image display apparatus 100 according to an exemplary embodiment of the present invention includes a broadcast signal receiving unit 110, a network interface unit 120, an external device input / output unit 130, a remote control device interface unit 140 A control unit 150, a video / audio processing unit 170, a display unit 180, and an audio output unit 185.

The broadcast signal receiving unit 110 may receive an RF broadcast signal corresponding to a channel selected by the user or all pre-stored channels among RF (Radio Frequency) broadcast signals received through the antenna. The broadcast signal receiving unit 110 may convert the received RF broadcast signal into an intermediate frequency signal, a baseband image, or a voice signal, and output the converted signal to the video / audio processing unit 170.

In addition, the broadcast signal receiver 110 may receive an RF broadcast signal of a single carrier according to an Advanced Television System Committee (ATSC) scheme or an RF broadcast signal of a plurality of carriers according to a DVB (Digital Video Broadcasting) scheme. The broadcast signal receiving unit 110 sequentially selects RF broadcast signals of all broadcast channels stored through the channel memory function among the received RF broadcast signals, and converts the RF broadcast signals into an intermediate frequency signal, a baseband image, or a voice signal. This is to show a thumbnail list including a plurality of thumbnail images corresponding to broadcast channels on the display unit 180. [ Therefore, the broadcast signal receiving unit 110 can receive the RF broadcast signals of the selected channel or all the stored channels sequentially / periodically.

The network interface unit 120 provides an interface for connecting the video display device 100 to a wired / wireless network including the Internet network.

The network interface unit 120 may include a wireless communication unit capable of wirelessly connecting the video display device 100 to the Internet. For wireless Internet access, WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access)

The network interface unit 120 may receive content or data provided by a content provider or a network operator through a network. That is, it is possible to receive contents such as broadcasts, games, VOD, broadcasting signals, and the related information provided from a contents provider through a network. In addition, the update information and the update file of the firmware provided by the network operator can be received.

The external device input / output unit 130 can connect the external device 30 and the image display device 100. [ To this end, the external device input / output unit 130 may include an A / V input / output unit or a wireless communication unit.

The external device input / output unit 130 is connected to an external device 30 such as a digital versatile disk (DVD), a Blu ray, a game device, a camera, a camcorder, a computer (notebook) The external device input / output unit 130 transmits video, audio, or data signals input from the outside through the connected external device 30 to the video / audio processing unit 170 of the video display device 100. Also, the video / audio processing unit 170 can output the processed video, audio, or data signals to the connected external device 30.

The A / V input / output unit includes an Ethernet terminal, a USB terminal, a CVBS (Composite Video Banking Sync) terminal, a component terminal, and a video terminal so that the video and audio signals of the external device 30 can be input to the video display device 100. [ Video terminal (analog), a DVI (Digital Visual Interface) terminal, an HDMI (High Definition Multimedia Interface) terminal, an RGB terminal, a D-SUB terminal, and the like.

Further, the wireless communication unit can perform wireless communication with another external device. The video display device 100 is connected to other electronic devices in accordance with communication standards such as Bluetooth, Radio Frequency Identification (RFID), IrDA (Infrared Data Association), UWB (Ultra Wideband), ZigBee, Can be connected.

Also, the external device input / output unit 130 may be connected to various set-top boxes via at least one of the various terminals described above to perform input / output operations with the set-top box.

For example, when the set-top box is a set-top box for an IP (Internet Protocol) TV, the external device input / output unit 130 transmits video, audio, or data signals processed in the set- To the voice processing unit 170. Also, the video / audio processing unit 170 can transmit the processed signals to the IP TV set-top box.

Meanwhile, the IPTV may include ADSL-TV, VDSL-TV, FTTH-TV and the like depending on the type of the transmission network. The IPTV may include a TV over DSL, a video over DSL, a TV over IP BTV), and the like. In addition, IPTV may also mean an Internet TV capable of accessing the Internet, or a full browsing TV.

Also, the external device input / output unit 130 may be connected to a communication network capable of video or voice communication. The communication network may mean a broadcasting communication network, a public telephone network, a mobile communication network, etc. connected through a LAN.

The augmented remote control device interface unit 140 includes a wireless communication unit capable of wirelessly transmitting and receiving signals with the augmented remote control device 200 and a wireless communication unit capable of calculating the coordinates of a pointer corresponding to the operation of the augmented remote control device 200 And a coordinate calculation unit. The augmented remote control device interface unit 140 can wirelessly transmit and receive signals to and from the augmented remote control device 200 through the RF module. Also, the augmented remote control device 200 can receive a signal transmitted in accordance with the IR communication standard through the IR module.

The coordinate calculation unit of the augmented remote control unit interface unit 140 can correct the camera shake or error from the signal corresponding to the motion of the augmented remote control device 200 received through the wireless communication unit of the augmented remote control device interface unit 140 . The coordinate calculation unit may calculate the coordinates (x, y) of the pointer to be displayed on the display unit 180 of the image display device 100 after the camera shake or error is corrected.

The transmission signal of the augmented remote control device 200 inputted to the image display device 100 through the augmented remote control device interface unit 140 is outputted to the control unit 150 of the image display device 100. [ The control unit 150 can determine the information on the motion and the key operation of the augmented remote control apparatus 200 from the signals transmitted from the augmented remote control apparatus 200 and control the image display apparatus 100 in response to the information .

As another example, the augmented remote control device 200 may calculate pointer coordinates corresponding to the motion of the augmented remote control device 200 and output the coordinates to the augmented remote control device interface unit 140. [ In this case, the augmented remote control device interface unit 140 can transmit information about the coordinates of the received pointer to the control unit without any additional camera shake or error correction process.

The control unit 150 controls the overall operation of the image display apparatus 100. The control unit 150 can receive signals transmitted from the augmented remote control device 200 through the augmented remote control device interface unit 140. [ In addition, a command can be input through the local key provided in the image display apparatus 100. The controller 150 determines a command included in the received signal or a command corresponding to a local key operation, and controls the image display apparatus 100 to correspond to the command.

For example, when the user inputs a predetermined channel selection command, the control unit 150 controls the broadcast signal receiving unit 110 to receive the broadcast signal provided from the selected channel through the broadcast signal receiving unit 110. In addition, the video / audio processing unit 170 is controlled so that the video signal and the audio signal of the selected channel are processed. In addition, the control unit 170 controls the video / audio processing unit 170 so that channel information selected by the user can be output through the display unit 180 or the audio output unit 185 together with the video signal and the audio signal.

As another example, the user can input another kind of video or audio output command through the augmented remote control device 200. [ That is, the user may desire to watch a camera or camcorder image signal inputted through the external device input / output unit 130 instead of the broadcast signal. In this case, the control unit 150 controls the video / audio processing unit 170 to output the video or audio signal input through the USB input unit of the external device input / output unit 130 or the like through the display unit 180 or the audio output unit 185 Can be controlled.

The image display apparatus 100 may include a user interface control unit for generating a graphic user interface associated with the image display apparatus 100. The function of the user interface control unit can be handled by the control unit 150. [ In the present embodiment, the user interface control unit will be described as a separate component for the sake of convenience.

The graphic user interface generated through the user interface control unit is output to the display unit 180 or the sound output unit 185 through the video / audio processing unit 170. [ The graphic user interface generated by the user interface control unit includes a command included in the signal received through the augmented remote controller 200, a command input through the local key included in the image display apparatus 100, As shown in FIG.

For example, when a signal is received through the augmenting remote controller 200, the user interface controller generates a pointer image signal corresponding to the motion of the augmenting remote controller 200, 170). The controller 150 can output information on the pointer coordinates calculated from the signal received through the augmented remote controller 200 to the user interface controller. The user interface control unit can generate a pointer image signal based on the information about the pointer coordinates output from the control unit 150. [ The video / audio processing unit 170 processes the video signal output to the display unit 180 so that a pointer corresponding to the pointer video signal generated by the user interface control unit is displayed on the display unit 180. The pointer displayed on the display unit 180 corresponds to the movement of the augmented remote controller 200. [

As another example, the user interface control unit may include a command included in the signal received through the augmented remote controller 200, a command input through the local key, or a user interface composed of objects corresponding to the operation of the image display apparatus 100 And outputs the generated video signal to the video / audio processing unit 170. [

The object includes various kinds of widgets displayed on the display unit 180 so as to be able to input commands to the image display apparatus 100 or display information related to the image display apparatus 100. [ The widget can be displayed as an OSD (On Screen Display).

The object includes an image or text representing information about the image displayed on the image display apparatus 100 or the image display apparatus 100, such as a sound output (volume) level of the image display apparatus 100, channel information, do. The object may be displayed in the image display apparatus 100 or may be implemented in a form (ex) movie according to the kind of information to be displayed), and the object of the present embodiment does not limit the scope of rights of the present invention.

A widget is a graphical user interface (GUI) that allows a user to directly change specific data. For example, the widget may be one of a volume control button, a channel control button, a menu, an icon, a navigation tab, a scroll bar, a progress bar, a text box, and a window displayed on the display unit 180 of the image display apparatus 100 . The type of the widget implemented in the image display apparatus 100 may be changed in accordance with the specific specification of the GUI that can be implemented or implemented in the image display apparatus 100 and the widget of the present embodiment does not limit the scope of rights of the present invention No.

The video / audio processing unit 170 receives video signals or audio signals included in signals received through the broadcast signal receiving unit 110, the network interface unit 120, and the external device input / output unit 130, . The video / audio processing unit 170 may process the video signal or the audio signal based on the information included in the data signal received together with the video signal or the audio signal.

The video / audio processing unit 170 processes the video signal or the audio signal inputted through the user interface control unit so that the video signal or the audio signal can be output from the display unit 180 or the sound output unit 185. The user can grasp the operation state of the image display apparatus 100 using the graphic user interface displayed on the display unit 180 based on the image signal or the voice signal generated by the user interface control unit, You can enter commands.

The video / audio processing unit 170 can select a video signal or a voice signal to be processed in response to a user command input through the control unit 150. The video signal or audio signal processed by the video / audio processing unit 170 is a video or audio signal output through the display unit 180 or the audio output unit 185. The command input through the control unit 150 may include a broadcast channel selection command, a content selection command to be reproduced among the contents input to the image display apparatus 100, and the like.

The video / audio processing unit 170 of the present embodiment can process a video signal so that a 2D or 3D video signal input from the outside can be displayed on the display unit 180. [ The video / audio processing unit 170 can process the video signal so that the user interface generated through the user interface control unit is three-dimensionally displayed on the display unit 180. A detailed description of the video / audio processing unit 170 will be described later with reference to FIG.

The display unit 180 displays a video signal, a data signal, an OSD signal processed by the video / audio processing unit 170, a video signal and a data signal received through the external device input / output unit 130 as R, G, B Signal to generate a driving signal. The display unit 180 may be a PDP, an LCD, an OLED, or a flexible display. In this embodiment, the display unit 180 preferably has a 3D display.

The three-dimensional display can be divided into an additional display method and a single display method according to a method of recognizing a three-dimensional image by a user.

In the single display method, a 3D image can be implemented independently on a display without a separate auxiliary device. A user who views a display using a single display method can view a three-dimensional image even without the additional apparatus (ex) flat glasses). The single display method may be a lenticular method, a parallax barrier, or the like.

The additional display method is a method of implementing a 3D image using an auxiliary device. Additional display methods include a head-mounted display (HMD) system, a glasses system, and the like. Further, polarized glasses, shutter glasses, spectral filters, and the like can be applied to the glasses used in the glasses system.

Meanwhile, the display unit 180 may be configured as a touch screen and used as an input device in addition to the output device.

The audio output unit 185 receives a signal processed by the video / audio processing unit 170, for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs it as a voice. The sound output unit 185 may be implemented by various types of speakers.

4 is an internal block diagram of the video / audio processing unit 170 of the video display device according to the embodiment of the present invention.

The video / audio processing unit 170 may include a demodulator 171, a demultiplexer 172, a decoder 173, and a formatter 175.

The demodulator 171 may demodulate the broadcast signal received by the broadcast signal receiver 110. For example, the demodulator 171 may receive the digital IF signal DIF converted by the broadcast signal receiver 110 and perform a demodulation operation. The demodulator 171 may perform channel decoding. For this, the demodulator 171 may include a convolution decoder, a deinterleaver, and a reed-solomon decoder to perform convolutional decoding, deinterleaving, and reed solomon decoding.

The demodulator 171 can demodulate and decode the channel and output the stream signal TS. The stream signal may be a signal in which a video signal, a voice signal, or a data signal is multiplexed. For example, the stream signal may be an MPEG-2 TS (Transport Stream) multiplexed with an MPEG-2 standard video signal, a Dolby AC-3 standard audio signal, or the like. Specifically, the MPEG-2 TS may include a header of 4 bytes and a payload of 184 bytes.

The demodulator 171 may be separately provided according to the ATSC scheme and the DVB scheme.

The stream signal output from the demodulator 171 may be input to the demultiplexer 172.

The demultiplexer 172 demultiplexes the received stream signal, for example, the MPEG-2 TS into a video signal, a voice signal, and a data signal. The stream signal input to the demultiplexer 172 may be a stream signal output from the demodulator 171, the network interface unit 120, and the external device input / output unit 130.

The demultiplexed data signal may be an encoded data signal. The encoded data signal may include EPG (Electronic Program Guide) information including broadcast information such as a start time and an end time of a broadcast program broadcasted on each channel. For example, the EPG information may be ATSC-PSIP (ATSC-Program and System Information Protocol) information in the case of the ATSC scheme and DVB-Service Information (DVB-SI) information in case of the DVB scheme .

The decoder unit 173 can decode the demultiplexed signal. The decoder unit 173 of the present embodiment may include a video decoder for decoding the demultiplexed video signal and a scaler for adjusting the resolution of the decoded video signal to be output from the video display device 100.

The video / audio processing unit 170 according to an exemplary embodiment of the present invention may include a mixer for mixing a video signal input from the outside to the video display apparatus 100 and a video signal generated from the user interface control unit 160 have. The formatter 175 may be responsible for the function of the mixer. In the present embodiment, the mixer will be described separately from the formatter 175 for the sake of convenience. The display unit 180 displays an image based on the mixed video signal. The video signal mixed in the mixer is output to the formatter 175.

The formatter 175 can determine the format of the mixed video signal by referring to the data signal related to the video signal. The formatter 175 may convert the mixed video signal into a format suitable for the display unit 180 and output the formatted video signal to the display unit 180.

The video display device 100 of this embodiment can display a 3D image on the display unit 180. [ The formatter 175 may generate a 3D image signal according to a predetermined format by separating the mixed image signal into a plurality of view image. The formatter 175 outputs the generated 3D video signal to the display unit 180. The display unit 180 displays a 3D image based on the generated 3D image signal.

The 3D image is composed of a plurality of viewpoint images. The user can view the multiple view image through the left eye and the right eye. The user can feel the stereoscopic effect of the 3D image through the difference of the image viewed through the left eye and the right eye. A multi-view image for realizing a 3D image is generally composed of a left eye image that the user can recognize through the left eye and a right eye image that can be recognized through the right eye.

The 3D image signal format may be determined according to a method of arranging the left eye image and the right eye image generated to implement the 3D image. The manner in which the left eye image and the right eye image are arranged left and right is called a side by side format. The method of arranging the left eye image and the right eye image up and down is called a top / down format. The method of arranging the left eye image and the right eye image in a time-division manner is called a frame sequential format. The method of mixing the left eye image and the right eye image line by line is referred to as an interlaced format. A method of mixing the left eye image and the right eye image on a box basis is called a Checker Box format.

A video signal included in a signal input from the outside to the image display apparatus 100 may be a 3D image signal capable of implementing a 3D image. Also, the graphic user interface video signal generated by the user interface control unit 160 may be a 3D video signal capable of implementing 3D video. The mixer can mix the 3D image signal included in the signal input from the outside to the image display apparatus 100 and the 3D image signal generated by the user interface control unit, and output the mixed image to the formatter 175.

The formatter 175 can determine the format of the mixed 3D video signal by referring to the related data signal. The formatter 175 may process the 3D video signal to fit the determined format and output the processed 3D video signal to the display unit 180. When the 3D image signal format that can be output from the display unit 180 is limited, the formatter 175 converts the received 3D image signal into a 3D image signal format that can be output from the display unit 180 And output it to the display unit 180.

If the format can not be determined from the data signal of the mixing 3D video signal input to the formatter 175, the formatter 175 can determine the format of the 3D video signal input using a predetermined algorithm. For example, the formatter 175 may determine the format of a 3D video signal by analyzing the edge of an image generated based on the input 3D video signal.

In addition, when the mixing video signal input to the formatter 175 is a 2D video signal capable of implementing a 2D video, the formatter 175 can generate a 3D video signal through an algorithm for converting the 2D video to the 3D video.

5 is an internal block diagram of an augmented remote controller 200 according to an embodiment of the present invention. The augmenting remote controller 200 includes a wireless communication unit 210, a storage unit 220, a user interface unit 230, a sensor unit 240, an identification unit 250, a display unit 260, ), And a control unit 280.

The wireless communication unit 210 can transmit and receive signals to and from an electronic device such as the video display device 100, the external device 30, the home server 50, or the network server 300. In the present embodiment, the augmented remote control device 200 may include an RF module capable of transmitting and receiving a signal with a peripheral device according to an RF communication standard. In addition, the augmented remote control device 200 may include an IR module capable of transmitting and receiving a signal with a peripheral device according to an IR communication standard.

In addition to the wireless communication unit 210, the wireless communication unit 210 has a module capable of transmitting / receiving signals to / from an electronic device according to various communication standards and transmitting / receiving signals according to the communication standard. Examples of wireless communication standards for signals that can be transmitted and received through the wireless communication unit 210 of the augmented remote control device 200 include Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), UWB (Ultra Wideband), and ZigBee (ZigBee).

The wireless communication unit 210 can transmit and receive signals to and from the Internet network in accordance with various types of wireless Internet standards and has a module suitable for transmitting and receiving signals according to the wireless Internet standard. The wireless Internet standard that can be used in the augmented remote control apparatus 200 includes a WLAN (Wireless LAN) (Wi-Fi), a Wibro (Wireless broadband), a Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) Can be.

In the present embodiment, the augmented remote control device 200 transmits a signal containing information on the operation of the augmented remote control device 200 to the electronic device through the wireless communication unit 210. [ Also, the augmented remote control device 200 can receive the signal transmitted by the electronic device through the RF module. In addition, the augmented remote control device 200 can transmit commands to the electronic device via the IR module, such as power on / off, channel change, volume change, and the like, as needed.

The storage unit 220 may store various kinds of programs, application data, and the like necessary for control or operation of the augmented remote control device 200. [ If the augmented remote control device 200 wirelessly transmits and receives a signal through an electronic device and an RF module, the augmented remote control device 200 and the electronic device transmit and receive signals through a predetermined frequency band. The control unit 280 of the augmented remote control device 200 can store information about a frequency band and the like capable of wirelessly transmitting and receiving a signal with a peripheral device paired with the augmented remote control device 200 in the storage unit 220, have.

The user interface unit 230 may include a keypad or a button. The user can input a command related to the image display apparatus 100 to the augmented remote control apparatus 200 by operating the user interface unit 230. [ When the user interface unit 230 includes a hard key button, the user can input a command related to the electronic device to the augmented remote control device 200 through the push operation of the hard key button. When the user interface unit 230 includes a touch screen, the user can input a command related to the electronic device to the augmenting remote controller 200 by touching the soft key of the touch screen. Also, the user interface unit 230 may include various types of input means such as a scroll key, a jog key, and the like that can be operated by the user.

The sensor unit 240 includes a sensor that can be used to collect user-related information using the augmented remote control device 200. [ The sensor unit 240 may include a GPS, a compass, a gyro sensor, an acceleration sensor, an infrared sensor, and the like. GPS can be used to determine the location of the user. The compass can be used to determine the direction of the user.

The gyro sensor may be used to determine information about the motion of the augmented remote control device 200 operated by the user or the user. For example, the gyro sensor may sense information about the motion of the augmented remote control device 200 based on x, y, and z axes. The acceleration sensor 243 can sense information on the moving speed and the like of the spatial remote controller.

The augmented remote control device 200 can refer to the user related information collected through the sensor unit 240 and use it to identify an object existing in the user's surrounding environment. Also, the augmented remote control device 200 can determine the user gesture by referring to the user-related information collected through the sensor unit 240. [ The augmented remote control apparatus 200 can be controlled according to a command corresponding to the user gesture. Also, if the command corresponding to the user gesture is an electronic device control command, the augmented remote control device 200 can transmit a signal including the corresponding control command to the electronic device.

The identification unit 250 may identify a user peripheral object including the electronic device. The identification unit 250 may include a camera, an RFID reader, an infrared sensor, or the like. The identification unit 250 can photograph a user's peripheral seam using a camera. The image of the object photographed through the identification unit 250 may be compared with images of various objects stored in the storage unit 220, the home server 50, or the network server 300. [ The identifying unit 250 may analyze the pattern of the image and recognize information about the object corresponding to the image having the same pattern as the pattern of the captured image to recognize the photographed object.

In addition, the identification unit 250 can read the RFID-related information attached to the object through the RFID reader and recognize the object. In addition, the identification unit 250 can determine whether an object exists around the user through the infrared sensor. The augmented remote control device 200 can refer to information about an object that exists for each user location or direction. Information about an object existing according to a user location or direction may be stored in the storage unit 220, the home server 50, or the network server 300.

The identification unit 250 identifies the location and direction of the current user based on the user-related information collected through the sensor unit 240, identifies the position of the object detected through the infrared sensor among the information about the stored object, And direction of the object, and recognize the object existing around the user. For example, the augmented remote control device 200 may refer to map information stored with information on a building located by a predetermined user location and direction. In this case, the augmented remote control device 200 can recognize the actual user's neighboring building by referring to the map information on the information of the surrounding objects corresponding to the user's location and direction possessing the augmented remote control device 200. [

Also, the identification unit 250 can recognize the face, fingerprint, iris, and the like of a person who has photographed through the camera. The identification unit 250 can recognize a person who has photographed through the augmented remote control device 200 by comparing the recognized face, fingerprint, and iris patterns with stored face images, fingerprints, and iris patterns. The control unit 280 can retrieve and output the content related information recognized through the identification unit 250. [ For example, when a person is recognized through the identification unit 250, the user can retrieve and output the recognized person-related information (ex) name, age, individual preferred content, etc.).

The display unit 260 and the sound output unit 270 may correspond to the operation of the user interface unit 230 or may correspond to the operations of the image display apparatus 100, the external apparatus 30, the home server 50, the network server 300, It is possible to output video or audio corresponding to the signal transmitted from the device. The user can recognize whether the user interface unit 230 is operated or whether the electronic device is controlled through the display unit 260 and the sound output unit 270.

The display unit 260 may display object related information included in the image captured through the camera. The display unit 260 may display an augmented real image that overlaps the augmented information searched on an image photographed through the camera. As another example, when the display unit 260 is a transparent display unit including a transparent panel, the user can view the actual image of the surrounding environment through the transparent display unit. The augmented remote control apparatus 200 can retrieve object related information included in the actual image being projected on the transparent display unit. The augmented remote control device 200 can display information about the object searched in the transparent display unit.

The control unit 280 may overlap the enhancement information with an image photographed through the camera and output the enhancement information to the display unit 260. The control unit 280 outputs the image signal related to the enhancement information to the display unit 260 so that the enhancement information related to the object included in the image projected on the transparent display unit overlaps with the actual image currently projected on the transparent display unit . In addition, the control unit 280 controls various matters related to the control of the augmented remote control device 200.

The control unit 280 transmits a signal corresponding to the predetermined key operation of the user interface unit 230 or a signal corresponding to the operation of the augmented remote control device 200 sensed by the sensor unit 240 through the wireless communication unit 210 It can be transmitted to an electronic device.

The block diagrams of the image display apparatus 100 and the augmented remote control apparatus 200 shown in Figs. 3 to 5 are block diagrams for an embodiment of the present invention. Each component of the block diagram can be integrated, added, or omitted according to specifications of the video display apparatus 100 and the augmented remote control apparatus 200 actually implemented. That is, two or more constituent elements may be combined into one constituent element, or one constituent element may be constituted by two or more constituent elements, if necessary. In addition, the functions performed in each block are intended to illustrate the embodiments of the present invention, and the specific operations and apparatuses do not limit the scope of the present invention.

6 is a flowchart illustrating a flow of an operation method of the augmented remote controller 200 according to an embodiment of the present invention.

The augmented remote controller 200 of the present embodiment can recognize a user's peripheral object (S10). The augmented remote control device 200 can retrieve the recognized object related information (S20). The augmented remote controller 200 can display the retrieved object related information (S30).

There are various kinds of user peripheral objects that the augmented remote control device 200 can recognize. The user can select the type of object to be recognized through the augmented remote control device 200. [ For example, the user may select to recognize the electronic device in the augmented remote control device 200. [ In this case, the augmented remote control device 200 can recognize the electronic devices existing around the user by using the RFID attached to the electronic device.

The user can also select to recognize the content provided by the video display device 100 or the external device 30. [ In this case, the augmented remote control apparatus 200 can control the current image display apparatus 100 or the external apparatus 30 using the metadata provided from the image display apparatus 100, the external apparatus 30 or the home server 50, It is possible to recognize the contents provided by the user. That is, when the user is positioned such that the identification means such as a camera or an RFID reader mounted on the augmented remote control device 200 faces the video display device 100 in which the content is currently being reproduced, the augmented- The content that the current user wants to identify can be recognized by the current user and the content being played back in the video display device 100 can be recognized using the relevant electronic device related metadata.

The user can also choose to recognize a particular person. In this case, the augmented remote controller 200 can read the face, fingerprint, iris, and the like of a specific person through the camera, the fingerprint reader, and the iris reader in the identification unit 250. The augmented remote control device 200 can recognize the identity of a specific person photographed through the augmented remote control device 200 by comparing the information about the person's face, fingerprint, iris, etc. stored in the database with the read information.

The user can also choose to recognize nearby objects (ex) buildings, furniture, etc.). In this case, the augmented remote control device 200 can collect user related information related to the user's location or direction using a GPS or a compass. In addition, the augmented remote control device 200 can photograph an image of a surrounding object using a camera. The augmented remote control apparatus 200 can recognize the captured peripheral object by referring to the image information of the peripheral object corresponding to the current user position or direction among the databases storing the object information per user position and direction.

The information referred to by the augmented remote control device 200 in the process of recognizing the object may be transmitted to the image display device 100, the external device 30, the storage unit 220 of the augmented remote control device 200, the home server 50, Or may be stored in network server 300. The augmented remote control apparatus 200 may transmit the recognized object related information to the image display apparatus 100, the external apparatus 30, the storage unit 220 of the augmented remote control apparatus 200, the home server 50, And can be retrieved from the server 300.

For example, when the augmenting remote controller 200 recognizes the video display device 100 or the external device 30, the augmenting remote controller 200 can search the list of contents being played back in the corresponding electronic device. The content list stored in the video display device 100 or the external device 30 can also be searched. In addition, it is possible to retrieve broadcast signal related information received by the video display device 100. [ It is also possible to search for information including a menu for controlling the electronic apparatus such as the video display apparatus 100 or the external apparatus 30. [

In addition, the augmented remote control apparatus 200 can retrieve content related information when the image display apparatus 100 or the external apparatus 30 recognizes the content being reproduced or stored. The content-related information may include title information, shopping information, and the like of the content.

Further, when the augmenting remote controller 200 recognizes a specific person, the augmenting remote controller 200 can retrieve information related to the specific person. Information related to a specific person may include information about a specific person's name, age, occupation, contact information, or the content preferred by a specific person or the history of a content viewed by a specific person.

In addition, the augmented remote control device 200 can retrieve information related to an actual object when recognizing a real object (ex) building, furniture, artwork, etc. of the user's surroundings). Information related to an actual object may include the name of an actual object, a maker, a selling price, a selling place, a usage method, and the like.

The augmented remote control device 200 can display the searched information on the display unit 260. [ The augmented remote control device 200 can display the searched information by using a pop-up window or an icon. Also, an image or text corresponding to the retrieved information can be displayed. The augmented remote control apparatus 200 can display the enhancement information searched on the image photographed through the camera on the display unit 260 in an overlapping manner.

At this time, the control unit 280 may configure a screen displayed on the display unit 260 so that the object corresponding to the information retrieved from objects included in the surrounding environment does not overlap with the retrieved information. The controller 280 may configure a screen to be displayed on the display unit 260 so that the augmenting information is displayed in an area not overlapping with the object corresponding to the searched information.

7 is a view schematically showing an appearance of an augmented remote control device 200 according to an embodiment of the present invention.

The screen 261 displayed on the display unit 260 of the augmenting remote controller 200 displays the object 1001 recognized by the augmented remote controller 200 and the information 1002 and 1003 related to the recognized object 1001 .

The object 1001 recognized by the augmented remote control device 200 in the present embodiment is an image of an object actually photographed through the camera of the augmented remote control device 200. [ The related information 1002 and 1003 are enhancement information composed of information retrieved by the augmented remote control device 200. [ As shown in the figure, the augmenting remote controller 200 displays the augmenting information on the screen 261 together with the image of the object to be actually photographed. The user can confirm the information about the object to be actually photographed through the augmentation information included in the screen 261. [

The object-related information may include a menu for controlling the object. In this embodiment, the related information 2 1003 may be a menu capable of inputting related commands to the video display device 100 and the external device 30. Also, in this embodiment, the related information 2 1003 may be a menu capable of interrupting the reproduction of the currently reproduced content or transmitting it to another electronic apparatus.

When the screen 261 is displayed on the touch screen, the user can touch the touch screen to display the information related to the object 1001 on the screen 261. Also, a predetermined icon among the menus included in the related information 2 (1003) displayed on the screen (261) can be selected and an instruction corresponding to the selected icon can be inputted to the augmented remote control device (200).

The augmented remote control apparatus 200 may include a keypad 231. [ The user can input a predetermined command to the augmented remote controller 200 by operating a predetermined key constituting the keypad 231. [

The method and method of pairing a remote controller according to the present invention are not limited to the configuration and method of the embodiments described above, but the embodiments may be applied to all or some of the embodiments May be selectively combined.

8 is a flowchart illustrating an operation method of the augmented remote control apparatus according to an embodiment of the present invention.

According to the embodiment described with reference to FIG. 8, the augmented remote control device recognizes an electronic device such as an image display device or an external device connected to the home network through the identification part 250 (S810). The process of recognizing the external device can be performed together with the process of generating a preview image by taking a picture of the external device by the augmented remote control device. When the user shoots an electronic device including an image display device or an external device for which content is being reproduced through a camera mounted on the augmented remote control device, the augmented remote control device links and stores the content-related information currently being played back with the captured image . Accordingly, the augmented remote control device includes a module 250 having a function of recognizing a subject, a preview image generating function and a function of capturing and storing images, an identification unit 250 such as a camera, and a display unit 250 capable of displaying and storing the preview image or the stored image in real time And a display unit 260.

Also, as described above, the identification unit 250 of the augmented remote control device can recognize the external device through the RFID or the like. The augmented remote control device reads or acquires external device identification information such as a serial number or a product code of the external device through various methods. Alternatively, the image data generated by photographing the external device may be used for external device identification. Here, the image data generated by photographing an external device may be at least one of preview image data that is streamed or image data that is stored after capture.

In step S820, the augmented remote control device recognizes the content being played back through the recognized image display device through the first identification part 250. [ Of course, if the content is recognized, if the identification information of the electronic device in which the content is being reproduced is provided, a separate step S810 of recognizing the electronic device such as the video display device 100 may be omitted.

The recognition of the content being reproduced may mean that the augmented remote control device acquires information capable of identifying the content being reproduced through an electronic device such as an image display device. The user can refer to the information about the content being played back in the corresponding electronic device through the metadata transmitted from the recognized electronic device or the related electronic device related metadata transmitted from the home server connected to the recognized electronic device.

The user interface unit 230 of the augmented remote control device receives a bookmark setting signal for editing a part of the content and generating a bookmark (S830).

When the bookmark setting signal is input, the controller 280 of the augmented remote controller can generate a bookmark for the content according to the bookmark setting signal. A bookmark refers to a copy of a portion of a content or a reproduction portion at a specific point in time, a mark for the portion, or information on the portion, depending on the convenience and necessity of the user. If a bookmark is created and stored for a required portion of the content, the user can easily find the desired image without playing back the entire content. For example, when the user wants to display the middle part of the moving picture that is played back for a certain period of time, the user can create a bookmark for the part.

The bookmark can be generated in the form of an image corresponding to a specific time point designated by the user during content reproduction. For example, the control unit 280 may capture an image displayed at the time when the bookmark setting signal is input, and may generate a bookmark image using the captured image. In this case, the bookmark list displayed on the display unit 260 may include bookmark images. A bookmark may be generated in the form of a text that identifies a bookmark for a certain content or a bookmark for a certain portion of the content, or may be a form of an image at a specified point in the content, Can be generated.

When the bookmark is generated, the storage unit 220 stores the bookmark setting information for the content corresponding to the bookmark setting signal (S840). To this end, the storage unit 220 of the augmented remote control device may store images taken during the process of recognizing the image display device and contents together with the information about the content being reproduced. The information on the content being played back may include information on the electronic device on which the content is being played, and information on the current playback time.

The bookmark setting information may include at least one of information for identifying the bookmarked content and information for identifying an electronic device such as a video display device in which the bookmarked content is being reproduced. Also, the playback point information in which the bookmark setting signal is input may be included in the bookmark setting information. The playback point-in-time information may be information indicating when the bookmark setting signal is input to the user interface unit 230 of the augmented remote control apparatus. However, here, information indicating whether the bookmark setting signal is advanced to be. For example, if a user wishes to generate a bookmark for a portion reproduced for 10 seconds from a portion of 1 minute and 30 seconds elapsed from a moving image having a total 3 minutes of reproduction time, the reproduction time point information of the bookmark is '1 minute 30 seconds' Or '1 minute 30 seconds to 1 minute 40 seconds'.

Also, bookmark generation is possible even if the user wants to apply the bookmark function to a broadcast program received from the current broadcast station. That is, an electronic device such as a video display device that receives and reproduces the broadcast program may record a broadcast program received when the bookmark is generated, and may store the copy in the storage unit 220 together with the bookmark setting information.

Then, the display unit 260 of the augmented remote control device can display a bookmark list. The bookmark list is a list of the bookmarks generated according to the bookmark setting signal. As mentioned above, the bookmark list may be an image generated for a specific image constituting the bookmarked content, and such an image may be generated in the form of a thumbnail image. Therefore, the augmented remote control device can display a bookmark having a thumbnail corresponding to the photographed image on the display unit 260 of the augmented remote control device.

9 is a flowchart illustrating an operation method of an augmented remote control apparatus according to another embodiment of the present invention. For the operation method according to the embodiment described with reference to Fig. 9, the operation of the augmented remote control device described with reference to Fig. 8 can be preceded.

The display unit 260 of the augmented remote control device can display the bookmark list (S910). The bookmark list may be a list of the bookmarks generated according to the bookmark setting signal, and the bookmark list may include text and / or a specific image indicating related information such as bookmarked contents or names of contents.

Here, the user can select or designate an electronic device to reproduce the bookmarked content. When images of two or more electronic devices including an image display device are displayed through the display unit 260 of the augmented remote control device, the bookmark list may be displayed on the display of the electronic device capable of reproducing the bookmarked content, Can be displayed on a portion adjacent to the image of the display device. That is, the user wants to reproduce the bookmarked content through the video display device. The video display device can reproduce the bookmarked content, and the augmented remote control device captures the video display device, Is displayed on the display unit 260, a list of bookmarks through the augmented reality technology can be displayed around the video corresponding to the video display device.

The user interface unit 230 receives a bookmark selection signal for one or more bookmarks in the bookmark list (S920). That is, when the bookmark list is displayed, the user can select one or more bookmarks displayed on the list, and input a bookmark selection signal for the selected bookmark to the augmented remote control device. The user interface unit 230 transmits the inputted bookmark selection signal to the control unit 280.

The control unit 280 may generate a remote control signal for reproducing the corresponding content according to the bookmark selected according to the bookmark selection signal. The electronic device receiving the remote control signal reproduces the bookmarked contents.

The augmented remote control device can select an electronic device to reproduce the bookmarked content (S930). The electronic device to reproduce the bookmarked content may be an electronic device designated by the user or an electronic device designated by the default setting of the augmented remote control device. Alternatively, an electronic device in which bookmarked content is stored, that is, a video display device, can also play back bookmarked content according to a selection of a user or the like.

For example, when the video display device is the default designated electronic device, the content can be reproduced by the video display device even when the user inputs only the bookmark selection signal to the augmented remote control device without dragging the bookmark to the video display device.

The user can input a selection command to the augmented remote control device by selecting a bookmark when the bookmark is displayed on the touch screen. In addition, in the case of the augmented remote control device capable of handwriting input or tracking of the user's eyes, the contents of the bookmark corresponding to the handwriting input or the tracked line of the user are reproduced in the image display device.

When the user interface unit 230 receives a bookmark selection signal for one or more bookmarks in the bookmark list and the control unit 280 selects an electronic apparatus to reproduce the content corresponding to the bookmark selected in accordance with the bookmark selection signal, A signal for selecting an electronic device that reproduces the bookmarked content may be input to the user interface unit 230. Alternatively, the bookmark selection signal may include information for selecting an electronic device reproducing the bookmarked content.

For example, when a user displayed on the display unit of the augmented remote control device having a touch screen function drags and drops one bookmark to an image of a specific electronic device, a signal for selecting a bookmark selection signal and an electronic device is generated at once And can be transmitted to an electronic device. That is, when designating or selecting the electronic device to be reproduced, the user can drag the bookmark corresponding to the content desired to be reproduced from the bookmark displayed on the display unit of the augmented remote control device to the side where the electronic device desired to reproduce the content is displayed have. The augmented remote control device transmits a content reproduction command corresponding to the dragged bookmark to the corresponding electronic device so that the content corresponding to the dragged bookmark is played back on the electronic device.

The remote control signal generated to reproduce the content corresponding to the bookmark is transmitted to the selected video display device or other electronic device (S940). At this time, the wireless communication unit 210 of the augmented remote control device transmits a remote control signal to an electronic device such as a video display device.

When the video display device receives the remote control signal, the video display device reproduces the bookmarked content and displays it as an image. In this case, the video display device can reproduce the content corresponding to the portion after the point in time corresponding to the reproduction point information included in the bookmark setting information. That is, when receiving the remote control signal, the image display apparatus can reproduce the content corresponding to the point in time after the bookmark setting signal is input from the bookmarked contents.

The same is true when an electronic device other than the video display device selected by the user or the augmented remote control device receives the remote control signal. Here, the electronic apparatus is provided with a function of reproducing contents and a function of displaying images. The selected electronic device reproduces the bookmarked content and displays it as an image. In this case, the video display device can reproduce the content corresponding to the portion after the point in time corresponding to the reproduction point information included in the bookmark setting information. That is, when receiving the remote control signal, the image display apparatus can reproduce the content corresponding to the point in time after the bookmark setting signal is input from the bookmarked contents.

The remote control signal for commanding the content reproduction may include at least one of information for identifying the bookmarked content, information indicating the location where the content is stored, or playback point-in-time at which the bookmark setting signal is input . Here, the reproduction time information is information indicating which part of the entire contents of the content is to be reproduced from when the bookmarked contents are reproduced.

The electronic device receiving the remote control signal searches the electronic device or server storing the bookmarked contents by using the information included in the remote control signal. The electronic device may be an image display device or an external device as described above. The bookmarked content may be stored together with the video display device having the history of reproducing the original content before the bookmark creation. Also, the electronic device receiving the remote control signal can access the server through the wired / wireless communication network to retrieve the bookmarked contents. Here, the server is a server storing contents corresponding to bookmarks, and may be a home server or a network server. In the case of a home server, connection through a home network is possible. In the case of a network server, connection via an IP network is possible. If the power of the electronic device storing the bookmarked contents is turned off, content search through the home server is possible.

When the bookmarked content is acquired as a result of searching the electronic device or various servers, the electronic device can reproduce the bookmarked portion of the content according to the bookmark setting information or the information included in the bookmark selection signal, and display the image.

FIGS. 10A to 13 illustrate how the augmented remote control device displays bookmark-related information according to an embodiment of the present invention.

10A to 10C, the main image according to the content reproduction is displayed on the display unit 180 of the image display apparatus 100 and the display unit 260 of the augmented remote controller 200 displays the main image, An image 1000 of the image display device 100 and a user interface image 1010 for generating a bookmark are displayed in the main image. According to the example shown in FIG. 10A, the user can input a bookmark setting signal by touching the 'start' portion of bookmark creation from the user interface image 1010 for bookmark creation.

The augmented remote controller receives a bookmark setting signal from a user through a user interface image 1010 for bookmark creation. The user may input a bookmark setting signal using the user interface unit 230 of the augmented remote control apparatus 200. [ Then, a bookmark is generated according to the information included in the bookmark setting signal.

According to the example shown in FIG. 10A, the user can input a bookmark setting signal by touching the 'start' portion of bookmark creation from the user interface image 1010 for bookmark creation.

When the bookmark is generated, the bookmark setting information is stored. Information on which content the user has created bookmarks for, which electronic device is stored, which portion of the content has been bookmarked, And can be stored as setting information. Also, the point of time when the bookmark setting signal is input can be stored as bookmark setting information. Whether or not the bookmark setting signal has been inputted after the lapse of a time from the start of the content reproduction can be inferred from the point of time when the bookmark setting signal is inputted. This information is information indicating which part of the content has been bookmarked Can be used.

In particular, when creating a bookmark, a specific portion of the entire playback portion of the content can be bookmarked, and the user can set the start point 1021 and the end point 1023 of the bookmark. Another example of a method of inputting the bookmark setting signal in this regard is shown in FIG. 10B. The augmented remote control device can display the reproduction status of the content using the progressive bar 1020. [ The bookmark setting information further includes information about a bookmark start point 1021 and an end point 1023. In the case of generating a bookmark image in accordance with bookmark creation, the bookmark image may be a thumbnail image capturing an image displayed at the start point 1021 of the bookmark.

Accordingly, the augmented remote control device can display an icon for inputting an adjustment command for the playback time of the bookmarked content on the display unit of the augmented remote control device. That is, the user can cause the bookmarked content to be reproduced from the beginning in a predetermined electronic device, or the content transmitted in real time on a predetermined channel or the live broadcasted content can be reproduced in a predetermined electronic device.

FIG. 10C is a diagram illustrating a user interface (UI) for allowing a user to select a bookmark to be bookmarked or to store the edited bookmark after designating a start point 1021 and an end point 1023 of a portion to be bookmarked by the user And an image 1010. When the user desires to create a bookmark for the corresponding part, the user selects 'save' displayed on the user interface image 1010. That is, the bookmark setting signal can be generated by specifying the start point 1021 and the end point 1023, but in the case of FIG. 10C, the bookmark setting signal is generated by the user once again inputting the confirmation command. In the case of FIG. 10C, since the user confirms the bookmark before the bookmark is generated immediately, it is easy to modify or edit the bookmarked portion.

Referring to FIGS. 11A and 11B, a list of bookmarks is displayed on the display unit 260 of the augmented remote controller 200. FIG. Then, the bookmark list is provided with a bookmark image and information in the form of text for each bookmark. 11A, no image is displayed on the display unit 180 of the image display apparatus 100, and the user wants to select the bookmarked content to be reproduced through the image display apparatus 100. [

 The bookmark list displayed on the display unit 260 of the augmented remote control apparatus 200 includes a first bookmark 1101, a second bookmark 1103, a third bookmark 1105 and a fourth bookmark 1107 . The bookmark list displays bookmark images and information on electronic devices (electronic device 1, electronic device 2, electronic device 3, etc.) and file names in which bookmarked contents are stored. The bookmark images have the form of a thumbnail image of an image corresponding to a bookmarked time point of each of the contents.

The user selects the first bookmark 1101 among the bookmarks 1101, 1103, 1105, and 1107 displayed in the bookmark list through the user interface unit 230 of the augmented remote controller 200 A bookmark selection signal can be input. Then, the video display device 100 plays the content according to the first bookmark, and starts playback from the bookmarked portion. That is, according to the reproduction time information included in the bookmark setting information of the first bookmark, the corresponding portion of the contents is reproduced. In accordance with the bookmark setting information stored at the time of creating the bookmark, it is determined which electronic device the corresponding content is stored in. Here, the content corresponding to the first bookmark 1101 is stored in the electronic device 1. [ The image display apparatus 100 receives or downloads the content corresponding to the first bookmark 1101 from the electronic apparatus 1 and reproduces the portion specified by the bookmark setting information. Referring to FIG. 11B, the image corresponding to the reproduction of the content corresponding to the first bookmark 1101 is displayed on the display unit 180 of the image display apparatus 100.

In the video display device 100 shown in FIG. 12, one piece of content is being played back, and there is a pre-created bookmark list for the content. The augmented remote control apparatus 200 displays a bookmark list image 1210 showing a video of the image display apparatus 100 and a pre-generated bookmark list together on the display unit 260. [ That is, information on the image display device 100 recognized by the augmented remote control device 200 and the content being reproduced is provided according to the augmented reality technology, wherein information on the content is a list of pre-generated bookmarks for the content .

According to the embodiment described with reference to FIG. 12, a plurality of bookmarks are generated for one content. Therefore, the first bookmark 1210, the second bookmark 1220, the third bookmark 1230, and the third bookmark 1230 are displayed on the bookmark list image 1200 displayed on the display unit 260 of the augmented remote controller 200 of FIG. Four bookmarks 1240 are included. These bookmarks are all bookmarks for the same contents, and the bookmark creation time or the length of the bookmarked portion may be different.

The user can input a bookmark selection signal for a bookmark of one of the first bookmark 1210, the second bookmark 1220, the third bookmark 1230 and the fourth bookmark 1240. [ Then, the control unit 280 of the augmented remote control apparatus 200 generates a remote control signal corresponding to the bookmark selection signal and transmits the remote control signal to the image display apparatus 100. Then, the video display device can reproduce the content according to the bookmark from the bookmarked portion. Accordingly, the user can extract a desired portion of the content and watch it using the bookmark.

Referring to FIG. 13, an image according to content reproduction is displayed on the image display apparatus 100, and the augmented remote control apparatus 200 recognizes the image display apparatus 100 and the content being reproduced. The image 1000 of the image display apparatus 100 is displayed on the display unit 260 of the augmented remote controller 200 as the recognition process is performed.

Then, the augmented remote controller 200 displays the bookmark list for the corresponding content on the display unit 260 as the content is recognized. The bookmark list includes a first bookmark 1301, a second bookmark 1303, and a third bookmark 1305.

When the user wants to reproduce a portion corresponding to the third bookmark of the content, the user can input a bookmark selection signal for the third bookmark 1305. [ According to the embodiment shown in Fig. 13, the user wants to reproduce the content of the portion corresponding to the third bookmark 1305 in the external device 30. [ In this case, the augmented remote control device 200 receives a user signal for designating an electronic device separately from the bookmark selection signal, or receives a bookmark selection signal including information for designating the electronic device. Therefore, when generating the remote control signal according to the bookmark selection signal, the augmented remote control device 200 can acquire the bookmark setting information such as the reproduction time information and the content identification information as well as the information specifying the electronic device to reproduce the content according to the bookmark And transmit the remote control signal to the designated electronic device.

13, the augmented remote control device 200 transmits a remote control signal including bookmark setting information corresponding to the third bookmark 1305 and information designating the external device 30 to the external device 30 ). Then, the external device 30 reproduces the content according to the third bookmark in accordance with the remote control signal. The external device 30 can receive the content from the video display device 100 or the home server to reproduce the content according to the third bookmark.

There are various ways in which electronic devices to reproduce bookmarked contents are selected. Here, the case where a user selects an electronic device will be described as an example. For example, when the user wishes to reproduce content according to the third bookmark 1305 through the external device 30, the user can set the external remote control device 200 together with the video display device 100 . The augmented remote control apparatus 200 recognizes the external device 30 and displays the image 1300 taken on the external device 30 on the display unit 260. [ The display unit 260 of the augmented remote control device 200 displays the image 1000 captured by the image display device 100 and the bookmarks 1301 1303 and 1305 and the image 1300 captured by the external device 30 ) Are displayed together.

In this case, when the user inputs a bookmark selection signal, the user selects both the third bookmark 1305 and the image 1300 taken by the external device 30. For example, when the display unit 260 of the augmented remote controller 100 has a touch screen function, the user drags and drops the third bookmark 1305 to the image 1300 of the external device 30 drag and drop. Accordingly, the augmented remote control device 200 senses the bookmark selection signal and transmits the remote control signal corresponding to the bookmark selection signal to the external device 30. [ The external device 30 reproduces the content according to the third bookmark according to the remote control signal received from the augmented remote controller 200 and displays the image.

Meanwhile, the present invention can be embodied as a code readable by a processor in a processor-readable recording medium provided in an image display apparatus. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium readable by the processor include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also a carrier wave such as transmission over the Internet. In addition, the processor readable recording medium may be distributed over networked computer systems so that code readable by the processor in a distributed manner can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

10: Broadcast station 30: External device
40: home network 50: home server
100: Image display device 200: Augmented remote control device
300: Network server

Claims (20)

A method of operating an augmented remote control self-governing system,
Recognizing a video display device;
Capturing a content being played back on a video display device;
Generating a preview image corresponding to the photographed contents;
Receiving a bookmark setting signal for generating a bookmark for the preview image; And
And storing bookmark setting information corresponding to the bookmark setting signal.
The method according to claim 1,
The bookmark setting information may include at least one of information for identifying the bookmarked content, information for identifying the video display device from which the content is reproduced, and reproduction time information for inputting the bookmark setting signal Wherein the operation of the augmented remote control device is performed by the remote control device. The method according to claim 1,
And displaying a list of bookmarks, which is a list of the bookmarks generated according to the bookmark setting signal. The method of claim 3,
Wherein the video display device displays an image according to the reproduction of the content,
Capturing an image displayed on the image display device when the bookmark setting signal is input; And
Further comprising generating a bookmark image using the captured image,
Wherein the bookmark list includes the bookmark image. The method of claim 3,
Receiving a bookmark selection signal for one or more of the bookmarks from the bookmark list;
Transmitting a remote control signal for reproducing the content corresponding to the bookmark selected according to the bookmark selection signal to the image display apparatus,
Wherein the video display device reproduces the bookmarked content and displays the reproduced content as an image when receiving the remote control signal. 6. The method of claim 5,
Wherein the image display device reproduces a portion corresponding to a point in time after the bookmark setting signal is input from the bookmarked contents upon receiving the remote control signal. The method of claim 3,
Receiving a bookmark selection signal for one or more of the bookmarks from the bookmark list;
Selecting an electronic device to reproduce the content corresponding to the bookmark selected in accordance with the bookmark selection signal; And
Further comprising transmitting a remote control signal for reproducing the bookmarked content to the electronic device,
Wherein the electronic device reproduces the bookmarked content and displays it as an image when receiving the remote control signal. 8. The method of claim 7,
Wherein the remote control signal includes at least one of information for identifying the bookmarked content, information indicating a position at which the content is stored, and playback point-in-time at which the bookmark setting signal is input. A method of operating a control device. 9. The method of claim 8,
Wherein the electronic device searches the image display device or another electronic device in which the content corresponding to the bookmark is stored using the remote control signal. 9. The method of claim 8,
Wherein the electronic device accesses a server storing the content corresponding to the bookmark via a wired / wireless communication network and searches the content corresponding to the bookmark using the remote control signal. How it works. An identification unit for capturing the content being played back on the video display device;
A control unit for generating a preview image corresponding to the photographed contents;
A user interface unit for receiving a bookmark setting signal for generating a bookmark for the preview image;
A storage unit for storing bookmark setting information corresponding to the bookmark setting signal; And a remote control unit for controlling the remote control unit.
12. The method of claim 11,
The bookmark setting information may include at least one of information for identifying the bookmarked content, information for identifying the video display device from which the content is reproduced, and reproduction time information for inputting the bookmark setting signal Characterized by a remote control device. 12. The method of claim 11,
And a display unit for displaying a bookmark list which is a list of the bookmarks generated according to the bookmark setting signal. 14. The method of claim 13,
Wherein the video display device displays an image according to the reproduction of the content,
The controller captures an image displayed at the time when the bookmark setting signal is input, generates a bookmark image using the captured image,
Wherein the bookmark list includes the bookmark image. 14. The method of claim 13,
Wherein the user interface unit receives a bookmark selection signal for one or more bookmarks of the bookmark list and the control unit generates a remote control signal for reproducing the content corresponding to the bookmark selected according to the bookmark selection signal,
And a wireless communication unit for transmitting the remote control signal to the video display device,
Wherein the video display device reproduces the bookmarked content and displays it as an image when receiving the remote control signal. 16. The method of claim 15,
Wherein the image display device reproduces a portion of the bookmarked content corresponding to a point in time after the bookmark setting signal is input upon receiving the remote control signal. 14. The method of claim 13,
Wherein the user interface unit receives a bookmark selection signal for one or more bookmarks of the bookmark list and the controller selects an electronic apparatus to reproduce the content corresponding to the bookmark selected according to the bookmark selection signal,
And a wireless communication unit for transmitting a remote control signal for reproducing the bookmarked content to the electronic device,
Wherein the electronic device reproduces the bookmarked content and displays it as an image when receiving the remote control signal. 18. The method of claim 17,
Wherein the remote control signal includes at least one of information for identifying the bookmarked content, information indicating a position at which the content is stored, and playback point-in-time at which the bookmark setting signal is input. Control device. 19. The method of claim 18,
Wherein the electronic device searches the image display device or another electronic device in which the content corresponding to the bookmark is stored using the remote control signal. 19. The method of claim 18,
Wherein the electronic device accesses a server storing the content corresponding to the bookmark via a wired / wireless communication network, and searches the content corresponding to the bookmark using the remote control signal.

Images