JP2005341041A - Remote controller, remote control terminal device, and device to be remotely controlled - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize a further wider range of remote control executed by a household network. <P>SOLUTION: A DTV 110 receives first operation information for controlling a VCR 100 from the VCR 100 connected via an IEEE 1394 network and generates second operation information, on the basis of the first operation information. The DTV 110 transmits the second operation information to a mobile PC 140 connected via the Internet. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique capable of remotely controlling a remote control apparatus via a network.

  Conventionally, there has been an increasing demand for connecting home audio / visual devices and home appliances that have been operating independently or operated by a unique connection for each type of device through a single network. Network communication that enables high-speed digital communication technology to transfer data that requires real-time performance such as moving image data and audio data, and data that requires communication reliability such as still image data and control data. Has realized this.

  The device that provides the user interface as the center of the home network is a television, and the remote operation screen of the connected device is displayed on the television screen, and the device is controlled by inputting the user's operation with the remote control device of the television. A so-called remote control service is provided, but a program for executing device control in accordance with display of an operation screen or user operation is required for each device. Since there are a wide variety of devices connected to the network and models with new functions appear over time, it is difficult to prepare a control program for all devices that may be connected on the TV side.

  As one approach for solving these problems, Patent Document 1 discloses the following invention. That is, the image data of the operation panel is transmitted from the controlled device connected to the network, and the remote control device equipped with a display device represented by a television receives the data, and displays the operation panel image to the operator. A system for controlling a remote control device by transmitting an operation input of an operator to the remote control device at the same time as displaying on a device is disclosed.

  The device that provides the user interface as the center of the home network is a television, and the remote operation screen of the connected device is displayed on the television screen, and the device is controlled by inputting the user's operation with the remote control device of the television. A so-called remote control service is provided, but a program for executing device control in accordance with display of an operation screen or user operation is required for each device. Since there are a wide variety of devices connected to the network and models with new functions appear over time, it is difficult to prepare a control program for all devices that may be connected on the TV side.

  As one approach for solving these problems, Patent Document 1 discloses the following invention. That is, in Patent Document 1, image data of an operation panel is transmitted from a controlled device connected to a network, and a remote control device including a display device typified by a television receives the image data to the operator. A system is disclosed in which an operation panel image is displayed on a display device and at the same time an operation input from an operator is transmitted to the controlled device to control the controlled device.

Japanese Patent Laid-Open No. 11-3314

  However, the remote operation based on the remote control display data sent from the remote controlled device has a great feature that any specific function of the remote controlled device can be operated by displaying it as remote display data. However, there is a demand for controlling the convenience of such remote control not only in the home but also from a remote location, for example, from outside.

  The network of IEEE 1394 AV equipment used as a home network has restrictions on the number of connected devices and the distance of connected cables, and the reachable range of the network is limited.

  On the other hand, an IP-based network crosses various physical connection means such as Ethernet (registered trademark), ATM wide area network, and telephone network, which are widely used in office environments, and can be connected over a very wide range. Network that is widely used as social infrastructure. In particular, the so-called Internet using the HTTP protocol can be connected not only to PCs, PDAs connected via mobile phones and PHS, but also to mobile phones themselves. It is one of the familiar communication means that can be used regardless.

  Although these connected devices have a common function of connecting to the Internet, their display capabilities and operability are greatly different in order to make the characteristics of each device. For example, a PC has a relatively high-performance display unit with a large number of pixels and colors, and has free operability by pointing in addition to a keyboard. On the other hand, a cellular phone that can be connected to the Internet has only a display device with a limited number of display pixels and a simple operation system limited to up / down / left / right direction keys or numeric keys.

  The operation environments that are easy to use in the display system and the operation system of these devices are different. For example, on a PC, a graphical icon is displayed rather than a text menu selection, and the method of selecting directly with a mouse pointer is intuitive and easy to use. In addition, the small display device of the mobile phone has good display of text, and selection by the up / down direction keys may be suitable.

  The difference in the appropriate operating environment for each display system and operating system is large, and it is difficult to cover much with one operating environment provided by the remote controlled device. A mechanism that absorbs the difference in display capability and operability of various connected devices is required.

  Therefore, an object of the present invention is to make remote control by a home network wider.

  The remote control device according to the present invention comprises: operation information receiving means for receiving first operation information for controlling the remote control device from a remote control device connected via a first network; Operation information generation means for generating second operation information based on one operation information, and operation information transmission for transmitting the second operation information to a remote control terminal connected via a second network A control information receiving means for receiving, from the remote control terminal device, first control information for the remote control device corresponding to the second operation information, and a second based on the first control information. Control information generating means for generating the control information, and control information transmitting means for transmitting the second control information to the remote control apparatus.

  The remote control terminal device of the present invention receives first operation information for controlling the remote control device from a remote control device connected via a first network, and the first operation information Operation information receiving means for receiving the second operation information from the remote control device that generates the second operation information based on the second network via the second network, and the remote target according to the second operation information Control information generating means for generating first control information for the control device, second control information is generated based on the first control information, and the second control information is transmitted to the remote control device. Control information transmitting means for transmitting the first control information to the remote control device.

  The remote control device of the present invention generates operation information generation means for generating first operation information for controlling the remote control device, and generates second operation information based on the first operation information. An operation information transmitting means for transmitting the first operation information to a remote control device that transmits the second operation information to a remote control terminal device via a first network; and the second operation information. The second control information generated by the remote control device based on the first control information when the remote control device receives the first control information for the remote control device corresponding to the first control information via the second network. Control information receiving means for receiving the information via the first network.

  The remote control system of the present invention includes the remote control device, the remote control terminal device, and the remote control device.

  The remote control device control method of the present invention includes an operation information receiving step of receiving first operation information for controlling the remote control device from a remote control device connected via a first network. The operation information generation step for generating the second operation information based on the first operation information, and the second operation information is transmitted to the remote control terminal device connected via the second network. Based on an operation information transmitting step, a control information receiving step for receiving, from the remote control terminal device, first control information for the remote control device corresponding to the second operation information, and the first control information A control information generating step for generating second control information, and a control information transmitting step for transmitting the second control information to the remote control device; Characterized in that it contains.

  The remote control terminal device control method of the present invention receives first operation information for controlling the remote control device from a remote control device connected via a first network, and An operation information receiving step for receiving the second operation information via a second network from a remote control device that generates second operation information based on the operation information of the operation information, and according to the second operation information A control information generating step for generating first control information for the remote control device; and generating second control information based on the first control information and supplying the second control information to the remote control device. And a control information transmission step of transmitting the first control information to the remote control device that transmits the information to the remote control device.

  The method for controlling a remote control device according to the present invention includes an operation information generation step of generating first operation information for controlling the remote control device, and second operation information based on the first operation information. The operation information transmission step of transmitting the first operation information via a first network to a remote control device that generates the second operation information to the remote control terminal device; and Second control information received by the remote control device via the second network and generated by the remote control device based on the first control information. Control information receiving step of receiving the control information via the first network.

  According to a first aspect of the program of the present invention, a computer executes the control method of the remote control device.

  According to a second aspect of the program of the present invention, a computer is caused to execute the control method of the remote control terminal device.

  A third aspect of the program of the present invention causes a computer to execute the control method of the remote control device.

  The computer-readable recording medium of the present invention is characterized in that the program is recorded.

  According to the present invention, the remote control terminal device acquires the operation information of the remote control device via the first and second networks, and transmits the control information of the remote control device according to the operation information. Therefore, for example, by applying a network capable of wide-range communication such as the Internet to the second network, remote control by the home network can be further widened.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a remote control system according to an embodiment of the present invention.
This system is connected to a network 1 by a high-speed serial interface, and displays a display for operation on a digital television (DTV) 110 having a user operation input by a remote controller 120 and the digital television 110, and remotely based on the remote control input of the digital television 110. It includes a digital video (VCR) 100 to be controlled, a digital video camera (DV) 131, a tuner 132, a mobile PC 140 connected to the Internet including a public line, a PDA 151, and a portable terminal 152.

FIG. 2 is a diagram illustrating a main configuration of the VCR 100 that is remotely controlled via the digital television 110 and serves as a remotely controlled device.
104 receives a desired broadcast program from a broadcast wave, records a program received by a built-in digital video recorder or a network 1 through a high-speed serial interface and an input from an analog input / output terminal (not shown), and from the digital video recorder A VCR main unit 103 for outputting stream data including video and audio data to be reproduced to the network 1, a display for storing or generating display data (GUI data) and operation information to be displayed on the digital television 110 for remote control A data generator 102 is a control unit that controls the VCR main unit 104 and the display data generator 103 by communicating via a network. Reference numeral 101 denotes a 1394 I / F unit that is an interface of a 1394 high-speed serial bus, and is connected to the 1394 I / F unit of the digital television 110 or the 1394 I / F unit of another device via the network 1.

FIG. 3 is a diagram illustrating a configuration of the digital television 110 that accepts user operation input from the remote controller.
111 is a 1394 I / F unit that is an interface of a 1394 high-speed serial bus, and 113 is a decoder that includes a function of separating stream data received by the 1394 I / F unit 111 into a video stream and an audio stream and decoding each stream data. Reference numeral 119 denotes an audio output unit including D / A conversion and amplification of the decoded audio data and a speaker.

  114 is a display data memory for storing operation data and display data (GUI data) for remote control received by the 1394 I / F unit 111 and HTML operation page data converted from the GUI data; 116 is infrared or A remote control unit 115 for inputting a user's operation using wireless or the like, 115 is a display image (operation for remote control) of a remotely controlled device connected based on display data (GUI data) for remote control. A drawing control unit 117 that generates a panel), an image synthesis unit 124 that synthesizes the video data output from the decoder 113 and the display image output from the drawing control unit 115 and displays them on the display device 118; The GUI data stored in the display data memory 114 is stored in a plurality of HTML operation panel data. The operation data conversion unit 123 converts the data into the display data memory 114 and stores the operation page data in HTML in response to a request received through the network interface 123. The server unit 126 that extracts the user's operation content from the received request message 126 displays the ID of the button element selected by the user for display on the mobile terminal for each transmission source device of the panel being edited on the panel editing screen. An edit control unit 125 that creates a later-described edit list to be stored and holds the list in an edit data memory 125 described later when a memory store signal is received from the control unit 112, is edited by an instruction from the edit control unit 126. The edit data memory 112 for storing the list for use is 1 394 I / F unit 111, decoder 113, display data memory 114, edit control unit 126, edit data memory 125, remote control unit 116, 1394 I / F unit 111, and control unit that controls the drawing control unit 115 and connected to the network In addition to receiving the stream data transmitted from the VCR 100 and outputting it to the display device 118 and the audio output unit 119, the display for remote control displayed on the display device 118, the remote control operation input, and the extraction from the server unit 122 The operation of the entire digital television 110 that controls the remote control device connected to the 1394 I / F unit 111 is integratedly controlled based on the operation input.

  In addition, the control unit 112 will be described in terms of well expressing the features of the present embodiment. When the user selects start of the panel editing menu and the menu key 320 and the selection and determination key operation signal are input from the remote control unit 116, the panel editing screen as shown in FIG. The panel editing screen is transmitted to the image composition unit 117.

  In this panel editing screen, when a determination key 316 operation signal is input from the remote control unit 116 to select a button to be displayed on the mobile PC 140, which will be described later, an image for coloring the button is displayed to indicate that the button is selected. The data is transmitted to the combining unit 117. Then, the control unit 112 passes a vendor ID and chip ID (to be described later) of the device that has transmitted the panel being edited to an element control unit (to be described later) of the previously selected button to the editing control unit 126 described above.

  Next, when the user selects the end of the panel editing menu and a menu key 320 operation signal (for example, a TV mode switching signal) is input from the remote control unit 116, the panel editing screen is deleted and the editing control unit 126 is displayed. A memory store signal for holding an edit list to be described later is sent.

  FIG. 4 is a diagram showing a configuration of a mobile PC 140 that is connected to the Internet 2 and that serves as a remote control terminal having a function of displaying operation page data in HTML received from the DTV 110 and transmitting a user operation based on the display to the DTV 110. .

  The PC generally includes a CPU that controls the entire control, a storage unit connected to the CPU via a bus, a drawing control unit, a display unit, an operation input unit, and an Internet I / F. In this embodiment, A browser function is required that receives HTML data as application software executed by the CPU, interprets the data, and displays the data.

  In addition, a function for outputting a user input to the display screen through the Internet I / F as an operation to HTML data is required. FIG. 4 shows the configuration of such a function, which includes an Internet I / F 141 using Ethernet (registered trademark), an operation input unit 142, a browser unit 143, a display unit 144, and a control unit 145 using a CPU.

  The PDA 151 and the mobile phone 152, which are other remote control terminals shown in FIG. 1, also have components responsible for operations unique to the respective devices, as well as the functions responsible for the remote control terminal described in the present embodiment. As an element, it has the same structure as FIG.

  The 1394 I / F (IEEE 1394 high-speed serial bus) is a network bus that supports a daisy chain method, a node branching method, and a combination thereof, and enables a highly flexible connection. In a data transfer method in which serial transfer is performed at 100 Mbps, 200 Mbps, and 400 Mbps, devices having higher transfer rates support lower data transfer rates, and thus devices having different transfer rates can be mixed.

  Asynchronous transfer for the transfer of asynchronously transferred data such as control signals such as commands and file data, and transfer of data that requires temporal continuity such as video data and audio signals (so-called stream data) Has two transfer modes of isochronous transfer in which each is sequentially performed every predetermined period (125 us).

  AV / C (Audio Video Control) commands for controlling AV devices are constructed on the Asynchronous transfer, and at the same time, models of AV devices controlled by these commands are defined as AV / C units or subunits. In addition, for data transfer that requires high reliability and high speed, such as still image file transfer and print data transfer, Asynchronous Serial Bus Connection, which is a protocol based on Asynchronous transfer, is connected to or eliminated from this connection. Procedures and control commands are defined and used.

  Stream data transfer by isochronous transfer includes image / audio data transfer in DV format and image / audio stream transfer in MPEG2 format. A procedure and a control command for establishing or canceling a connection for transferring these stream data are also defined as AV / C commands by Asynchronous transfer.

  The server unit 122 and the browser 143 connected to the IP-based network 2 transfer HTML data, image data, and the like as response messages from the server in response to request messages from the browser on the TPC and HTTP protocol stacks. To perform the operation.

  FIG. 5 is a diagram for explaining the transfer and display of display data (GUI data) for remote control of the remote control system and the procedure of remote control, focusing on communication on the 1394 network and the Internet. In the following description, the procedure with the VCR 100, which is one of the remotely controlled devices on the 1394 network, is described, but the same communication is performed when another remotely controlled device is connected to the network. In addition, the same control can be performed in parallel with the VCR 100.

  In FIG. 5, the control unit 112 of the digital TV first transmits a SubunitInfo command based on the AV / C digital Interface Commad Set specification (AVC specification) to the devices connected to the 1394 network first, and the device is provided. The function is investigated (201). If the device has a remote control function, the necessary contents in the configuration ROM standardized by IEC1212 (ANSI / IEEE Std 1212 Control and Status Register (CSR) Architecture for Microcomputer Busses) are read (202). Acquire information about the device, image data of a device icon representing the device, and the like.

  Here, the node vendor ID (16 bits) and the chip ID (upper and lower bits are 40 bits) are acquired as information about the device. Since these 7-byte vendor ID and chip ID values are unique to the device, the device can be specified. Therefore, the control unit 112 can recognize a transmission source device of a signal such as a command received from the 1394 bus. The control unit 112 uses these 7-byte values to create a list of devices that can be operated by the user (hereinafter referred to as a device list). The device list and the device icon group are stored in the display data memory 114, and at the same time, the device icon group is displayed on the display device 116 through the drawing control unit 115 to prompt the user to select an operation target. However, the control unit 112 can refer to this device list.

  Next, when a device icon corresponding to the VCR 100 is selected as one of these device icons by the user's remote control operation, the control unit 112 of the digital television starts a session using the VCR 100 as a remote control device. An AV / C command Gui_update (open) command is issued (205). FIG. 6A shows a command frame of the Gui_update command. The instruction to the remote control device by this command is specified by the value of subfunction (FIG. 6B) stored in operand [3]. open indicates the start of a remote control session, and close indicates the end of the session. start instructs transfer of GUI data for remote control. The remote control apparatus transmits the GUI data to the control apparatus according to the procedure described later when the GUI data is changed due to the state of the apparatus or the operation result until the next stop instruction is given. “restore” is designated when the Asynchronous serial bus connection for GUI data transfer is restored when a bus reset of the 1394 network occurs. “change” is specified when the change of the reporting range of the GUI data change is specified.

  The control unit 102 of the VCR 100 that has received the Gui_update (open) command stores the plug number to be used for Asynchronous serial bus connection in the operand [0] as a response to the command and returns it. The control unit 112 of the digital television uses this plug number to establish a necessary internal connection of the VCR 100 and a connection between the VCR 100 and the DTV 110 by issuing a Connect command (206) and an Allocate_attach command (207).

  Subsequently, the control unit 112 of the digital television issues a Gui_update (start) command to instruct the start of GUI data transfer (208), and issues a Push_gui_data command to instruct the transfer of GUI data of the VCR 100 (209). ).

  FIG. 6C shows a command frame of the Push_gui_data command. In operand [0], specify the plug number acquired by Gui_update (open). The indicator of operand [5] stores the level and with_data designation shown in FIG. The level designates the transfer range of the GUI data of the remotely controlled device that is transferred by the instruction by the remote control Push_gui_data command according to the tree structure of GUI data described later. The value to be specified is a 3-bit 2-bit value shown in FIG. 6A. In 00 (itself), an element specified by an element ID described later is specified, and in 01 (itself and next level), an element ID is specified. An element and an element belonging to the element are designated in a range of 01 (itself and next level) in 11 (all level) and up to a next element belonging to an element included in the range.

  With_data specifies actual data such as text, image, and sound of the element specified by the element ID and level when this bit is 1. In 4 bytes from operand [6], an element ID for identifying an element of GUI data to be described later is stored, and GUI data for instructing transfer through Asynchronous serial bus connection is specified, but a panel element as a root is specified. Only in some cases, the element ID can be substituted with FFFFh.

  The control unit 102 of the VCR 100 that has received the Push_gui_data command returns a response to the command, and transfers display data for remote control from the panel data generator 103 of the VCR 100 to the digital television 110 through the asynchronous serial bus connection (210). The transferred display data is stored in the display data memory 114, read out element by element by the control unit 112, sent to the drawing control unit 115, displayed on the display device 118 through the image composition unit 117, and operation information for the user Provided as (operation panel).

  The display data is a set of elements displayed as operation information and has a tree-structured data structure with a panel element as a root. FIG. 7 is a diagram showing such elements and essential data or selection data subordinate to each element, and FIG. 8 is a diagram showing user actions that are operations that the user can take on each element.

  The panel element indicates a set of operation elements, and holds a column of IDs that each element has as a link. In addition, it has a link to a label, which is character string data serving as a title of the panel element, and position and size data on the display screen displayed by the panel. Also, it is possible to retain options for specifying the background color of the panel, linking to bitmap data displayed on the background, and selecting fonts for the character string to be displayed. In addition, isochronous transfer channel data for transferring video and audio streams can be stored as an option. When this option is specified, audio video data decoded from the transferred stream is displayed on the entire panel element. To do.

  The button element is an element that the user can press or release with the remote control, and holds a link to two bitmap data to be displayed on the key top when pressed and when released. . The size of the bitmap is indicated by size data, and the display position is indicated by position data. The position data is obtained by designating the relative position in the panel element to which this element belongs in pixel units. In addition, a link to a label serving as a title (or functional description) of a button element can be held, and font options for a character string to be displayed can be held as an option. Focus navigation that can be held as an option will be described later.

  The text element is an element that displays the title and description as a character string, and holds the position, size, and the link to the text data that becomes the label, and the background color specification, font choices, and focus navigation as options. I can do things.

  The icon element is an element that displays a bitmap and can be selected by the user, holds a link to the bitmap data, a label, a position, a size and a link to the text data that becomes the label, a font option, Audio / visual specification and focus navigation can be held as options.

  The animation element is an element that can hold a plurality of bitmaps, switch them in order at regular time intervals, and display them so that the user can select them. It holds the position, size, and the link to the text data that is the label, and can hold font options and focus navigation as options.

  FIG. 9A, FIG. 9B, and FIG. 9C are diagrams illustrating a data structure of elements that constitute GUI data. FIG. 9A shows a data structure of element data such as a panel element and a button element. All elements hold element IDs for identification in the display data. The element type is a code indicating the type of the element, and the element type shown in FIG. The element data stores the data held by this element in the order of essential data and selection data.

  FIG. 9-1 (b) shows a data structure of actual data such as text, bitmap, and sound linked from each element as essential data or selection data. All real data holds element IDs for identification as well as elements. The element type is a code indicating the type of actual data, and stores the data type shown in FIG. If the actual data is text data, character string data is stored, if bitmap data is stored, a bitmap data string is stored, and if sound is stored, a sound data string is stored.

  FIGS. 9-1 (c), (d), FIGS. 9-2 (a) to (f), and FIGS. 9-3 (a), (b) are essential data or selection data held by the element data. A linked data structure of elements or data. FIG. 9-1 (c) shows a link to the actual text data, and the element ID of the actual text data at the link destination is stored in the element ID. FIG. 9-1 (d) is a link to bitmap actual data, and the element ID of bitmap actual data is stored in the element ID. FIG. 9-2 (a) is a link to an element. The element ID stores the element ID of an element such as a button element or text element belonging to the panel element, and the element type is the element shown in FIG. 10 (a). Stores the type. FIG. 9B is a data structure of color data designated as a background color, and stores color data expressed by 8 bits for each of RGB.

  FIG. 9-2 (c) shows the data structure for designating the display position of the element. The position of each element is expressed in a two-dimensional coordinate system on the display device as a relative position with the upper left corner of the upper element to which the element belongs as the origin. That is, the position on the display device is specified in units of pixels when the element is the root panel element, and the position on the panel element is specified if the element belongs to the panel element.

  FIG. 9-2 (d) shows a data structure for designating a size for displaying an element, and “width” and “height” store values expressing the size of the element in units of pixels. FIG. 9-2 (e) shows a data structure for designating the font size of the text displayed by the element, and the font code shown in FIG. 10 (d) is stored.

  FIG. 9-2 (f) shows a data structure in the case where selection data is stored in element data of an element. The selection data type shown in FIG. 10C is stored in “selection data type”, and “selection data”. A data structure to be selectively added, such as background color and font, is stored.

  FIG. 9-3 (a) shows a data structure for designating an isochronous channel for transferring audio video data displayed on the background of the panel element or the icon element. The isochronous channel is stored in the lower 6 bits, and the upper 2 bits are both. 00B is set.

  FIG. 9-3 (b) shows a data structure for designating an element to which the focus is to be moved when there is a focus movement in each of the upper, lower, left, and right directions when the element has the focus and a selection input for this element.

  As shown in FIG. 9-3 (b), when the focus movement destination is input in the four directions of up, down, left and right as the element ID, and when there is an operation input for selecting an element. The focus destination is stored as an element ID. When FFFFh is stored as the element ID of the focus movement destination, or when its own element ID is designated, the focus is not moved even if there is a corresponding operation input.

  FIG. 11 is a diagram showing an example of a data structure in display data for remote control in a tree structure. FIG. 12 is a diagram showing the data structure of the panel element in the data structure. Such data is transferred to the digital television 110 through the asynchronous serial bus connection.

  FIG. 13 shows a display example (operation panel) of display data for remote control. The control unit 112 displays such a display on the display device 118 through the drawing control unit 115, and accepts a user operation input through the remote control unit 116. For example, in FIG. 13, the button highlight display 313 indicates that the PLAY button 304 indicating reproduction of the VCR 100 has a focus as a user selection candidate, and the focus position is changed in response to a remote control operation. This operation is called “farcus navigation”. In addition, in response to a remote control operation to determine and select the focused button, the button display is switched from the released bitmap display to the depressed bitmap display.

  When detecting an allowed user action for each display element, such as pressing a button, the control unit 112 of the digital television issues a User_Action command and notifies the control unit 102 of the VCR 100 (213 in FIG. 5). The control unit 102 performs control corresponding to a user action predetermined for each notified display element on the VCR main body unit 104. The display data generator 103 reflects the change in the operating state of the VCR main body 104. When the display data changes, the display data having changed is transferred to the digital television 110 by the asynchronous serial bus connection (FIG. 5). 214). Depending on the control to the VCR main body 104, transfer of new stream data is started through an isochronous connection (216). A series of operations for performing focus navigation and notifying the remote control apparatus of user operation inputs as user actions is called user interaction.

  FIG. 14 is a diagram showing a remote controller 120 used for user interaction. The remote controller 120 includes direction keys 312, 313, 314, and 315 for instructing movement of focus on display data and a determination key 316 for instructing selection of a display element with focus. When the display element is a button element, the selection decision instruction generates a button depression (and release) user action, and when the display element is an icon or animation element, a selection user action is generated.

  14 also includes a power on / off key 317, a volume up key 318, a volume down key 319, and a menu key 320. Operations on these keys are always operations on the DTV 110, notified to the control unit 112 through the remote control unit 116, and passed to power on / off of the DTV 110, volume adjustment, and menu call control of the DTV 110.

  Next, the remote control operation from a remote location via the DTV 110 by the mobile PC 140 connected to the Internet 2 will be described. In the DTV 110, the panel data received from the remote control device is stored in the display data memory, and the HTML data to be transmitted to the remote control terminal connected through the Internet 2 is created and held based on the panel data. To do.

  In FIG. 5, the DTV 110 displays a list of device that can be operated by the user and a device icon obtained by issuing a SubunitInfo command (201) and reading to the configuration ROM (202) for each device connected to the IEEE 1394 network. The operation page data for device selection by HTML is created by the operation data conversion unit 124 and stored in the display data memory 114 as devicelist.html. As a response message to the access (203) by the get message to the server unit 122 of the DTV 110 from the mobile PC 140, the operation page data by HTML is transferred to the browser unit 143 of the mobile PC 140 (203) and displayed on the display unit 144. FIG. 15 shows an example of operation page data for a device selection page by HTML, and FIG. 16 shows an example of an operation page displayed on the display unit 144 of the mobile PC 140.

  When the user selects one of the icons representing devices that can be remotely operated on the display operation page, the server for receiving the get message (204) is received by the program for generating the panel of the device designated as the link on the operation page The remote control procedure is started by issuing the Gui_Update command (205) to the target device through the control unit 112.

  When the GUI data (210) sent from the device is stored in the display data memory 114, it is converted into operation page data by HTML by the operation data conversion unit 124 and stored in the display data memory 114. The program of the server unit 122 transmits operation page data to the browser unit 143 of the mobile PC 140 as a response to the get message (204) (211) and is displayed on the display unit 144 as the operation page.

  FIG. 17 is an example of the displayed operation page. Although the mobile PC 140 is small, it has a keyboard and pointing device and a liquid crystal display capable of displaying color graphics, and a graphics display element such as an icon displayed on the display can be freely selected using a mouse cursor. In the operation page, an operation panel based on GUI data received by the DTV 110 is reproduced as it is by displaying color graphics including an image.

  FIG. 18 is a diagram showing an example of a conversion rule when converting operation panel data by HTML used in a remote operation terminal having such display capability and operability from GUI data. A display element having image data such as a button element or an icon element can use the image data of the GUI data as it is. In the button element, image data by an open bitmap link that is an image in a state where the button is released is used, and in the animation element, the first image data pointed to by the animation bitmap link constituting the animation is used. If the operation data conversion unit 124 has the function, it can be converted into a GIF file including an animation.

  In addition, information on the position and size specified for each element is also specified in the operation panel data by HTML. A background color and a background bitmap given to a panel element and a text element are also designated as selection data of GUI data. In the present embodiment, other essential data and selection data are not reflected in the operation page data by HTML.

  When the user of the mobile PC 140 looks at the displayed operation page and selects a display element indicating a desired function, the processing program program defined for each element type designated as a link receives the get message (212). The user action command is issued from the control unit 112 based on the element ID and the action code which are started by the unit 122 and passed as arguments (213).

  When GUI data newly transmitted in response to a UserAction command from a device that is a remote control device is received by the DTV 110 and stored in the display data memory 112, the operation data conversion unit 124 converts it into operation page data by HTML. And stored in the display data memory 114. The program of the server unit 122 transmits new operation page data to the browser unit 143 of the mobile PC 140 as a response to the get message (212) (215), and is displayed on the display unit 144 as a new operation page.

  There are various types of remote control terminals connected to the present embodiment, and their operation environments are also greatly different. FIGS. 21, 22, and 23 are examples showing the appearance of a mobile PC (140 in FIG. 1), PDA (151 in FIG. 1), and portable terminal (152 in FIG. 1), respectively.

  Although the mobile PC 140 is small, it has a keyboard and pointing device and a liquid crystal display capable of displaying color graphics, and a graphics display element such as an icon displayed on the display can be freely selected using a mouse cursor.

  The PDA 151 is connected to a mobile phone / PHS (not shown) and is connected to the Internet via a public line. A black-and-white or color small area display is provided, and operations can be performed through a touch panel provided on the display surface. FIG. 24 is a display example of an operation page displayed on a remote terminal such as a PDA. FIG. 24A is a list of devices that can be remotely controlled, and is a screen for selecting a device to be remotely controlled. FIG. 24B is a diagram for VCR control when the VCR 100 is selected as a remotely controlled device. It is a display screen and is an example in which operation page data in HTML converted from the GUI data shown in FIG. 13 is displayed.

  In addition to the functions of a mobile phone, the mobile terminal includes a limited display mainly suitable for displaying text, and is connected to the Internet through a server provided in the mobile phone network. You can select and operate mainly text displayed on the display using the up and down cursor movement keys.

  FIG. 25A is a text list of devices that can be remotely controlled, and is a screen for selecting a device to be remotely controlled. FIG. 25B is a VCR when the VCR 100 is selected as a remote control device. It is an example of a display of an operation page for control, and is an example in which operation page data in HTML composed of text converted from the GUI data shown in FIG. 13 is displayed.

  As described above, the operation environment varies greatly depending on the device connected as the remote control terminal, and the appropriate operation page or operation page data for realizing an environment that is easy for the user to operate on each terminal is different. Therefore, the rules used when converting HTML data from GUI data must be different from the rules for mobile PCs shown in FIG.

  FIG. 19 shows an example of a conversion rule for PDA. Although image data of GUI data is used, since a limited PDA display is premised, designation regarding the display position is not utilized. Also, the size of the image data is not necessarily as specified, but is resized or clipped to an appropriate size and displayed in an arrangement that can display the PDA.

  FIG. 20 is an example of a conversion rule for a portable terminal, and only text data of GUI data is adopted according to the limited display area of the portable terminal and the operation selection means. Elements that are originally displayed as images, such as button elements and icon elements, are also replaced with text based on label data attached to the elements. On the other hand, the displayed operation page has many restrictions, but on the other hand, it is possible to obtain the convenience that the remote control can be performed from anywhere where the mobile terminal can be used.

  In the DTV 110, it is desirable to send out the most appropriate operation page data as a response when connected from any of these remote control terminals. In the DTV 110 according to the present embodiment, when converting information on device connected to the IEEE 1394 network or GUI data to operation page data by HTML,

  A plurality of types of operation page data appropriate for the categorized remote control terminal are created and stored separately in URLs indicating the types of the remote control terminals. FIG. 26 shows a directory structure managed by the server unit 122 that can be accessed from the remote control terminal. Operation page data for a mobile PC, a PDA, and a portable terminal are stored in directories of “pc”, “pda”, and “cellular” having directory names representative of remote control terminal configurations, respectively.

The remote control terminal accesses the server unit 122 of the DTV 110 by designating a directory that is considered to give optimum operation page data as a URL in consideration of its display and operation mode. For example, in the case of the mobile PC 140 described above, the operation page shown in FIG. 16 can be obtained by specifying the URL = “http://123.45.67.89/pc/devicelist.html”. In case of PDA, URL = "http:
By specifying //123.45.67.89/pda/devicelist.html ", the operation page in Fig. 24 (a) can be obtained. In the case of a portable terminal, URL =" http://123.45.67.89/cellular By specifying “/devicelist.html”, the operation page of FIG. 25A can be obtained, and by selecting the character “VCR” from the displayed operation page, the VCR 100 of FIG. 25B is selected. An operation page to be operated can be obtained.

  Next, a description will be given of a case where the operation panel 302 displayed on the display device 118 of the DTV 110 as shown in FIG. 13 which is well characterized in the present embodiment is edited for portable terminal display.

  In FIG. 13, when the menu key 320 of the remote controller 120 is pressed, a menu 500 as shown in FIG. 27 is displayed. Next, when “Start” 401 in the “Operation panel edit” menu 400 is selected, the screen on the display device 118 is changed to a panel edit screen for selecting an operation button to be used from the mobile terminal as shown in FIG. Become. The user operates the direction keys 312 to 315 of the remote controller 120 according to the instruction 411 on the display screen, and the control unit 112 moves the focus 412 to the position of the recording button 307 as shown in FIG. When the enter key 316 is pressed, the control unit 112 recognizes that the enter key 316 has been pressed through the remote control unit 116. Then, an image is sent to the image composition unit 117 in order to add a color as a mark on the element that is focused when the enter key 316 is pressed.

  Next, the control unit 112 extracts the element ID of the focused button from the GUI data, and sends it to the editing control unit 126 together with the vendor ID and chip ID indicating the transmission source device of the panel being edited. However, the vendor ID and the chip ID are obtained by referring to the device list described above.

  The edit control unit 126 creates an edit list as shown in FIG. 30 for storing the selected element ID for a set of vendor ID and chip ID based on the received value. The edit list as shown in FIG. 30 is created by selecting the stop button 306 as an operation button to be used from the portable terminal according to the same procedure.

  In the present embodiment, the record and stop buttons are selected, but thereafter, other operation buttons may be selected using the direction keys and the enter key. In this case as well, the button element ID is added to the edit list of FIG. 30 and stored. Next, in order to end the panel editing, when “end” 421 of the “operation panel editing” menu 500 is selected from the menu key 320, the direction keys 312 to 315 and the enter key 316 of the remote controller 120 in FIG. The control unit 126 uses the memory store signal sent from the control unit 1122 as a trigger, and passes the created edit list to the edit data memory 125 for holding.

  Next, the control unit 112 sends GUI data as shown in FIG. 12 to the display data memory 114. The display data memory 114 refers to the edit list in the edit data memory 125 in the operation data conversion unit 124, and displays only elements having the same ID as the IDs in the edit list from the received GUI data. According to the conversion rules of FIGS. 18 to 20, the operation page data is converted into HTML format, and the previous data is overwritten and saved.

  Thereby, when performing remote control from a remote place, only the button that the user wants to use can be designated and displayed by the user himself, and the operation screen is simplified.

  For example, in the case of the portable terminal described above, an operation page for operating the VCR 100 edited by the user in FIG. 32 can be obtained by selecting the characters “VCR” from the displayed operation pages. In the case of the PDA 151, an operation page as shown in FIG. 33 can be obtained.

  Further, a description will be given of a case where the user performs an operation using the operation data after editing described above. For example, when the recording button on the operation panel displayed on the screen of the mobile terminal is pressed, the DTV 110 issues a UserAction command with the recording button pressed down to the VCR 100 via the server unit 122 according to the same procedure as described above. . After that, GUI data including animation data for recording with element ID = “10h” is received from the VCR 100 in the previously received data. However, in the present embodiment, GUI data different from the previous one is received, but the same GUI data as the previous GUI data may be received.

  Then, the control unit 112 passes this GUI data to the display data memory 114. The display data memory 114 refers to the edit list as shown in FIG. 30 in the edit data memory 125 in the operation data conversion unit 124, and from the newly received GUI data, the ID having the same value as the element ID in the edit list Are converted into HTML-format operation page data in accordance with the conversion rules shown in FIGS. In this case, since the animation data element ID = “10h” is not stored in the edit list, the animation data is not converted into the HTML format.

  As described above, according to the edit list, only the button specified by the user is faithfully converted. Therefore, even if the GUI data increases, the operation page contents can be prevented from being tampered with, and the operation screen is simple. It becomes possible to keep it.

  According to the above embodiment, the remote control device receives and uses data for remote control transmitted from the remote controlled device, so that the control application software for each remote controlled device on the remote control device side. A remote network system that can operate a remote controlled device having any specific function without preparing a network is not only used from within a home network, but also has an extremely wide range of remote functions through an IP network. It can be operated from the ground.

  Also, by using multiple types of typified display data, the remote control terminal can use operation data suitable for different display capabilities and operation environments for each connected remote control terminal. It is possible to improve the operability of the user at.

  In addition, the operation data to be used on the remote control terminal can be limited according to the needs of individual users, the operation screen on the display screen of the remote control terminal becomes simpler, and the user's The operability can be further improved.

  Another object of the present invention is to supply a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the storage medium. Needless to say, this can also be achieved by reading and executing the program code stored in.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (basic system or operating system) running on the computer based on the instruction of the program code. Needless to say, a case where the functions of the above-described embodiment are realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion board or function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows the remote control system which concerns on embodiment of this invention. It is a figure which shows the main structures of VCR which is remotely controlled via a digital video and becomes a remote control apparatus. It is a figure which shows the structure of the digital television which receives the user operation input by a remote control. It is a figure which shows the structure of the mobile PC used as a remote control terminal. It is a figure for paying attention to communication on a 1394 network and the internet about the transfer and display of display data (GUI data) for remote control of a remote control system, and the procedure of remote control. It is a figure for demonstrating the control command for remote control. It is a figure for demonstrating the control command for remote control. It is a figure for demonstrating the element of operation information, its essential data, and selection data. It is a figure for demonstrating the user action which the element of operation information can take. It is a figure which shows the data structure of GUI data. It is a figure which shows the data structure of GUI data. It is a figure which shows the data structure of GUI data. It is a figure for demonstrating the type and code | cord | chord stored in GUI data. It is a figure which shows the example of the tree structure of GUI data. It is a figure which shows the example of data of a panel element among GUI data. It is a figure which shows the example of a display in DTV of the GUI data for remote control. It is an external view of a remote control. It is a figure which shows the operation page data sent to a remote control terminal. It is a figure which shows the operation page displayed on mobile PC which is a remote control terminal. It is a figure which shows the operation page displayed on mobile PC which is a remote control terminal. It is a figure for demonstrating the conversion rule at the time of producing the operation page data by HTML from GUI data. It is a figure for demonstrating the conversion rule at the time of producing the operation page data by HTML from GUI data. It is a figure for demonstrating the conversion rule at the time of producing the operation page data by HTML from GUI data. It is an external view of mobile PC which is a remote control terminal. It is an external view of PDA which is a remote control terminal. It is an external view of the portable terminal which is a remote control terminal. It is a figure which shows the example of a display of the operation page displayed on PDA which is a remote control terminal, and a portable terminal. It is a figure which shows the example of a display of the operation page displayed on PDA which is a remote control terminal, and a portable terminal. It is a figure for demonstrating the directory structure of URL which the server part of DTV manages. It is a figure for demonstrating the start of operation panel editing. It is a figure for demonstrating operation panel editing. It is a figure for demonstrating operation panel editing. It is a figure which shows the list for edit. It is a figure for demonstrating completion | finish of operation panel editing. It is a figure which shows the operation page displayed on a remote control terminal. It is a figure which shows the operation page displayed on a remote control terminal.

Explanation of symbols

100 VCR
101, 111 1394 interface unit 102, 112, 145 Control unit 103 Panel data generator 104 VCR main unit 110 DTV
DESCRIPTION OF SYMBOLS 113 Decoder 114 Display data memory 115 Drawing control part 116 Remote control part 117 Image composition part 118 Display device 119 Audio | voice output part 120 Remote control 122 Server part 123 Network interface part 124 Operation data conversion part 125 Edit data memory 126 Edit control part 131 DV
132 Tuner 140 Mobile PC
141 Interface unit 143 Browser 144 Display unit 151 PDA
152 Mobile terminal

Claims (21)

Operation information receiving means for receiving first operation information for controlling the remote control device from a remote control device connected via a first network;
Operation information generating means for generating second operation information based on the first operation information;
Operation information transmitting means for transmitting the second operation information to a remote control terminal connected via a second network;
Control information receiving means for receiving, from the remote control terminal device, first control information for the remote control device in accordance with the second operation information;
Control information generating means for generating second control information based on the first control information;
Control information transmitting means for transmitting the second control information to the remote control device. Further comprising an operation information changing means for changing from the second operation information to the third operation information in response to a user instruction input;
The remote control device according to claim 1, wherein the operation information transmitting unit transmits the third operation information to the remote control terminal device.   The second operation information includes operation element information indicating one or more operation elements of the remote control device, and the operation information changing unit selects an operation element selected by a user's instruction input from among the operation elements. The remote control device according to claim 2, wherein the third operation information is generated.   4. The remote control device according to claim 1, wherein the second operation information further includes operation information for selecting the remote control device as a control target. 5.   The remote control device according to any one of claims 1 to 4, wherein the second operation information includes an operation element including text data.   5. The remote control device according to claim 1, wherein the second operation information includes an operation element including image data. 6. First operation information for controlling the remote control device is received from a remote control device connected via a first network, and second operation information is received based on the first operation information. Operation information receiving means for receiving the second operation information from the remote control device to be generated via the second network;
Control information generating means for generating first control information for the remote control device according to the second operation information;
The first control information is transmitted to the remote control device that generates the second control information based on the first control information and transmits the second control information to the remote control device. A remote control terminal device comprising control information transmission means.   The operation information receiving means receives, from the remote control device, third operation information changed from the second operation information in the remote control device in response to a user instruction input. 8. The remote control terminal device according to 7.   The second operation information includes operation element information indicating one or more operation elements of the remote control device, and the third operation information changed from the second operation information includes the operation elements. The remote control terminal device according to claim 8, further comprising an operation element selected by a user instruction input.   10. The remote control terminal device according to claim 7, wherein the second operation information further includes operation information for selecting the remote control device as a control target. 11.   The remote control terminal device according to any one of claims 7 to 10, wherein the second operation information includes an operation element including text data.   The remote control terminal device according to claim 7, wherein the second operation information includes an operation element including image data. Operation information generating means for generating first operation information for controlling the remote control device;
The second operation information is generated based on the first operation information, and the first operation information is transmitted to the remote control terminal device that transmits the second operation information to the remote control terminal device via the first network. Operation information transmitting means for transmitting operation information;
The remote control device receives first control information for the remote control device corresponding to the second operation information via the second network, and the remote control device generates based on the first control information Control information receiving means for receiving the second control information to be received via the first network.   A remote control device according to any one of claims 1 to 6, a remote control terminal device according to any one of claims 7 to 12, and a remote control device according to claim 13. Remote control system featuring. A control method for a remote control device, comprising:
An operation information receiving step of receiving first operation information for controlling the remote control device from a remote control device connected via a first network;
An operation information generating step for generating second operation information based on the first operation information;
An operation information transmission step of transmitting the second operation information to a remote control terminal device connected via a second network;
A control information receiving step of receiving, from the remote control terminal device, first control information for the remote control device according to the second operation information;
A control information generating step for generating second control information based on the first control information;
A control information transmission step of transmitting the second control information to the remote control apparatus. A method for controlling a remote control terminal device, comprising:
First operation information for controlling the remote control device is received from a remote control device connected via a first network, and second operation information is received based on the first operation information. An operation information receiving step of receiving the second operation information from the remote control device to be generated via the second network;
A control information generating step for generating first control information for the remote control device according to the second operation information;
The first control information is transmitted to the remote control device that generates the second control information based on the first control information and transmits the second control information to the remote control device. And a control information transmitting step. A control method for a remote control terminal device, comprising: A method for controlling a remote controlled device, comprising:
An operation information generating step for generating first operation information for controlling the remote control device;
The second operation information is generated based on the first operation information, and the first operation information is transmitted to the remote control terminal device that transmits the second operation information to the remote control terminal device via the first network. An operation information transmission step for transmitting operation information;
The remote control device receives first control information for the remote control device corresponding to the second operation information via the second network, and the remote control device generates based on the first control information And a control information receiving step of receiving the second control information to be transmitted via the first network.   The program for making a computer perform the control method of the remote control apparatus of Claim 15.   The program for making a computer perform the control method of the remote control terminal device of Claim 16.   A non-transitory computer-readable storage medium storing a program for causing a computer to execute the control method for a remote control device according to claim 17. A computer-readable recording medium on which the program according to any one of claims 18 to 20 is recorded.

Images