WO2001023994A1

WO2001023994A1 - Handheld retrieves ui from server for control of apparatus via server

Info

Publication number: WO2001023994A1
Application number: PCT/EP2000/009220
Authority: WO
Inventors: Rik Sagar; Greg Roelofs
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 1999-09-27
Filing date: 2000-09-21
Publication date: 2001-04-05
Also published as: JP2003510716A; KR20010080609A; EP1133726A1

Abstract

A handheld remote (102) has a wireless modem (108) to send an identifier to a server (110) on the Internet. The server has a look-up table (112) to associate the identifier with a URL. The URL specifies a CGI program (118) on another machine on the Internet. The machine controls equipment (122-128) through execution of the CGI program.

Description

Handheld retrieves UI from server for control of apparatus via server

FIELD OF THE INVENTION

The invention relates in particular to remote control of one or more apparatus in a consumer electronics (CE) environment.

BACKGROUND ART

Remote control units are widely known. An example is the Pronto (TM) manufactured by Royal Philips Electronics. The Pronto (TM) is a so-called universal programmable remote control unit. It enables storing the IR codes of practically any existing CE apparatus. The Pronto (TM) has an LCD with touch screen functionality as a user- interface (UI). The control functionalities of the Pronto (TM) can be fully customized by the user him/herself. The Pronto (TM) communicates with nearby CE equipment using infra-red (IR) radiation.

SUMMARY OF THE INVENTION The invention expands the Aremote controls concept. It is an object of the invention to provide full remote control to the user regardless of the distance between the equipment to be controlled and the user.

To this end, the invention provides a method of enabling an end-user to control an apparatus through a control device. The control device is, e.g., a palmtop computer with a graphical user-interface (GUI), and the apparatus is, e.g., a piece of CE equipment. A server is enabled to receive, via a data network such as the Internet, an identifier sent by the control device. The server has a look-up table or data base to determine a URL associated with this identifier. Then, the server is enabled to contact a software program as specified by the URL to control the apparatus. The software program is, e.g., a CGI application running on the server itself or on another machine. Preferably, the user is enabled to specify or program an association between the identifier and the URL. For example, the user is enabled to program the association via a browser. An aspect of the invention relates to enabling the server to download a component for a UI to the control device for enabling the control device to send the identifier upon user-activation of the component.

Another aspect relates to enabling respective ones of multiple servers to receive the identifier for determining a respective one of multiple URLs representative of a respective one of multiple software programs. In this manner, a single identifier sent to the server can cause several control events to take place at one or more of the controlled apparatus as in a macro or script.

The invention relates to an information processing system comprising a device connects to a server. The server uploads a UI to the device. The UI enables the user to cause a program to be executed on the server or on another machine. The server or the other machine is coupled to a sub-system. The sub-system is controllable through the program (or script). Preferably, the user-interface is customizable or user-programmable through, e.g., a PC connecting to the server or being a functional part of the server. Preferably, the user is enabled to specify which interaction with the UI is to cause what control-related event at the sub-system.

In an implementation of the invention, the control device is a handheld device that connects to the Internet via a wireless modem. The device retrieves from a server a UI comprising labeled widgets. The UI is preferably user-programmable or customizable. Activating a widget at the handheld device causes the label to be transferred to server. The label is associated at the server with a URL that enables the server to select a resource on the Internet, in this case a CGI application on a further Internet-enabled machine or on the server itself. The machine or the server executes the CGI application and communicates with the apparatus to be controlled using a proprietary protocol. The invention allows the transfer of the UI for, e.g., a home control system, to portable, handheld Internet-connected devices. These devices provide the means for the user to interact with the home network, regardless of where the user or the home network is located. UI devices are then Athin-clients≡, in terms of the resources they need to interact with the home network. The UI devices do not need specific hardware to control the home network directly. Instead, they rely on the server (with which they communicate exclusively) to provide the means to control the home network.

Accordingly, the invention provides several advantages, among which are the following: the UI is stored on server and is independently usable by several control devices of the same user; the UI is associated with the user, rather than with the control device; storing the actions as URL=s at the server allows the server to call programs in different machines; the invention leverages the robust Internet infrastructure; the control device can be a very thin client.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is explained in further detail, by way of example and with reference to the accompanying drawings, wherein:

Fig.1 is a block diagram of a system in the invention; and Fig.2 is a diagram of an example of a configuration page.

DETAILED EMBODIMENTS

Fig.l is a diagram of a system 100 in the invention. In system 100, a handheld device 102 is connected to a data network 104, e.g., the Internet, via a wireless IP connection 106, using a wireless modem 108 for communication with a server 110. Device 102 comprises, for example, a personal digital assistant (PDA), e.g., a Palmlllx manufactured by 3COM or a Windows CE-based handheld, that is connected to the Internet via wireless modem 108. Modem 108 is, for example, a Minstrel or a Ricochet. The Minstrel marketed by Novatel Wireless is a two-way wireless modem for a PDA that lets the user browse the Web and receive email, among other things. In a more general sense, a wireless modem like the Minstrel or Ricochet provides the handheld device with an IP address that can be used by any type of application that uses the Internet for communication (within limitations of throughput, latency and coverage). The Minstrel uses a technology referred to as Cellular Digital Packet Data (CDPD) that is supported by the cellular service providers.

Upon a valid connection server 110 uploads code for a user interface (UI) to device 102. The UI comprises a number of UI Awidgets≡. The term A widgets is used in various ways in the field of computers. A widget is an element of a UI that provides information for a user to interact with the operating system and application programs. Widgets include icons, pull-down menus, buttons, selection boxes, progress indicators, on-off checkmarks, scroll bars, windows, window edges for resizing the window, toggle buttons, etc. for displaying user interactivity functionalities. In programming, a widget also means the small program that is written in order to describe what a particular widget looks like, how it behaves, and how it interacts in response to user actions. Most operating systems include a set of ready-to-tailor widgets that a programmer can incorporate in an application, specifying how it is to behave. New widgets can be created. Most application development languages today, such as Java and Tel, come with a ready-made library of widgets that a programmer can incorporate and modify. Using Microsoft's Visual Basic, a widget can be implemented as, or part of, an ActiveX control, etc., (see, e.g., www.whatis.com). The UI does not need to know the functionality of the particular device to be controlled, it only needs to know which widget to present. The widgets determine how the UI is structured.

Each widget has a label, which conveys to the user of device 102 an action that will occur if the user interacts with that widget on device 102. An alphanumeric example of a graphically represented label is the text ATurn Lights On≡, another example is a graphical representation of a lamp that has been turned on. When the user interacts with a widget, an identifier for that widget is passed back to server 110. For instance, a UI comprising four momentary buttons on a touch screen of device 102 passes back AButtonl ", AButton2", AButton3" or AButton4", dependent on which button was activated by the user.

When server 110 has received the identifier, server 110 looks up what is to happen next using a data base or look-up table 112 for associating the identifier with a particular action to be taken.

To configure the labels for each of the widgets of the UI, i.e., to set up the relationship between actions desired and labels, the user interacts through a configuration panel. The configuration panel is made available to the user through, e.g., a web page, as is discussed in further detail below. The configuration panel can be made accessible via handheld-device or via another piece of equipment 114, e.g., a PC or a set-top box, that has a browser 116 available. The web page is preferably dynamically created, uniquely for each individual user. Fig.2 shows an example of such a configuration page 200 and is discussed below. In addition to setting the label through the web site, the user sets or specifies an action that is to occur when the user interacts with the labeled widget in the UI of device 102 as a result of which the associated identifier is sent to server 110.

The action is specified in the form of a Unified Resource Locator (URL). A URL is a means by which a resource on an Internet site is selected. A resource is, e.g., an HTML document, a WAV (sound) file, an image file, etc. In the case of the URL provided for the widget=s action, the URL as specified in data base 112 preferably points to a Common Gateway Interface (CGI) application 118. CGI application 1 18 is a program that is started by a server and is executed on a server, like any other program that might run on a computer. In this case, CGI program 118 is executed on an action server 120. Action server 120 is here drawn as being among a plurality of servers. Each of these servers has its own collection of CGI=s and is used to control further equipment in a similar manner as explained here.

Information can be passed from the requestor of the CGI (traditionally a remote user using a web browser, such as Netscape Navigator or Internet Explorer), via a mechanism such as the AQuery String≡.

CGI application 118 implements whatever action the user required. In the case of the ATurn Light On≤ example, CGI 118 application communicates with a hardware actuator 122 connected to server 120 and that is capable of turning on a light. The connection between actuator 122 and server 120 can be a cable or wireless connection, or is implemented using a network itself. Hardware actuator 122 might be a relay or triac connected directly to an output pin of a processor (i.e., an I/O port) of server 120, or it might be an X-10 module (such as the PL513 available from http://www.xl0.com) connected to server 120 via an RS232 serial port. In this example, actuator is an X-10 compliant device. Other actuators 124, 126, ..., 128 use, for example an IR protocol, a HAVi protocol or a Afuture protocols. The latter example shows that the system in the invention is readily upgradeable. Actuators 122- 128 directly control the relevant apparatus (not shown) as indicated with the outgoing arrows.

It is worth noting that the URL provided can point to a CGI program on a machine anywhere on the Internet 104 (assuming the route is not blocked by firewalls, etc.). Alternatively, the URL can point to look-up server 108 itself, through the Aloopback≤ or Alocalhost≡ address (see address Al 27.0.0.1 " in Fig.2), incorporated into the IP specification and referred to in Fig.l by reference sign 124.

When the UI has been reconfigured, or if handheld device 102 requests it, the description of the UI is transferred from server 110 to handheld device 102. In this example, the UI is transferred to handheld device 102 by sending, e.g., an XML document, which lists the widgets and their associated labels. The actions taken by server 120 are never sent to the UI, as handheld device 102 relies on server 110 to interpret the keys pressed or other user- input entered on handheld 102 and to call the appropriate action URL.

Storing the actions as represented by URLs allows server 110 to easily call programs in different machines, such as server 120. It is likely that home control, automation and monitoring applications will use multiple, distributed computers, to provide different components of the services. By using the existing, robust HTTP interface to invoke the applications on the various computers, the existing infrastructure of the Internet is leveraged, including the extensions for security/privacy, i.e., encryption, authentication, etc. Because the UI is stored at server 110 on the Internet, it allows it to be associated with the user, rather than with device 102 through which that user accesses it. This means that if the user has several devices 102 through which he/she can control his/her environment, he/she only needs to update the UI once and the change will take effect on all interaction devices 102. By using the Internet to store the UI, modification can be done through an interface that is more convenient than that which would be available on the handheld devices themselves. For example, the UI of a handheld device is typically limited to a small, Avirtual≡ keyboard or the Ahunt-peck≡ interface implemented via a numeric keypad (as on a telephone). See for an improvement on this U.S. serial no. 09/062,364 (Attorney docket PHA 23,287) filed 4/17/98 for Sung Choi and Jan van Ee for GRAPHICAL USER INTERFACE TOUCH SCREEN WITH AUTO ZOOM FEATURE. The virtual keyboard and the hunt-and-peck approach are typically slower and more error prone to use than a traditional keyboard with full-fledged PC 114. With the UI stored and accessible through network 104 in the way described, it can be modified through any client that has a suitable application to interact with it. Web browser 116 is sufficient. An extension of the above would be that when newer modes of interaction are developed, e.g., graphical or speech commands, new clients can be created that take advantage of these new modes. The new interaction methods can be used to modify the UI, replacing the more conventional method mentioned above. Handheld device 102 does not need to be changed in any way to allow this. Because handheld device 102 interacts with server 110, it allows the UI to control devices that it does not have the means to directly connect to itself. For example, handheld device 102 is not connected to the home power network and would not normally be able to control an X-10 device. However, using the system of the invention, handheld device 102 can communicate to server 110, which in turn communicates (through HTTP) with a CGI application 118 on machine 120 that has such a connection to the home power network.

The invention allows the transfer of the UI for, e.g., a home control system, to portable, handheld Internet-connected devices. These devices provide the means for the user to interact with the home network, regardless of where they are located. The devices are then Athin clients≡, a term that indicates their modest requirements regarding the resources they need to interact with the home network. They do not need specific hardware to control the home network directly, but rely instead on the server (with which they communicate exclusively) to provide the means to control the network.

Fig.2 shows an example of a configuration panel 200 as presented in browser 116. Panel 200 has a column 202 with header AButton≤, a column 204 with header ALabel≡, and a column 206 with a header AAction≡. The term AButton≤ refers to a button widget on the UI as presented to the user at device 102. In column 202, the buttons are listed as Abutton l≤, Abutton 2≡, AAbutton 3_s, Abutton 4≡ and Abutton 5≡. Each of these buttons is labeled as shown in column 204. The labels specify the event triggered when the user activates the corresponding button. Column 206 lists the URLs that specify the address of the associated CGI=s. Panel 200 allows the user to edit the UI and reprogram each individual one of the buttons.

An aggregate UI can be provided by server 110 having widgets (e.g., buttons, sliders, etc.) that correspond with CGI=s at different action servers. For example, in a home, one action server, e.g., server 120, controls a sprinkler system, another one is associated with a set-top box (STB), and a third server is to control an X-10 network and act as the preferred Internet gateway. A GUI on device 102 comprises, e.g., a spin button and on/off switch for the sprinkler system, an on/off switch, channel selector, volume control, and mute for the STB, and a house-code selector and on/off switch for the X-10 server. The action servers need not be in close proximity. The UI equally well aggregates webcam info from Internet servers all over the world with controls including "switch to other camera" and "zoom/pan/tilt".

Beyond aggregate control, multiple look-up servers can negotiate with each other to transfer parts of the UI, either from a selection of common widgets (sliders, radio buttons, checkboxes, toggles, etc.) or as active code (e.g., Java applets) that provides new GUI elements. Remote device 102 may control the negotiation, or the look-up servers may obey a protocol that decides on relative placement, sizing and priorities of the widgets based on static preference information retrieved from the remote device or previously stored by the user. A common example in another technical field is an aggregate meta text-search, displaying 5 results from AltaVista, 2 from Lycos, 3 from HotBot, etc. A richer example includes a time-varying percentage of the display devoted to the controls of a home server 120 and negotiated percentages devoted to stock-market graphs, traffic info, weather info, Slashdot and CNET headlines, and e-mail from one or more accounts at other action servers. Depending on the negotiation, only three of the servers might provide UI elements at one time of the day, while seven or more might provide UI elements at a later time.

What has been explained with regard to GUI=s applies, mutatis mutandis, also to modalities of user-interaction other than purely graphical, e.g., voice control. Within this context see, e.g., U.S. serial no. 09/464,855 (attorney docket PHA 23,875) filed 12/16/99 for Willem Bulthuis et al., for HAND-EAR USER INTERFACE FOR HAND-HELD DEVICE. This document relates to a hand-held information processing device, such as a mobile phone or remote control, that has a thumb wheel which lets the user scan a circular array of options. Each respective one of the options is represented by a respective audio output that gets played out when the wheel is turned a notch up or down. This enables the user to select an option with one hand and without having to look at the device. It also allows for a form factor smaller than that of a conventional mobile phones since a keypad is not needed for entering digits to make a call from a personalized directory. This device can also be used as a AVery Remote Controllers as specified herein above, wherein a menu of control options can be scanned by means of the thumb wheel and the audio output. Upon finding the desired option as represented by the audio output, the user validates the current option, whereupon the device sends the associated identifier to the server for further processing as explained above.

Above, the remote control over the Internet has been explained with reference to some conceptually simple examples. It is clear that other and more sophisticated scenarios can be implemented using the invention. For example, a recording device such as a VCR or a TiVo (i.e., a personal video recorder with hard-disk drive (HDD) and capable of simultaneous record and play-out of a TV broadcast) can be programmed from anywhere in the world when the user realizes that he/she has forgotten or was unable to set the device to record a favorite TV program. Within this context, also see U.S. serial 09/283,545 (attorney docket PHA 23,633) filed 4/1/99 for Eugene Shteyn for TIME-AND LOCATION-DRIVEN PERSONALIZED TV. As another example, the user may turn on the central heating and turn on the lights at home and in the garden while returning home by car late at night. As another example, the user may program or activate the sprinkler installation in the garden at home while at work or traveling. Also, what has been explained with regard to CGI scripts can also be implemented using other protocols in proprietary data networks to invoke a script or application on an action server..

Further, reference is made to the following patent documents: - U.S. Serial no. 09/427,821 (Attorney docket PHA 23,786) filed 10/27/99 for Joost Kemink and Richard Sagar for PDA HAS WIRELESS MODEM FOR REMOTE CONTROL VIA THE INTERNET. This document relates to an information processing system that has a handheld computing device, e.g., a PDA,(Personal Digital Assistant) with a user-interface and a wireless modem coupled to the handheld. The wireless modem enables communication with a server via a data network such as the Internet. A control network is coupled between the server and controllable equipment. The handheld is now capable of functioning as a wireless remote control device for the equipment via the Internet and the server. The system may comprise a video camera together with hardware and software to create a formatted still image suitable for being displayed on the handheld device. The user can now instruct retrieval of a still image from the server via the Internet. This application serves as, e.g., a security system that enables the remote user to monitor his/her front porch, or to monitor a child by way of a remote (or fall-back) baby-sit. The user-accessibility of equipment is guaranteed by the ubiquity of the Internet, thus enabling to expand the range of control and monitoring capabilities for a mobile user. - U.S. serial no. 09/210,416 (attorney docket PHA 23,522) filed 12/11/98 for

Joost Kemink and Eugene Shteyn for REMOTE CONTROL DEVICE WITH LOCATION DEPENDENT INTERFACE. This document relates to a user control interface that is location dependent within the home environment. Context control parameters are associated with location, and the user control interface is customized to the context within which the device is being operated. The control interface includes the presentation of context sensitive information and the communication of corresponding context sensitive user commands via the interface. The location determination is effected using any number of commonly available techniques, such as direct entry, infrared sensors and active badges for relative positioning, as well as the conventional absolute positioning devices such as LORAN and GPS. In a preferred embodiment, the device communicates with a remote information source that provides the context sensitive control information. The remote information source may be a home network server, an Internet server, a public service network, or other communication network.

- U.S. serial no. 09/434,155 (Attorney docket PHA 23,783) filed 11/4/99 for Martin Freeman and Bonghan Cho for REMOTE INITIATES RETRIEVAL OF CONTROL

CONFIGURATION. This document relates to a programmable remote control unit that is capable of initiating retrieval of a control configuration from a storage device external to the unit, The unit comprises a memory to store the retrieved control configuration; a display for display of icons representing the configuration; and a touch screen for entering a selection based on the icons displayed. The storage device is a component a CE apparatus. By storing or backing-up the control configuration for a specific apparatus in the apparatus itself the remote is made truly universally programmable.

- U.S. serial no. 09/160,490 (attorney docket PHA 23,500) filed 9/25/98 for Adrian Turner, Simon Pearce, David Eves and Allan Timms for CUSTOMIZED UPGRADING OF INTERNET-ENABLED DEVICES BASED ON USER-PROFILE, and U.S. serial no. 09/189,535 (attorney docket PHA 23,527) filed 11/10/98 for Eugene Shteyn for UPGRADING OF SYNERGETIC ASPECTS OF HOME NETWORKS, both of which relate to a server system that maintains a user profile of a particular end-user of consumer electronics network-enabled equipment and a data base of new technical features for this type of equipment. If there is a match between the profile and a new technical feature, and the user has indicated he/she is willing to receive the information about updates or sales offers, the user gets notified via the network of the option to obtain the feature. Synergy is detected between pieces of equipment of the user in order to notify him/her of further possibilities of using or expanding his/her equipment.

- U.S. serial no. 09/165,682 (Attorney docket PHA 23,484) filed 10/2/98 for Eugene Shteyn for CONTROL PROPERTY IS MAPPED ONTO MOD ALLY COMPATIBLE GUI ELEMENT. This document relates to an information processing system that has an electronic device and a controller for control of a functionality of the device. An abstract representation of the functionality is provided to the controller. The abstract representation exposes a modality of controlling the functionality. The controller enables controlling the functionality through interaction with the abstract representation. The modality controls associating the control of the functionality with a modally compatible controlling capability of the controller. The modality exposed can be, for example, ABooleans, Afloats, Ainteger array..;.

- U.S. serial no. 09/519,546 (attorney docker PH-US 000014) filed 3/6/00 for Erik Ekkel et al., for PERSONALIZING CE EQUIPMENT CONFIGURATION AT SERVER VIA WEB-ENABLED DEVICE. This document relates to facilitating the configuring of CE equipment by the consumer by means of delegating the configuring to an application server on the Internet. The consumer enters his/her preferences in a specific interactive Web page through a suitable user-interface of an Internet-enabled device, such as a PC or set-top box or digital cellphone. The application server generates the control data based on the preferences entered and downloads the control data to the CE equipment itself or to the Internet-enabled device.

Claims

CLAIMS:

1. A method of enabling to control an apparatus (122-128) through a control device (102), the method comprising:

- enabling a server (110 )to receive an identifier sent by the control device;

- enabling the server to determine a URL associated with the identifier; and - enabling the server to contact a software program (118) as specified by the URL to control the apparatus.

2. The method of claim 1, comprising enabling the server to download a component for a UI to the control device for enabling the control device to send the identifier upon user-activation of the component.

3. The method of claim 1 , comprising enabling a user to specify an association between the identifier and the URL.

4. The method of claim 3, wherein the user is enabled to program the association via a browser (116).

5. The method of claim 1, wherein respective ones of multiple servers are enabled to receive the identifier for determining a respective one of multiple URLs representative of a respective one of multiple software programs

6. The method of claim 6, wherein the respective one of the multiple software programs controls a respective one of multiple apparatus (122-128).

7. A look-up table (112) for being stored at a server (110) and for associating an identifier received from a user (102) with a URL that specifies a CGI program (118).

8. The table of claim 7, being user-programmable.

9. The table of claim 7, customized for an individual user.