WO2006120596A2

WO2006120596A2 - Remote control through orientation change of handheld

Info

Publication number: WO2006120596A2
Application number: PCT/IB2006/051334
Authority: WO
Inventors: Albertus J. N. Van Breemen; Elmo. M. A. Diederiks
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2005-05-10
Filing date: 2006-04-28
Publication date: 2006-11-16
Also published as: WO2006120596A3

Abstract

A handheld device is network-enabled and comprises a sensor for sensing the device's orientation with regard to a reference, and a display monitor. Changing the orientation results in transmission of data being sent that is representative of the change. This enables control of a remote apparatus via the network. The apparatus comprises a camera, or is being monitored by a camera, that captures images sent back to the device for providing visual feedback to the user via the display.

Description

Remote control through orientation change of handheld

FIELD OF THE INVENTION

The invention relates to a handheld device for controlling an apparatus via a data network. The invention relates in particular, but not exclusively, to tele-presence.

BACKGROUND ART The term tele-presence refers to the ability to be virtually present at a remote location. The remote location accommodates one or more sensors, such as a CCD camera, a microphone and speakers. The signals captured by these sensors are sent to the user as data via a communication path, e.g., a data network. Tele-presence is used for, e.g., monitoring dangerous environments in an industrial setting, monitoring ones home for security purposes, tele-conferencing and baby monitoring.

Various companies currently make devices available for use within a telepresence context, for example, Logitech's QuickCam Orbit, and personal robots such as the AIBO from Sony.

SUMMARY OF THE INVENTION

The inventors address the problem of how to let a user conveniently and remotely control the tele-presence apparatus such that the user can scan the remote location and have the sensors pointed in the proper direction. A computer joystick or a computer mouse could be used to control the position of the tele-presence apparatus through a personal computer (PC). Alternatively, a display monitor could be used with control buttons ("left", "right", "zoom", etc.) for user-control of the remote sensor's position and/or orientation. A yet further alternative could be a head-tracker for tracking the orientation of the user's head or eyes to generate control signals for control of the tele-presence apparatus. The main problem with these solutions is that they are not convenient for use within a consumer electronics (CE) environment: the user would then need to carry around lot of equipment for tele-presence purposes. The expression "consumer electronics" used in this text refers to the mass-market of products and services for which the end-user is the consumer relating to, e.g., audio, video, still pictures, gaming, communication, or any technology relating to electronic processing of data or signals for providing information and/or entertainment. The inventors propose to provide a control device with a user interface that facilitates tele-presence control in an intuitive manner.

To this end, the invention provides a handheld CE device that has means for communicating via a data network. The device comprises, e.g., a mobile telephone or a palmtop PC or a laptop PC. The device further comprises a sensor for determining a change in orientation relative to a reference (e.g., gravity or a reference orientation and/or position), and electronic circuitry coupled to the sensor for supplying data representative of the change to an addressable apparatus on the network. Accordingly, the user is enabled to transmit data to the apparatus to change a property of the apparatus. The data itself may comprise the command for changing the property, or the data may get interpreted locally and converted locally into a command e.g., via a user-defined look-up table or another protocol.

The invention also relates to a set or a system comprising a handheld consumer electronics device and an apparatus. The device comprises means for communicating via a data network, a sensor for determining a change in orientation relative to a reference; and electronic circuitry coupled to the sensor for supplying data representative of the change. The apparatus is configured for being made addressable on the network for receipt of the data. In an embodiment of the invention, the device comprises a display monitor and the apparatus comprises a camera for capturing an image. The apparatus and the device can be marketed as a set of entities configured for cooperating in a pre-specified or user programmable manner.

BRIEF DESCRIPTION OF THE DRAWING

The invention is explained in further detail, by way of example and with reference to the accompanying drawing wherein Fig. 1 is a diagram of a system in the invention.

DETAILED EMBODIMENTS

The invention relates to a handheld CE device that has means for communicating via a data network, e.g., the Internet. In an embodiment of the invention, the device comprises, e.g., a mobile telephone or a palmtop PC or a laptop PC. The device further comprises a sensor for determining a change in orientation relative to a reference (e.g., relative to the direction of gravity or relative to a reference orientation and/or position), and electronic circuitry coupled to the sensor for supplying data representative of the change to an addressable apparatus on the network. Accordingly, the user is enabled to transmit data to the apparatus, by means of changing the device's position or orientation, in order to change a controllable property of the apparatus. The data itself may comprise the command for changing the property, or the data may get interpreted locally and converted locally into a command e.g., via a user-defined look-up table or another protocol. Fig. 1 is a diagram of a system 100 in the invention. System 100 comprises a handheld consumer electronics device 102, a data network 104 and an addressable apparatus 106. Device 102 has means 108 for communicating with apparatus 106 via data network 104. The connection between device 102 and data network 104 may be wired or wireless. Similarly, the connection between network 104 and apparatus 106 may be wired or wireless. Device 102 also has a sensor 110 for determining a change in orientation relative to a reference, e.g., gravity. Device 102 further has electronic circuitry 112 coupled to sensor 110 for supplying data representative of the change to apparatus 106.

Communication means 108 need not be discussed as this is well known in the art. See, e.g., the references listed at the end of this text. Sensor 108 may comprise any suitable component to sense a relative change in orientation: an onboard gyroscope, a magnetic compass, accelerometers, tilt sensor, etc. As device 102 is in particular meant for the consumer electronics market, a reliable low-cost implementation is desirable. Sensor 108 may be configured to sense orientation changes in one or more dimensions. For example, sensor 108 is configured to sense rotation about the device's main axes: X, Y and Z in order to sense pitch, roll and yaw. Instead, or in addition, sensor 108 may be configured to sense translation in the direction of one or more of these axes.

Apparatus 106 is addressable on data network 104. The user of device 102 may decide to communicate with, or control, another apparatus (not shown) also connected to network 104. The user then only has to select the proper network address or another identifier suitable for distinguishing between apparatus 106 and the other apparatus.

Preferably, device 102 further comprises a display monitor 114 for rendering an image received via network 104 or via another communication path. This enables the user to receive visual feedback on the controlling of apparatus 106 if the latter is monitored by a local network-enabled camera (not shown) or if apparatus 106 itself has a camera that transmits its image to device 102.

Preferably, electronic circuitry 112 is selectively operational upon a user input to device 102 via a suitable user interface (not shown). In this way, the user can select the operational mode of device 102. If device 102 has several different functionalities, the change of orientation does not result in data being sent to apparatus 106 inadvertently when circuitry 112 is turned off and device 102 is not being used for remote control purposes.

Preferably, device 102 comprises a mobile telephone or a personal computer with data network (e.g., Internet) access. A mobile telephone and a network-enabled notebook PC have means for communicating via data network 104, and also accommodate a display monitor such as display monitor 114. They have become commodities and personal trusted devices that the user carries along most of the time. Accordingly, integrating orientation sensor 110 with such a personal trusted device enables to equip the trusted device with a highly intuitive and ergonomic remote control functionality that is always within reach of the user.

For security reasons, the data as sent and the command associated with the data need to be hidden from a malicious eavesdropper or from a hacker. An example technology for achieving this is based on the technology as discussed in US ser. no. 09/670,129 (attorney docket US 000262) filed Sept 26, 2000 for Vladimir Pisarsky for SECURITY MONITOR OF SYSTEM RUNS SOFTWARE SIMULATOR IN PARALLEL, published under PCT as WO0227440 and incorporated herein by reference. This WO document discloses a monitoring system for monitoring multiple devices that perform respective primary tasks. For the purpose of the monitoring, each device has a finite state machine (FSM) with stochastic non-periodic behavior. The monitoring system simulates in software the behavior of the FSM's. A discrepancy between the states assumed by the FSM's after each time-step and the states assumed in the simulation in each time-step indicates a failure or a breach of the network's integrity. Hacking such as primary system is practically impossible without being detected. Each device comprises computational resources. In order to reduce the computational environment available to a virus, the idle time of the resources is absorbed by dynamically increasing the complexity of the FSM's. In the current invention, the data as generated on the basis of the change of orientation can be encrypted using an FSM with a stochastic or quasi-stochastic behavior as discussed in above WO publication. The destination of the transmission, here controllable apparatus 106 or its local network environment, has a simulation of the same FSM running synchronously. The latter can thus be used to decrypt the data in order to recover the intended command for control of apparatus 106. Note that the very data as sent can be used only once to achieve the same effect at controllable apparatus 106. Copying the data for future purposes is of no use at all to a hacker. Preferably, a relationship between the change and the command is programmable or user-programmable. An example of this has been discussed above with reference to WO0227440. As another example, the user may specify his/her own look-up table per individual controllable apparatus, such as apparatus 106, for converting a change in orientation of device 102 into a proper command.

Preferably, the change in orientation is proportional (for all practical purposes) to a change in the controllable property. The user receiving feedback has then the impression of being directly involved or directly connected so that the control intuitively acceptable. For example, apparatus 106 comprises a local stationary camera, e.g., a web cam mounted on a wall, for capturing images. The change of orientation of handheld device 102 ("pitch" and "yaw" in aircraft jargon) of device 102 can be converted in commands for the camera movement in both the vertical plane (tilt) and the azimuth (pan). As another example, apparatus 106 comprises a garden sprinkler system that is network-enabled and thus can be turned on and off through commands delivered via network 104. The system's water jet has a direction that can be monitored via a web cam on the porch and that can be controlled by means of the orientation of device 102, even when residing a continent further down the road, through a corresponding translation of orientation changes into commands.

Apparatus 106 itself may be a local (e.g., home) server that passes on the commands to further equipment on a local (home) network, either straight away or after data conversion or transcoding. The camera at the server's end may then merely monitor the visual effects of the commands as brought about by the equipment thus controlled. Device 102 is preferably to be used without any hassle or inconvenience to the user. Imagine that device 102 is being used to control apparatus 106 and that device 102 receives visual feedback that is rendered on display monitor 114. While the user is manipulating device 102 the orientation of device 102 relative to the user is changing as well. This may lead to the user having to change his/her position relative to device 102 in order to properly see the visual feedback. Returning device 102 to its initial ("normal") position and orientation is registered by circuitry 112 as a change that causes corresponding data to be sent to apparatus 106. Accordingly, the reference is made user-programmable, for example in the following manner. Device 102 is provided with a reset input 116 to enable to reset the reference used by circuitry 112. As explained above the reference is used to determine the change in orientation. Resetting the reference to the orientation state, assumed by device 102 at the moment of receiving the reset input, enables the user to send data representative of incremental state changes without having to change his/her position relative to display monitor 114 device 102. For example, in order to have a web cam 106 rotate 360 degrees in the horizontal plane, the user rotates device 102 about the vertical axis by, say, 30 degrees before pressing reset 116. Camera 102 has turned 30 degrees as well. Device 102 can now be brought back to its initial position from which the next 30 degrees at the camera can be controlled to have the camera pointing in the 60 degrees direction, and so on.

What has been discussed above with regard to changes in orientation of device 102, can also be made to apply to the relative position of device 102 with respect to the user. Sensor 110 or another sensor (not shown) is then operative to sense a quantity representative of a spatial distance between device 102 and the user. This could be achieved by, for example, using an accelerometer in combination with a reference position, echo location through sound, through infrared or radio frequency waves bouncing off the user's body, infrared detection of the body, or any other suitable manner. A change in relative position of device 102 with regard to the user can then be used to generate data that is transmitted to controllable apparatus 106 via network 104. The change in relative position can be used to, e.g., control from device 102 camera 106 to zoom in or out depending on the user's moving the camera closer or farther away from him/her. Alternatively, the relative change in position can be used for locally, i.e., at device 102 itself, to generate a zoomed in or out version of the image supplied to device 102. In the operational mode of device 102 based on registering displacements (relative changes in position) reset input 116 can be configured to serve a purpose similar to the one of reset input 116 in the scenario of sensing the roll, yaw and pitch of device 102.

Device 102 could also be used to control a moveable apparatus 106, e.g., a self-propelling vehicle (e.g., having a battery for power supply). Vehicle 106 then has a wireless connection to network 104 and accommodates a camera that captures images that get sent back to the user via network 104. Rotating device 102 in a horizontal plane can then be used to generate commands for steering the vehicle, and rotating device 102 about a horizontal axis can be used to generate commands for forward and reverse, the direction depending on the polarity of the angle or rotation, and the speed on the angle's magnitude. Incorporated herein by reference: US ser. no. 09/427,821 (attorney docket PHA 23,786) filed Oct 27, 1999 for

Joost Kemink et al., for PDA HAS WIRELESS MODEM FOR REMOTE CONTROL VIA THE INTERNET, and published under PCT as WOO 124473. This document discloses a PDA combined with a wireless modem to enable remote control of CE equipment via the Internet and a local home server. US ser. no. 09/544,666 (attorney docket US 000089) filed April 6, 2000 for Rik Sagar for HANDHELD RETRIEVES UI FROM SERVER FOR CONTROL OF APPARATUS VIA SERVER and published under PCT as WO0123994. This document discloses a handheld remote with a wireless modem for sending an identifier to a server on the Internet. The server has a look-up table to associate the identifier with a URL. The URL specifies a CGI program on another machine on the Internet. The machine controls equipment through execution of the CGI program.

US ser. no. 09/434,155 (attorney docket PHA 23,783) filed Nov. 4, 1999 for Martin Freeman et al., for REMOTE INITIATES RETRIEVAL OF CONTROL CONFIGURATION, published under PCT as WO0124387. This document discloses a programmable remote control unit that is capable of initiating retrieval of a control configuration from a storage device external to the unit. The unit comprises a memory to store the retrieved control configuration; a display for display of icons representing the configuration; and a touch screen for entering a selection based on the icons displayed. The storage device is a component of a Consumer Electronic apparatus. Storing or backing-up the control configuration for a specific apparatus in the apparatus itself makes the remote control unit truly universally programmable.

US ser. no. 09/932,105 (attorney docket US 018126) filed Aug. 17, 2001 for Pieter van der Meulen for SYSTEM FOR REMOTELY CONTROLLING CONSUMER ELECTRONICS USING A WEB CAM IMAGE and published as US2003/0037341. This document discloses a system with a camera to generate an image of an environment to be controlled by the user. The image is displayed on a display monitor of a control device. The control device allows the user to interact with a region of the image on the display monitor. The region corresponds to controllable consumer electronics (CE) equipment displayed within the image of the environment. Interaction with the region causes the CE equipment to be controlled in a pre-programmed manner.

US ser. no. 09/886,305 (attorney docket US 000146) filed June 19, 2001 for Greg Roelofs et al., for "SYSTEM AND METHOD FOR REMOTE CONTROL OF CONSUMER ELECTRONICS OVER DATA NETWORK WITH VISUAL FEEDBACK" and published as US20010053274. This document discloses a CE device with a generator for generating data to create an on-screen menu for a GUI. This data is transferred via the Internet to a remote location where the user is given visual feedback for control of the apparatus. Control is achieved via the Internet and a home server driving an IR blaster local to the apparatus.

Claims

CLAIMS:

1. A handheld consumer electronics device comprising means for communicating via a data network, and further comprising a sensor for determining a change in orientation relative to a reference; and electronic circuitry coupled to the sensor for supplying data representative of the change to an addressable apparatus on the network.

2. The device of claim 1, further comprising a display monitor for rendering an image received via the network.

3. The device of claim 2, wherein the electronic circuitry is selectively operational upon a user input to the device.

4. The device of claim 1 comprising a mobile telephone.

5. The device of claim 1 , wherein the data comprises a command for changing a property of the apparatus.

6. The device of claim 5, wherein a relationship between the change and the command is user-programmable.

7. The device of claim 5, wherein the change in orientation is proportional to a change in the property.

8. The device of claim 1, wherein the reference is user-programmable.

9. A set comprising: a handheld consumer electronics device with means for communicating via a data network, with a sensor for determining a change in orientation relative to a reference; and with electronic circuitry coupled to the sensor for supplying data representative of the change; and an apparatus for being made addressable on the network for receipt of the data.

10. The set of claim 9, wherein the device comprises a display monitor and the apparatus comprises a camera for capturing an image.