GB2471883A

GB2471883A - Controlling a software application in a thin client session using a mobile device

Info

Publication number: GB2471883A
Application number: GB0912374A
Authority: GB
Inventors: Frederic Fok; David Remy
Original assignee: NEC Corp
Current assignee: NEC Corp
Priority date: 2009-07-16
Filing date: 2009-07-16
Publication date: 2011-01-19
Also published as: GB0912374D0

Abstract

A touch-screen mobile phone 3 acts as a control device for a software application 67 hosted on a server 5 on behalf of a "thin" or "slim" (ie. low processing power) client device 7 (eg. a television or ) during a thin client session. The mobile device first registers with the server as a controller (41, 61) and indicates its input capabilities in order to be matched in a configuration database 69, after which the control events input by the user 40 are handled by the server (60, 63) in order to visually interact with the main output of the application 83 on the client device screen. The input configuration can be dynamically updated according to virtual dials, buttons or faders etc. (92, fig. 7) to switch from eg. "keyboard" mode to "camera" mode, and the software may be eg. a word processor, a computer game or an audio mixing tool. Numerous mobile controllers (eg. Smartphones, PDAs or laptops) can connect to the server via eg. Bluetooth, WiFi or USB protocols.

Description

Client-Server System The present invention relates to a client-server system and in particular, but not limited to, a client-server system in which a device operates as an interface via which a user can control an application hosted by a remote server.

Thin client systems are known in which a central server is responsible for hosting a software application on behalf of a client device (referred to as a thin' or slim' client) in a client-server network architecture. The server handles most of the processing requirements of the software, the server passing user input events for the hosted application received from the thin client to the software, and passing associated outputs generated by the software back to the thin client. Accordingly, a device with only minimal processing power may be used as the thin client, although devices with greater processing power may be configured to act as a thin client when it is practicable to do so.

With the increasing popularity of mobile computational and communication devices and in particular mobile (cellular) telephones it is now commonplace for individuals to carry their mobile devices with them most of the time. These devices are increasingly flexible in terms of their ability to communicate with one another and other computer devices (PCs, laptops, televisions etc.) using a variety of short and/or wide range wireless and wired technologies. It has also been suggested to use a mobile device as a thin client for an application hosted centrally by a remote server.

However remote applications accessible via the thin client paradigm remain difficult to control on mobile devices due to the small size of the screen and the difficulties to handle multiple input devices along with an output device on the same small size device.

The invention aims to introduce new mobile device applications and to solve mobile thin client problems by providing an alternative thin client based system in which a mobile communications device is used as an input device for an application running on a remote computer device, while the thin client session output is rendered on a separate thin client device. Moreover, embodiments of the invention take advantage

I

of the mobile device to dynamically adapt the input interface to the application running on the thin client device.

According to one aspect of the present invention there is provided a client server system comprising server apparatus, a mobile device and a client apparatus, the server apparatus comprising: means for running an application, the application being operable to accept inputs for controlling functions of the application and to generate outputs as a result; means for receiving, from the mobile device, a request for registration as a controller with the server apparatus, the request including an indicator from which an input capability of the mobile device can be obtained; means for obtaining the input capability of the mobile device using the indicator; means for obtaining input configuration information based on the obtained input capability and functions of the application; means for sending a signal carrying the input configuration information to the mobile device; means for receiving an input event from the mobile device for controlling said application and for providing information relating to the received input event to the application; and means for sending information representing the outputs generated by the application as a result of the provided input event, for output by the client apparatus; the mobile device having: input means for receiving inputs from a user and defining the input capability of the mobile device; means for generating the request for registration as a controller with the server apparatus, the request including the indicator from which the input capability of the mobile device can be obtained; means for sending the request to the server apparatus; means for receiving the signal carrying the input configuration information from the server apparatus, the input configuration information being based on the input capability of the mobile device and functions of the application run by the server apparatus; means for configuring the mobile device in accordance with said configuration information; and means for sending information identifying user inputs received via the input means to the server apparatus in accordance with said input configuration information; and the client apparatus comprising: means for receiving the information representing the outputs generated by the application from the server apparatus as a result of the provided input event; and output means for providing an output based on the received information.

The indicator may comprise a list of the input capabilities of the device, for example, the available input modules and/or the available input devices. The indicator may be an identifier for uniquely identifying the device based upon which the input capability may be looked up from a stored list, table, or database (or requested from another device). The indicator may, for example, comprise a caller ID, a telephone number, a Media Access Control (MAC) address, a radio network temporary identifier (RNTI), an International Mobile Subscriber Identity (IMSI), a Temporary Mobile Subscriber Identity (TMSI), A Packet Temporary Mobile Subscriber Identity (P-TMSI), an International Mobile Equipment Identity (IMEI), or any other such identifier as a skilled person would readily understand. The indicator may, be an identifier for identifying the type of device (e.g. the make and/or model of the device) based upon which the input capability may be looked up (or requested from another device).

According to another aspect of the present invention there is provided server apparatus for a client server system including a mobile device and a client apparatus, the server apparatus comprising: means for hosting an application, the application being operable to accept inputs for controlling functions of the application and to generate outputs as a result; means for receiving, from the mobile device, a request for registration as a controller with the server apparatus, the request including an indicator from which an input capability of the mobile device can be obtained; means for obtaining the input capability of the mobile device using the indicator; means for obtaining input configuration information based on the obtained input capability and functions of the application; means for sending a signal carrying the input configuration information to the mobile device; means for receiving an input event from the mobile device for controlling said application and for providing information relating to the received input event to the application; and means for sending information representing the outputs generated by the application as a result of the provided input event, for output by the client apparatus.

The server may comprise adaptation means for translating the received input events into modified input information for the application. The obtaining means may be operable to obtain the input capability: directly from the indicator; from storage means in which the input capability may be stored, in dependence on the indicator; and/or by requesting another device (or module of the server) based on the indicator.

The input capability may be indicative of at least one input module available on the mobile device, and the configuration information may comprise information identifying the at least one module as being required for controlling at least one function of the application. The input configuration information may comprise a mapping between at least one input means of the mobile device and at least one function of the application being run by the server apparatus. The input configuration information may comprise graphical information for rendering a virtual application control interface on a display of the mobile device (e.g. in the form of: a virtual control panel; individual virtual controls such as buttons, dials, sliders or the like; and/or text instructions to a user as to how physical input devices are configured).

The input configuration information may comprise configuration information for at least one input means available at the mobile device (e.g. a mapping of individual keys or other input devices to program functions).

In response to a change in at least one input required for controlling functions of the application, or in response to a change in the application being run: the obtaining means may be operable to obtain updated input configuration information, based on the obtained input capability; and the means for sending a signal carrying the input configuration information may be operable to send signals carrying the updated input configuration information to the mobile device.

According to another aspect of the present invention there is provided a mobile device for registration in a client server system including server apparatus and client apparatus, the mobile device having: input means for receiving inputs from a user and defining an input capability of the mobile device; means for generating a request for registration as a controller with the server apparatus, the request including an indicator from which the input capability of the mobile device can be obtained; means for sending the request to the server apparatus; means for receiving a signal carrying input configuration information from the server apparatus, the input configuration information being based on the input capability of the mobile device and functions of an application hosted by the server apparatus; means for configuring the mobile device in accordance with said configuration information; and means for sending information identifying user inputs received via the input means to the server apparatus in accordance with said input configuration information.

The mobile device may comprise means for configuring selected inputs and outputs of the mobile device in accordance with the received configuration information.

The input configuration information may comprise a mapping between input means of the mobile device and functions of the application hosted by the server apparatus; and the configuring means may be operable to configure output means of the mobile device, in accordance with the configuration information. For example, the configuring means may be operable to configure output means of the mobile device to output information identifying the mapping to the user in accordance with the configuration information.

The input capability may be indicative of at least one input module available on the mobile device for controlling the input means. The configuration information may comprise information identifying the at least one module as being required for controlling at least one function of the application. The configuring means may be operable to configure the input means, in accordance with the configuration information, to operate under the control of the at least one input module.

The input configuration information may comprise graphical information for rendering a virtual application control interface on the display. The configuring means may be operable to configure the display, in accordance with the configuration information, to display the virtual application control interface. The configuring means may be operable to configure the input means, in accordance with the configuration information, to receive user inputs via the virtual application control interface.

The means for receiving a signal carrying input configuration information may be operable to receive further signals carrying updated input configuration information.

The configuring means may be operable to reconfigure the mobile device, in accordance with the updated configuration information. The means for sending information identifying user inputs may be operable to send the information identifying user inputs to the server apparatus in accordance with the updated input configuration information.

The mobile device may comprise means for receiving information representing outputs generated by the application hosted by the server apparatus, and may comprise means for forwarding the received information for output by the client apparatus.

According to another aspect of the present invention there is provided a client server system comprising a server apparatus, a mobile device and a client apparatus, the server apparatus comprising: means for running an application, the application being operable to accept inputs for controlling functions of the application and to generate outputs as a result; means for receiving, from the mobile device, a request for registration as a controller with the server apparatus and for registering the mobile device with the server apparatus in accordance with said request; means for receiving an input event from the mobile device for controlling said application and for providing information relating to the received input event to the application; and means for sending to the client device information representing the outputs generated by the application as a result of the provided input event, for output by the client apparatus; the mobile device having: input means for receiving inputs from a user; means for generating the request for registration as a controller with the server apparatus; means for sending the request to the server apparatus; and means for sending information identifying user inputs received via the input receiving means to the server apparatus; and the client apparatus comprising: means for receiving the information representing the outputs generated by the application from the server apparatus as a result of the provided input event; and output means for providing an output based on the received information.

Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings in which: Figure 1 shows an embodiment of a thin client system; Figure 2 schematically illustrates the main components of a mobile device forming part of the system shown in Figure 1; Figure 3 schematically illustrates the main components of a thin client server forming part of the system shown in Figure 1; Figure 4 schematically illustrates the main components of a thin client device forming part of the system shown in Figure 1; Figures 5(a) to 5(c) are simplified flow diagrams illustrating operation of the main components of the system in Figure 1 in more detail; Figure 6 is a simplified illustration of one example of how the thin client system of Figure 1 may be used; and Figure 7 is a simplified illustration of another example of how the thin client system of Figure 1 maybe used.

Overview An embodiment of a thin client system will first be described in general terms with reference to Figure 1 which shows, in simplified form, a thin client system generally at 1. As shown in Figure 1, the thin client system I comprises a mobile device 3, a thin client serverS, and a thin client device 7.

The thin client server 5 is configured for hosting software applications (such as a virtual desktop', word processors, databases and spreadsheets, games, web browsers, media players and the like) on behalf of the thin client device 7. It comprises a storage device for holding the software applications 67 and a database 69 of associated input configurations for use in setting up the mobile device 3 as a controller for a hosted software application. The thin client server 5 also comprises a processor for running (hosting) the software applications on behalf of the thin client device 7 in an associated thin client session; for receiving user input from the mobile device 3; for generating output data associated with the software application; and to initiate transmission of the generated output data to the thin client device 7.

In this embodiment, the thin client device 7 comprises a television having a computer processor optimised for receiving and handling the output data generated by the software application being hosted by the server 5.

The mobile device 3 comprises a mobile telephone having a plurality of standard input modules for handling user interactions with a variety of input devices 32 including the sensors of a touch sensitive screen. The input modules include a module for providing a virtual keyboard/keypad, a module for providing and handling a finger controlled cursor, and a module for providing a touch driven menu system.

The mobile device 3 is configured to register with the thin client server 5 as a controller for a software application being hosted by the server 5 on behalf of the thin client device 7. In operation, when registering with the server 5, the mobile device 3 generates information identifying its input capabilities, and in particular information identifying the standard input modules available on the mobile device 3, and sends it to the server 5.

The thin client server 5 is operable, on registration of the mobile device 3, to provide the mobile device 3 with input configuration information, in dependence upon the input capabilities of the mobile device 3 and functions of the application being hosted on behalf of the thin client device 7. As will be described in more detail below, the server 5 also provides updated input configuration information dynamically to the mobile device 3 when the input context changes (for example, as the user navigates an interactive virtual menu system, as the user moves from one virtual control panel to another, or when the user selects another software application etc.) and in parallel sends updated output data to the thin client device 7 to drive the interaction between the user and the main output of the application on the thin client device 7.

The mobile device 3 is operable to set up and dynamically modify the configuration of the mobile device's inputs based on the input configuration information (and to update the configuration dynamically based on the updates to the information) received from the server 5, for example by rendering (and dynamically updating) appropriate virtual application controls (e.g. dials, buttons, keyboards etc.), composing a virtual application control interface on the touch sensitive display or simply by rendering information on a mapping between particular inputs of the mobile device 3 and their assigned functionality for the software application (e.g. indicating the assignment of keys on a virtual/physical keypad to particular control actions such as scroll up, scroll down, scroll left, scroll right etc. of the application).

After the mobile device 3 is registered, and its input devices have been configured, it is ready to act as a controller for the hosted application by accepting user input events (for example, cursor movement, double-click, selection area, zoom, device tilt etc.) and by generating and forwarding information identifying those events to the server 5, which processes the information before forwarding the inputs to the software application. The software application then dynamically changes the output data sent to the thin client device 7 in accordance with its programming and the received input events.

Accordingly, the mobile device 3 acts as a reconfigurable client' input device whilst the thin client device 7 acts as a client output device. In this way, the mobile device 3 provides a flexible and generic' input/control device for controlling a wide variety of software applications of different types and having significantly different control requirements. The input devices 32 of the mobile device 3 are dynamically adapted to the application context, in response to input configuration information provided dynamically by the server 5, while the application is being executed, and while the application's outputs are being rendered on the separate remote thin client device 7 which has better output capabilities.

Where the hosted application is a word processor, for example, the server 5 may configure the mobile device 3 so that it renders a virtual keyboard on the touch sensitive screen and/or will configure the mobile telephone's keypad to accept text inputs for the word processor. The inputs made by the user are transmitted back to the server 5 which uses the inputs to produce the text (graphics for the text) which is transmitted to the thin client device 7 for display to the user. In this way, the text typed in to the mobile device 3 appears on the display of the thin client 7. If the context changes, for example to an embedded application for inserting a photograph in the document, the server 5 may change the input configuration for the mobile device 3, to allow a camera on the mobile device to be used to accept a photographic input for embedding directly into the document.

The mobile device 3, the thin client server 5, and the thin client device 7 will now be described in greater detail with reference to Figures 2 to 4.

Mobile device In Figure 2, the main components of the mobile device 3 are illustrated schematically.

As seen in Figure 2, the mobile device 3 includes a network communication module 45 comprising transceiver circuitry 23 which is operable to transmit signals to and to receive signals from the thin client server 5 via an associated thin client server interface 25. The mobile device 3 also includes a mobile device controller 30 which controls the operation of the mobile device 3 and which is connected to the transceiver circuit 23, to a plurality of output devices 27, and to the input devices 32.

The output devices 27 include an audio output device 29 having a loudspeaker, a display output device 33 comprising the touch sensitive screen and other output devices 28 (such as a vibrating device or a printer). The input devices 32 include an audio input device 31 (a microphone), display sensors 34 for the touch sensitive screen 33 and other input devices 35 (for example, a physical keypad, camera, and proximity/motion/gyroscopic sensors, localisation device, etc.).

The controller 30 operates in accordance with software instructions stored within memory 37. As shown, these software instructions include, amongst other things, an operating system 39, an input event handling module 40, a controller registration module 41, an input update handling module 42 and a contextual input management module 43.

The controller registration module 41 manages the mobile device side registration of the mobile device 3 with the thin client server 5, as a controller for the application being hosted on behalf of the thin client device 7 and establishes the initial configuration for the input devices 32.

The input update handling module 42 is responsible for processing updates regarding the configuration of the input devices 32 of the mobile device 3, after successful registration, when required by the software application being hosted by the server 5 on behalf of the thin client device 7. The updates are received from the thin client server 5 via the thin client server interface 25. The input event handling module 40 is configured to process user initiated input events (for example, movement of a cursor, double-click, selection area, zoom, device tilt, information entry via physical/virtual keyboard etc.) and to provide information identifying the input events to the server 5 for control of the application.

The contextual input management module 43 manages the overall operation of the mobile device 3 as a controller for the thin client device 7 in the context of the software application being executed by the server 5. In accordance with this, the contextual input management module 43 is configured to interact with the various components and modules of the mobile device 3 to ensure that the correct context is maintained, for each input device required by the hosted application, at all times. A context change will typically occur when focus is switched between different applications being hosted by the server 5, for example when a new application is initiated or when the user switches between different applications running in parallel.

A context change may also occur when the focus remains directed to the same application but when the control requirements change, for example as a user navigates a menu system and/or selects a different function of the software application that requires a different set of controls.

Accordingly, the contextual input management module 43 ensures that the correct input modules are selected and correctly configured for each input device 32, that the correct input information is displayed to a user via the display output 33 (for example, virtual controls and/or information on the correct function of the physical controls), and that user initiated input events are correctly registered with the input event handling module 40.

Thin client server In Figure 3 the main components of the thin client server 5 are illustrated schematically.

As seen in Figure 3, the thin client server 5 also includes a network communication module 65 comprising transceiver circuitry 53 which is operable to transmit signals to and to receive signals from: the mobile device 3 via an associated mobile device interface 51; and the thin client device 7 via an associated thin client device interface 55.

As shown, thin client server 5 includes a server controller 50 which controls the operation of the thin client server 5 and which is connected to the transceiver circuit 53. The controller 50 operates in accordance with software instructions stored within memory 57. As shown, these software instructions include, amongst other things, an operating system 59, an input event handling module 60, a controller registration module 61, an input update handling module 62, and an output transmission module 63. Also stored in memory, as described previously, are the software applications 67 which the thin client server 5 may host on behalf of the thin client device 7 and the database of input configurations 69 for the mobile device 3.

The controller registration module 61 manages the server side registration of the mobile device 3, with the thin client server 5, as a controller for the application 67 being hosted on behalf of the thin client device 7; and provides the mobile device 3 with initial configuration information for the mobile device's inputs 32.

The input update handling module 62 is configured to generate updates to the configuration of the mobile device's inputs 32, and to initiate their transmission to the mobile device 3 via the mobile device interface 51, after successful registration, when required to do so by the application being hosted by the server 5 or by analysing the application context.

The input event handling module 60 is configured to process the information identifying the input events received from the mobile device 3 and to submit the input events to the hosted application 67 for handling in accordance with the application's programming.

The output transmission module 63 is operable to manage transmission of output data (graphical and/or audio) and updates to the output data, for the hosted application, to the thin client device 7 via the transceiver circuitry 53 and thin client device interface 55 of the network communication module 65.

Thin client device In Figure 4 the main components of the thin client device 7 are illustrated schematically.

As seen in Figure 4, the thin client device 7 includes a network communication module 85 comprising transceiver circuitry 73 which is operable to transmit signals to and to receive signals from the thin client server 5 via an associated thin client server interface 71.

As shown, thin client device 7 includes a thin client device controller 70 which controls the operation of the thin client device 7 and which is connected to the transceiver circuit 73, to an audio output device 76 having a speaker for outputting sound, and a display device 75 comprising a screen for outputting graphical images.

The controller 70 operates in accordance with software instructions stored within memory 77. As shown, these software instructions include, amongst other things, an operating system 79 and an output reception handling module 83.

The output reception handling module 83 is operable to manage the output data (graphical and/or audio) and the associated updates received from the thin client server 5 via the thin client server interface 71 and the transceiver circuitry 73 and to output the data via the display output device 75 and/or the audio output device 76 as required. Although not shown, the thin client device 7 may also include one or more input devices via which users can control the operation of the application being run on the remote server.

The operation and features of the mobile device 3, thin client server 5 and thin client device 7 will now be described, by way of example only, in further detail with reference to Figures 5(a) to 5(c) which are simplified flow diagrams illustrating the operation of the main components of the system of Figure 1.

Mobile Device Registration and Configuration Figure 5(a) is a flow diagram illustrating exemplary operation of the mobile device 3, thin client server 5, and thin client device 7 prior to, and during, registration of the mobile device 3 with the (or with a particular thin client session on the) thin client server 5.

In this example, prior to initiation of the registration procedure the thin client session is initiated between the server 5 and the thin client device 7 at step S500 and a software application 67 is launched on the server 5 (step S502) on behalf of the thin client device 7 in the thin client session (a plurality of such applications may be launched). The software application 67 runs (S504) in parallel with the controller registration procedure, generating output data which, as discussed previously, is processed and transmitted to the thin client device 7 by the output transmission module 63 on the server 5. Accordingly, the output reception module 83 in the thin client device 7 receives and processes the output data received from the server 5 and outputs it via the appropriate output device 75, 76 (S702).

The controller registration module 41 of the mobile device 3 initiates the registration procedure by generating and transmitting a registration request to the server 5 at S300 (via network communication module 45) in response to a user initiated input event. The registration request includes a mobile device ID (for example an Internet Protocol address, a Uniform Resource Identifier, Internal Mobile Equipment Identity etc) for identifying the mobile device 3, a session ID (for example a URI, a number etc) for identifying the thin client session which the device is to be registered and/or a thin client ID (for example an IP address, a URI, a number or a geographical location) for identifying the output device with which the device is to be registered.

The registration request also includes the information identifying the mobile device's input capabilities including the information identifying the available standard input modules (and/or input devices 32) as described previously. For example, in this embodiment the mobile device 3 includes a standard input module for rendering a virtual keyboard on the touch sensitive screen and, accordingly, the information identifying the mobile device's input capabilities indicates that a virtual keyboard module is available.

The controller registration module 61 of the server 5 receives the registration request at S506 and processes the request to determine the input capabilities of the mobile device 3 making the request. The controller registration module 61 then searches and extracts one or more input configurations, corresponding to the determined input capabilities determined for the mobile device 3 (S508), from the associated database 69 in dependence on the application being hosted on behalf of the client device 7 and its current context (or the application which has the current focus where multiple applications are running in parallel with the thin client 7).

The controller registration module 61 of the server 5 then generates a registration information signal that includes information identifying the extracted input configuration and transmits the message to the mobile device 3 at S510 via the transmission circuitry 53 and the mobile device interface 51.

The controller registration module 41 of the mobile device 3 receives the registration information message at S302 (via network communication module 45) and processes the message to determine the required input modules and/or devices and the appropriate configuration for those modules/devices. In this embodiment the configuration information indicates that the standard input module for rendering the virtual keyboard should be used. In another embodiment the configuration information may include graphical information for rendering a control panel directly on the touch sensitive screen without using a standard input module available on the mobile device 3. In yet another embodiment the configuration information may include both standard input module and graphical information to be displayed on mobile device 3.

The controller registration module 41 passes the received input configuration information to the contextual input management module 43. The contextual input management module (re)configures the input modules/devices (for example, by rendering appropriate application controls on the touch sensitive display) based on the input configuration information, whilst taking any technical constraints (for example, display resolution/size) imposed by the design of the mobile device 3 into account. In this way, the mobile device 3 is configured to form a remote controller for the application 67 being hosted by the server 5 and displayed on the screen of the thin client device 7.

Once the mobile device inputs 32 have been configured, the controller registration modules 41, 61 of the mobile device 3 and the thin client server 5 complete the registration process at step S306 allowing input handling to begin at step S308. In this embodiment this is achieved by the controller registration module 41 of the mobile device 3 confirming that the input configuration has been completed and the controller registration module 61 of the thin client server 5 acknowledging registration completion.

Accordingly, the mobile device 3 is now ready to act as a remote controller for the software application hosted by the server 5.

In put Event Handling Figure 5(b) is a flow diagram illustrating exemplary operation of the mobile device 3, to accept and handle user initiated input events for controlling the application hosted by the thin client server 5 on behalf of thin client device 7.

Prior to operation of the mobile device 3, as a controller for the hosted application, the thin client session is initiated (S500), the software application 67 is launched (S502), and the mobile device 3 is registered as a controller (S310), as described with reference to Figure 5(a). The software application 67 runs (S504) in parallel with the controller input event handling procedure, generating output data which, as discussed previously, is processed and transmitted for output by the thin client device 7 (S702).

The input event handling module 40 on the mobile device 3 accepts inputs (user input events) made by a user via the input devices 32 at step S312, in accordance with the configuration of the input devices 32 (or the modules controlling them). The input event handling module 40 then processes the user input events at step S314 and, depending on the nature of the events, handles them locally on the mobile device 3 (e.g. for simple cursor movements or double-click user action) and/or generates information identifying the events for transmission to the server 5 (e.g. on selection of a particular function by clicking' a virtual button or turning a virtual dial).

In the latter case, the input event handling module 40 initiates transmission of the user input event information (generated at step S314) to the thin client server 5 via the thin client server interface 25 at S316.

The input event handling module 60 of the thin client server 5 receives the information identifying the user input events at S512, processes the received information to determine the nature of the associated input event and provides the input event as an input to the software application 67 at S514. The software application 67 responds to the input event in accordance with its programming.

When the software application 67 responds with updated output data, the output transmission handling module 63 manages the transmission of this updated output data to the thin client device 7 (S505), which then updates the graphical/audio output accordingly (S702) as described previously.

Dynamic Input Configuration Update Figure 5(c) is a flow diagram illustrating exemplary operation of the mobile device 3, thin client server 5, and thin client device 7, to manage dynamic changes to the input configuration.

As seen in Figure 5(c), and as described previously, the thin client session is initiated (S500), the software application 67 is launched (S502), the mobile device 3 is registered as a controller and begins accepting user initiated input events (S310, S308). The software application runs (S504) in parallel with the input update handling procedure, generating output data which, as discussed previously, is processed and transmitted (S505) for output by the thin client device 7 (S702).

The input update handling module 62 of the server 5 monitors the application context and detects, at step S516 when there is a change in context which requires a corresponding modification in input configuration. The input update handling module 62 analyses the change in context and, in dependence on the nature of the context change, generates updated input configuration information and/or extracts updated input configuration information from the input configuration database 69 at S518. In this embodiment the context change is initiated by the hosted application (e.g. on completion or initiation of a particular process of the application). In other embodiments, however, the context change may be initiated directly in response to an input event submitted via the mobile device 3, in response to an input event made directly to an input device 32 of the mobile device 3 (for example requesting a change in the application which has the current focus), and/or in response to an error.

The input update handling module 62 generates a configuration update signal that includes information identifying the updated input configuration and transmits the message to the mobile device 3 at S520 via the transmission circuitry 53 and the mobile device interface 51.

The input update handling module 42 of the mobile device 3 receives and translates the configuration update signal and submits the updated input configuration information to the contextual input management module 43. The contextual input management module 43 then reconfigures the input modules/devices based on the updated input configuration information. Accordingly, the mobile device 3 is dynamically adapted by modifying the controls available to a user in dependence on the application context.

Two exemplary implementations of the thin client system will now be described, to illustrate the flexibility of the system, with reference to Figures 6 and 7 Exemplary Implementation 1: Remotely Rendered Control Area In Figure 6 a thin client is shown comprising a mobile telephone 3 which is registered with a thin client server 5 as a remote controller for an application hosted by the server 5 on behalf of a thin client device 7 (comprising a television).

The mobile telephone 3 has a touch sensitive screen 33 which has been configured with a virtual remote control panel 90 for the hosted application (e.g. a control panel for operating a graphic adventure computer game, and for displaying information about game play to the user such as inventory items, player health, scoring information etc.). The virtual control panel in this embodiment is remotely rendered and dynamically updated by the server 5 in its entirety.

As seen in Figure 6, the virtual remote control panel is larger than the screen size and accordingly only part of the control panel is displayed at a time.

The mobile telephone 3 is configured to receive input events comprising, amongst other things, a user touching the screen and dragging' the control panel, by sending information identifying the user's action directly to the server 5 which responds to the input event by sending updated input configuration information dynamically to the mobile telephone 3 to give the impression of moving (or panning) the control panel to display a previously unseen portion of the panel 90. In the case of the graphic adventure game software application 67, for example, the updated output information may comprise an unseen part of the inventory from which the user may then make a desired selection.

The mobile telephone 3 is also configured to receive input events indicative of a user's desire to zoom in or zoom out and to send information identifying the user's action to the server 5 (arrow 99) which responds to the input event by sending updated input configuration information dynamically to the mobile telephone 3 (arrow 98) to enlarge or shrink the control panel 90 accordingly.

When the user makes an input which has a bearing on the operation of the software application (for example to select a particular function) the input event is sent directly to the server 5 by the mobile telephone 3 which then submits the input event to the software application 67 for processing in accordance with its programming. Where the input event results in changes to the output of the software application, associated updated output data is transmitted to the television 7 which updates its outputs accordingly.

In this implementation, therefore, the server 5 effectively handles all the input events remotely, and sends updated information for reconfiguring the display of the control panel accordingly. This demonstrates the high level of flexibility, provided by the described embodiments, to render control panels of many different types on mobile devices without requiring time consuming and inconvenient download of additional software onto the mobile device itself. Furthermore, when the software application 67 is updated any changes in control requirements are automatically implemented as part of that update without requiring the reconfiguration of several mobile devices.

Exemplary Implementation 2: Remotely and Locally Rendered Control Area In Figure 7 a thin client system is shown comprising a mobile telephone 3 which is registered with a thin client server 5 as a remote controller for an application 67 hosted by the server 5 on behalf of a thin client device 7 (comprising a television).

The mobile telephone 3 has a touch sensitive screen 33 which has been configured with a virtual remote control panel 90 for the hosted application. The virtual control panel in this embodiment comprises a remotely rendered area 92 which is rendered remotely by the server 5 and another area 94 which is rendered locally using a standard input module.

In this embodiment, when the mobile telephone 3 receives inputs to the remotely rendered area 92 it transmits information identifying the user's action directly to the server 5 (arrow 99) which responds to the input event appropriately. For example, in this embodiment, the remotely rendered area 92 comprises a plurality of virtual dials and slider bars (e.g. for controlling an audio equaliser software application).

Accordingly, when a user touches the screen to move one of the sliders or to turn one of the dials, the mobile device 3 sends information identifying these input events to the server 5. The server 5 responds by sending updated input configuration information back to the mobile device 3 (arrow 98) which causes the mobile device 3 to update the control panel 90 by displaying the slider! dial in the new position and making any other associated changes (for example by changing a displayed numeric value representing the position of the slider! dial).

At the same time, the server 5 submits the input event to the software application 67 which responds in accordance with its programming by sending updated output information to the television 7. In the case of the audio equaliser software application 67, for example, the updated output information may comprise updated audio output information in the form of a reduced volume or a modified balance!bass!treble etc. and!or updated graphical information representing the modified audio output.

The locally rendered area 94 comprises a virtual keyboard based on a standard input module stored on the mobile telephone 3. When the mobile telephone 3 receives inputs to the locally rendered area 94 it deals with the input locally, in accordance with the configuration of the standard input module, before transmitting the information identifying the input events to the server 5.

Accordingly, when different mobile telephones having different standard input modules register with the server 5 the remotely rendered area 92 will remain substantially the same (subject to any constraints imposed by the design of the mobile telephone) while the locally rendered area 94 may be different, for example having a different style of virtual keyboard. Moreover, if the mobile telephone has a physical keyboard available the input configuration provided by the server 5 may indicate that this should be configured by the mobile device 3 instead of (or in addition to) the virtual keyboard. For instance the server 5 may send input configuration comprising information indicating that both the virtual keyboard and the physical keyboard can be used and information configuring the remotely rendered area 92 with the display sensors 34 attached to it. It is then up to the mobile device 3 to decide how the virtual and physical keyboards are used.

In this implementation, therefore, the server 5 handles only some of the input events and some of the control panel rendering remotely, allowing other input events and some control panel rendering to be handled locally on the mobile device 3. This provides the dual benefit of reducing the amount of data transmitted between the server and the mobile device (when the mobile device has an appropriate input capability pre-installed), and providing a user with a look and feel' consistent with other mobile device based applications thereby making familiarisation with a new set of controls more intuitive.

This further demonstrates the flexibility of the proposed embodiments to allow mobile devices of different types and configurations to be used as remote controllers for the same hosted software application.

Modifications and Alternatives A number of detailed embodiments have been described above. As those skilled in the art will appreciate, a number of modifications and alternatives can be made to the above embodiments whilst still benefiting from the inventions embodied therein.

In the embodiments described above, the mobile device 3, server 5 and thin client device 7 each include transceiver circuitry. Typically this circuitry will be formed by dedicated hardware circuits. However, in some embodiments, part of the transceiver circuitry may be implemented as software run by the corresponding controller.

In the above embodiments, a number of software modules were described. As those skilled in the art will appreciate the software may be provided in compiled or un-compiled form and may be supplied to the mobile device 3, server 5 and/or thin client device 7 as a signal over a computer network, or on a recording medium. Further, the functionality performed by part or all of this software may be performed using one or more dedicated hardware circuits. However, the use of software modules is preferred as it facilitates the updating of the mobile device 3, server 5 and/or thin client device 7 in order to update their functionalities. The functionality of one or more of the modules may be combined into a single module and in some embodiments may be built into the operating system.

Operation of the mobile device 3, server 5 and thin client device 7 has been described sequentially with reference to the flow diagrams (Figures 5(a) to 5(c)) for the purposes of clarity. It will be appreciated, however, that many of the steps need not run sequentially but may run in parallel with other steps.

It will be appreciated that whilst, in the implementations described above, input configuration information is provided to the mobile device 3 to allow the mobile device's inputs to be modified to the context of the application, the server 5 may equally provide other configuration information, for example output configuration information to allow the mobile device's outputs to be configured to the context of the application. For example, the output configuration information may include contextual output information for configuring vibratory, graphical and/or audio output devices of the mobile device 3 to provide associated feedback to the user directly through the control device and in doing so provide an enhanced and more interactive control experience for the user. The configuration information for the mobile device's output may also be used to provide information to the user defining the mapping between the available inputs of the mobile device and the inputs to the remotely run application.

Moreover, whilst the embodiment is described with reference to configuration of the mobile device 3 to act as a remote controller for software applications, the mobile device 3 could be configured to act as a remote controller for hardware applications.

For example, the mobile device 3 could be registerable with a device (e.g. a computer / microprocessor) for controlling operation of manufacturing equipment in a production environment or the like, as a remote controller for that equipment. In this exemplary scenario, an output terminal for displaying parametric, process verification, quality control, work in progress or other information relating to the manufacturing process may be thought of as being equivalent to the thin client device 7. Accordingly, the touch screen of the mobile device 3 may be configured with a locally or remotely rendered control panel for controlling the equipment and/or for changing process related parameters.

Whilst the embodiments have been described with reference to a single mobile device 3, a single server 5 and a single client device 7 it will be understood that a plurality of controllers (and/or thin client devices) may be registered with the server 5 for a single application or for multiple applications. Such configurations may allow, for example, multi-user functionality (such as multi-player games, conference calls, or the like) both when the users are located at the same location and can therefore use the same client device 7 and when they are located at different locations and use different thin client devices 7.

It will be further appreciated that communication between the mobile device 3 and the server 5 and between the server 5 and thin client device 7 (and mobile device to thin client device communications where applicable) may use any suitable wireless or wired communication protocol and associated technology. For example the respective network communication portions 45, 65, 85 may be configured for communication using a Bluetooth and/or WiFi protocol in the case of radio communication, an IrDA (Infrared Data Association) protocol in the case of infrared communication and/or a Universal Serial Bus (USB) protocol in the case of wired communication. The mobile device 3, server 5, and/or thin client device 7 may also be configured for long range communication via a mobile telephone network (e.g. via base station, radio network controller, core network etc) to allow long range remote control (for example, to allow control of a thin client device 7 comprising an interactive white board in a conference call having participants at different geographic locations or to allow games to be played by different players at different geographical locations). The communication protocol/technology used by the network communication portions 45, 65, 85 may be user configurable. A different protocol/technology may be used for mobile device to server communications than for server to thin client device communications (and/or mobile device to thin client device communications).

In Figure 3, whilst the software applications 67 and the database of input configurations 69 are shown as being stored on the same storage device (memory 59) as the controller software they may be stored on one or more other storage devices such as a hard disk, solid state memory device or the like.

Registration of the mobile device 3 with the server 5 as described with reference to Figure 5(a) may take place prior to initiation of the thin client session initiated between the server 5 and the thin client device 7 and prior to launch of the software application, or may take place between initiation of the thin client session and launch of the software application. Accordingly, the mobile device 3 may be operable to initiate the thin client session (in the former case) and/or launch of the software application (in the former or latter case) in its role as a controller registered with the server.

Whilst many of the communication steps described above only involve communication of a single message for reasons of clarity, the communication steps may comprise sending a plurality of messages. The registration request may, for example, comprise a plurality of messages comprising a message requesting registration, one or more messages providing connection information (e.g. a mobile device ID, a session ID, and/or a thin client ID), and/or at least one other message/signal carrying the information identifying the mobile device's input capabilities. Similarly, the transmission of configuration update information may comprise communication of a plurality of signals comprising, for example, a message for acknowledging / confirming the registration request in response to one or more messages requesting connection or other information from the mobile device, and/or one or more messages carrying the configuration information. It will be apparent to a skilled reader that where a communication step involves sending a plurality of messages each message will be sent at a time appropriate to the nature of the message. For example, the acknowledgement / confirmation message may be sent by the server 5, in response to the message requesting registration, before the message carrying input capability information is sent by the mobile device 3.

Although updates to the outputs of the thin client device 7 (e.g. graphics updates) have been described as being sent by the server 5 directly to the client device 7 it will be appreciated that the updates may be sent via the mobile device 3, the mobile device 3 acting in this case as an intermediary proxy for forwarding graphics or other output data between the thin client device 7 and the server 5.

In applications where only a single mobile device 3 will act as a controller at a time then provision of a mobile ID may not be necessary. In applications where only a single thin client session will be in operation on the server 5 at a time then provision of a thin client session ID may not be necessary. In applications where only a single thin client device 7 will be in use at a time, or when the output data is transmitted via the mobile device (acting as an intermediate proxy) then provision of a thin client device ID may not be necessary.

The thin client session and/or thin client device 7 with which the mobile device 3 is to be registered may be identified in an iterative manner with the server 5 providing a list of available thin client sessions and/or thin client devices 7 to a user via the mobile device 3 and the user selecting the appropriate thin client session/device by making an appropriate input. This is of particular benefit in an application where there is only a limited number of sessions and/or thin client devices to select from.

In other applications, there may be a multiplicity of thin client devices to choose from (for example, where the server is an internet server arranged to host applications for output to thin client devices at geographically remote locations such as in hotel rooms or in airport IT suites). In such applications it is advantageous for the registration request to include a thin client device ID (such as but not restricted to an IP address, URI, localisation information, etc.) or a session ID which may, for convenience, be displayed on the thin client device 7 for subsequent entry to the mobile device 3 by the user or automatically retrieved (such as using camera, NFC, localisation information etc.), when initiating the registration request.

The server 7 need not be remote from the mobile device 3, it may for example be located in a home belonging to the user of the mobile device 3, the user registering the mobile device 3 with the server at relatively short range using an appropriate short range wireless (or possibly wired) communication protocol.

Whilst it is particularly advantageous, in terms of flexibility, for the input capabilities of a mobile device 3 to be sent during the registration procedure, the input capabilities could be pre-stored on the server 5 (e.g. in a database) in association with the mobile device ID (or some other indicator identifying the type of mobile device). Accordingly, the registration request may include the mobile device ID (and/or information identifying its type) but not the input capability information. On receipt of such a registration request, the server 5 would use the mobile device ID (or the information identifying the type of mobile device) to obtain the input capability information for the mobile device from memory.

Although the thin client device 7 has been described as a television it may comprise any suitable output device having a graphical, audio and/or other output capabilities although the embodiments are particularly beneficial for graphical output applications. In other embodiments the thin client device 7 may also have input capabilities which allow a user to make inputs directly to the thin client device 7 for transmission to the server 5 for submission to the hosted software application.

It will be appreciated that although the mobile device 3 is described as a mobile telephone (e.g. a Smartphone) it may be any suitable device for example a personal digital assistant (PDA), palmtop or notebook computer.

The input devices of the mobile device may include any suitable device commonly available on such devices, for example a mouse, physical keyboard/keypad, camera, microphone, motion/proximity sensors, localisation device, etc. The input modules may include any suitable modules for handling inputs to the input devices and/or for displaying information on how they are configured for use by a user. For example, the input modules may include modules for handling the input of images (still or moving) via a camera, modules for recognising voice inputs to a microphone, modules for interpreting the speed and/or nature of movements, etc. Various other modifications will be apparent to those skilled in the art and will not be described in further detail here.

Claims

Claims 1. A client server system comprising server apparatus, a mobile device and a client apparatus, the server apparatus comprising: means for running an application, the application being operable to accept inputs for controlling functions of the application and to generate outputs as a result; means for receiving, from the mobile device, a request for registration as a controller with the server apparatus, the request including an indicator from which an input capability of the mobile device can be obtained; means for obtaining the input capability of the mobile device using the indicator; means for obtaining input configuration information based on the obtained input capability and functions of the application; means for sending a signal carrying the input configuration information to the mobile device; means for receiving an input event from the mobile device for controlling said application and for providing information relating to the received input event to the application; and means for sending information representing the outputs generated by the application as a result of the provided input event, for output by the client apparatus; the mobile device having: input means for receiving inputs from a user and defining the input capability of the mobile device; means for generating the request for registration as a controller with the server apparatus, the request including the indicator from which the input capability of the mobile device can be obtained; means for sending the request to the server apparatus; means for receiving the signal carrying the input configuration information from the server apparatus, the input configuration information being based on the input capability of the mobile device and functions of the application run by the server apparatus; means for configuring the mobile device in accordance with said configuration information; and means for sending information identifying user inputs received via the input means to the server apparatus in accordance with said input configuration information; and the client apparatus comprising: means for receiving the information representing the outputs generated by the application from the server apparatus as a result of the provided input event; and output means for providing an output based on the received information.
2. Server apparatus for a client server system including a mobile device and a client apparatus, the server apparatus comprising: means for hosting an application, the application being operable to accept inputs for controlling functions of the application and to generate outputs as a result; means for receiving, from the mobile device, a request for registration as a controller with the server apparatus, the request including an indicator from which an input capability of the mobile device can be obtained; means for obtaining the input capability of the mobile device using the indicator; means for obtaining input configuration information based on the obtained input capability and functions of the application; means for sending a signal carrying the input configuration information to the mobile device; means for receiving an input event from the mobile device for controlling said application and for providing information relating to the received input event to the application; and means for sending information representing the outputs generated by the application as a result of the provided input event, for output by the client apparatus.
3. Apparatus as claimed in claim 2 wherein the server further comprises adaptation means for translating the received input events into modified input information for the application.
4. Apparatus as claimed claim 2 or 3 wherein the obtaining means is operable to obtain the input capability directly from the indicator.
5. Apparatus as claimed in any of claim 2 or 4 wherein the obtaining means is operable to obtain the input capability from storage means in which the input capability is stored, in dependence on the indicator.
6. Apparatus as claimed in any of claims 2 to 5 wherein the input capability is indicative of at least one input module available on the mobile device, and wherein the configuration information comprises information identifying the at least one module as being required for controlling at least one function of the application.
7. Apparatus as claimed in any of claims 2 to 6 wherein the input configuration information comprises a mapping between at least one input means of the mobile device and at least one function of the application being run by the server apparatus.
8. Apparatus as claimed in any of claims 2 to 7 wherein the input configuration information comprises graphical information for rendering a virtual application control interface on a display of the mobile device.
9. Apparatus as claimed in any of claims 2 to 8 wherein the input configuration information comprises configuration information for at least one input means available at the mobile device.
10. Apparatus as claimed in any of claims 2 to 9 wherein, in response to a change in at least one input required for controlling functions of the application, or in response to a change in the application being run: the obtaining means is operable to obtain updated input configuration information, based on the obtained input capability; and the means for sending a signal carrying the input configuration information is operable to send signals carrying the updated input configuration information to the mobile device.
11. A mobile device for registration in a client server system including server apparatus and client apparatus, the mobile device having: input means for receiving inputs from a user and defining an input capability of the mobile device; means for generating a request for registration as a controller with the server apparatus, the request including an indicator from which the input capability of the mobile device can be obtained; means for sending the request to the server apparatus; means for receiving a signal carrying input configuration information from the server apparatus, the input configuration information being based on the input capability of the mobile device and functions of an application hosted by the server apparatus; means for configuring the mobile device in accordance with said configuration information; and means for sending information identifying user inputs received via the input means to the server apparatus in accordance with said input configuration information.
12. A mobile device as claimed in claim 11 further having means for configuring selected inputs and outputs of the mobile device in accordance with the received configuration information.
13. A mobile device as claimed in claim 11 or 12 wherein the input configuration information comprises a mapping between input means of the mobile device and functions of the application hosted by the server apparatus; and wherein the configuring means is operable to configure output means of the mobile device, in accordance with said configuration information to output information identifying the mapping to the user.
14. A mobile device as claimed in any of claims 11 to 13 wherein the input capability is indicative of at least one input module available on the mobile device for controlling said input means; wherein the configuration information comprises information identifying the at least one module as being required for controlling at least one function of the application; and wherein the configuring means is operable to configure the input means, in accordance with said configuration information, to operate under the control of the at least one input module.
15. A mobile device as claimed in claim 11 to 14 comprising a display; wherein the input configuration information comprises graphical information for rendering a virtual application control interface on the display; and wherein the configuring means is operable to configure the display, in accordance with said configuration information, to display the virtual application control interface.
16. A mobile device as claimed in claim 15 wherein the configuring means is further operable to configure the input means, in accordance with said configuration information, to receive user inputs via said virtual application control interface.
17. A mobile device as claimed in any of claims 11 to 16 wherein the means for receiving a signal carrying input configuration information is operable to receive further signals carrying updated input configuration information; wherein the configuring means is operable to reconfigure the mobile device, in accordance with the updated configuration information; and wherein the means for sending information identifying user inputs is operable to send the information identifying user inputs to the server apparatus in accordance with the updated input configuration information.
18. A mobile device as claimed in any of claims 11 to 17 further comprising: means for receiving information representing outputs generated by the application hosted by the server apparatus; and means for forwarding the received information for output by the client apparatus.
19. A client server system comprising a server apparatus, a mobile device and a client apparatus, the server apparatus comprising: means for running an application, the application being operable to accept inputs for controlling functions of the application and to generate outputs as a result; means for receiving, from the mobile device, a request for registration as a controller with the server apparatus and for registering the mobile device with the server apparatus in accordance with said request; means for receiving an input event from the mobile device for controlling said application and for providing information relating to the received input event to the application; and means for sending to the client device information representing the outputs generated by the application as a result of the provided input event, for output by the client apparatus; the mobile device having: input means for receiving inputs from a user; means for generating the request for registration as a controller with the server apparatus; means for sending the request to the server apparatus; and means for sending information identifying user inputs received via the input receiving means to the server apparatus; and the client apparatus comprising: means for receiving the information representing the outputs generated by the application from the server apparatus as a result of the provided input event; and output means for providing an output based on the received information.