WO2013045470A1 - A method and a system to remotely share an application between electronic devices - Google Patents

A method and a system to remotely share an application between electronic devices Download PDF

Info

Publication number
WO2013045470A1
WO2013045470A1 PCT/EP2012/068916 EP2012068916W WO2013045470A1 WO 2013045470 A1 WO2013045470 A1 WO 2013045470A1 EP 2012068916 W EP2012068916 W EP 2012068916W WO 2013045470 A1 WO2013045470 A1 WO 2013045470A1
Authority
WO
WIPO (PCT)
Prior art keywords
application
user
content
per
real
Prior art date
Application number
PCT/EP2012/068916
Other languages
French (fr)
Inventor
Javier GARCÍA PUGA
Javier MARTÍNEZ ÁLVAREZ
Roberto PÉREZ CUBERO
Original Assignee
Telefonica, S.A.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonica, S.A. filed Critical Telefonica, S.A.
Publication of WO2013045470A1 publication Critical patent/WO2013045470A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/541—Interprogram communication via adapters, e.g. between incompatible applications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/542—Intercept
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G06F40/174—Form filling; Merging

Definitions

  • the present invention generally relates, in a first aspect, to a method to remotely share an application between electronic devices, said electronic devices having equal or different capabilities, and more particularly to method that allows giving remote control of only a part of the desktop by sending to the remote user elements of the application chosen by the host user, said elements being described in a neutral language and being displayed to the remote user in an adapted way according to the characteristics of the remote electronic device.
  • a second aspect of the invention relates to a system arranged to implement the method of the first aspect.
  • One of the most common interactions is when one user shares a part of his desktop to another user. For instance, a user could be using his PC to book a flight for himself and a friend, and at some step, personal information of his friend could be required. Then he could share with his friend that part of the input form, so he could fill in from his mobile.
  • control For instance, the user who is sharing (“the sharer") should have the full control of what he is sharing and until when this sharing will be active.
  • the same concept could be applied for real time collaboration in a document or presentation. In this case the user could just predefine the page or paragraph which he would like to give control to his friend.
  • the sharer should have the full control of what he is sharing and until when this sharing will be active.
  • the same concept could be applied for real time collaboration in a document or presentation. In this case the user could just predefine the page or paragraph which he would like to give control to his friend.
  • RDP Remote Desktop Protocol
  • Microsoft Remote Desktop Protocol
  • This approach has a couple of important drawbacks. In first place it requires that the devices have any kind of permanent connection with the servers. If the connection breaks, then the remote access is stalled or closed until it is reconnected again. On the other hand, it works with a full desktop environment, not just with a set of Ul elements. Additionally, the user experience in small screen devices is very poor, since the virtual desktop needs to be rescaled to the smaller screen of a smartphone.
  • Citrix XenApp it is an on-demand application delivery solution that enables any Windows application to be virtualized, centralized, and managed in the datacenter and instantly delivered as a service to users anywhere on any device.
  • this virtualization- based alternative has the same problems as the previous one.
  • This invention generally relates to the field of remotely controlling a first computer from a second computer, where the first computer has a graphical user interface (GUI) and the second computer may have a dissimilar user interface and a dissimilar operating environment from the first computer.
  • GUI graphical user interface
  • the way of transmitting the remote desktop or application is by means of image screenshots. Therefore no layout adaptation or resizing is performed to be optimized for different computing devices such as smartphones.
  • the remote control is limited to the level of the whole desktop or applications, not to single Ul elements (i.e. a set of input boxes) as it is defined in the current invention.
  • Mission control is a software application for Linux Operating system which uses XMPP to transport the VNC data of a typical desktop sharing session. As the rest of the methods of desktop sharing it uses images or an image-based protocol to transfer the elements of the desktop. Indeed, it is only available to PC devices and it cannot be used on mobile devices or via web sites.
  • This protocol can be used to share the elements of the desktop between different peers. Indeed this is the protocol used in Unix systems to layout the graphical desktop. Although it doesn't use an image based protocol, it doesn't allow sharing the control between different users and it doesn't support content adaptation for each device.
  • Some virtualization-based solutions require that the devices have any kind of permanent connection with the servers. If the connection breaks, then the remote access is stalled or closed until it is reconnected again.
  • the present invention provides, in a first aspect a method to remotely share an application between electronic devices, said electronic devices having equal or different capabilities.
  • the method of the invention in a characteristic manner it further comprises:
  • a second aspect of the present invention concerns to a system to remotely share an application between electronic devices, said electronic devices having equal or different capabilities.
  • the system of the second aspect of the invention on contrary to the known systems mentioned in the prior state of the art section, and in a characteristic manner it comprises using a real-time hub module in said electronic devices in order to:
  • the system of the second aspect of the invention is adapted to implement the method of the first aspect.
  • Figure 1 shows the overall view of the invention architecture according to an embodiment of the present invention.
  • Figure 2 shows three different approaches to grab the components and the contents of the controls selected by the host user, according to an embodiment of the present invention.
  • Figure 3 shows the XML schema followed by the XMPP message structure that transports the description and the state of the host control, according to an embodiment of the present invention.
  • Figure 4 shows a concrete example of the XMPP message structure that transports the description and the state of the host control, according to an embodiment of the present invention.
  • Figure 5 shows the traffic that flows in real-time through XMPP network when connection between the host user and the remote user has been established, according to an embodiment of the present invention.
  • Figure 6 shows the XML schema followed by the XMPP message structure that transports the modifications introduced by the remote user, according to an embodiment of the present invention.
  • Figure 7 shows a concrete example of the XMPP message structure that transports the modifications introduced by the remote user, according to an embodiment of the present invention.
  • Figure 8 shows the general steps of the method, according to an embodiment of the present invention.
  • the invention consists of a system that allows a real-time interaction of a remote user with selected components of an application placed in the computer of a given user.
  • the invention also contemplate the remote user could have a different device with different capabilities. If the destination device is different from the source, the contents should be adapted to be displayed in the low-capable device.
  • the host will select the user interface components (i.e. A web form input box or a board of a chess game) he wants to share with his guest, and then the state of those components will be grabbed, adapted and send through the network, allowing his guest to interact with those components in real time.
  • the user interface components i.e. A web form input box or a board of a chess game
  • the different devices (101 ) which a user may have are the main participants in the idea proposed in this invention. These devices can be any kind of computing elements such as desktop computers, mobile phones, tablets, netbooks, etc.
  • the user of a given device might want to give remote control of one of his applications (103) (or a part of it) for a short period of time to a user using a different device.
  • Real-Time Hub a key element of this invention which is called Real-Time Hub (104).
  • the Real-Time hub is able to do five main tasks:
  • this real-time hub will modify the content of the controls involved in the session.
  • the Real-Time hub plays the role of the receiving one, it will display the controls and their content in the best way the device can show them. Allowing the user to modify their contents.
  • the hub When the user decides to start the real-time communication, he sends a request to the remote device using the real-time hub (104). To establish the communication, the hub will use a standard XMPP mechanism to inform the remote user the intention of the local user to start a remote control-session.
  • the communications server (102) acts as a connection point between the peers, it will coordinate the first steps in the communication setup and will act as a contact server. Thanks to this functionality, the users will be able to log on to the service and manage the contacts they will be using the service with. Moreover, the communication server will route the XMPP messages from the host client to the remote client.
  • the main tasks of the communications server can be described as follows: - Maintain the contacts list of each user. At the beginning of the connection process the user will select one of his contact to communicate with
  • the communication server can be any standard XMPP server with the appropriate extensions installed.
  • connection process can be described in five steps:
  • the real-time hub will ask the communications server the contacts of the current user
  • the real-time hub will present the user the set of contacts which are available to start a remote-control session
  • the real-time hub will try to establish the connection with the remote real-time hub by using the communications server as a helper.
  • the remote real-time hub will inform the remote user the will of the local user. If the communication is accepted by the remote user, the hub will begin by asking the user for the controls that are going to be remote controlled. The user will select them using his mouse and selecting them from the applications themselves. Once the controls are selected, the Real-Time Hub will try to grab the components and the contents of them using three different approaches, as shown in Figure 2:
  • This method is used when the application allows some kind of extensibility, giving the possibility of writing custom plugins to extend its functionality. Not only is needed the application to be extended with custom plugins, but also the application should allow the plugin to grab and set its state. Examples of this method would be a Microsoft Office Plugin, or a Firefox browser plugin.
  • the real-time hub will communicate with the plugin and ask for the description of the selected elements and their current contents.
  • the Real- Time hub will also communicate with the plugin to set the content of the controls when the remote user changes its contents.
  • the solution implies a set of procedure calls implemented in the Device-dependant communications hub which will be used by the third-party application.
  • These third party applications will implement the methods required by the provided interface and will able to send the state of the application themselves, and without requiring extra work by anybody else.
  • the Real-Time Hub will communicate with the extension point provided by the application developer directly. This scenario is similar to the first one, but developer of the application will be the one who will implement the communication mechanism.
  • the real-time hub will use this method to grab the description of the elements and to grab and set their contents. Once the real-time hub has obtained the description of the elements and their content, it will pack the data in a standard container ready to be sent through the network.
  • the invention uses the XMPP message structure as a container to transport the description and the state of the host control.
  • the application state data is where the host application serializes its state, and it is the data which uses the receiver to layout the component and allow the remote user to interact with the control which was sent.
  • the content of the controls are placed inside the description XML. This data is packed inside a standard XMPP IQ message and then sent to the remote client.
  • this device-neutral language is to allow each real-time hub to adapt the contents to their best way to show them to the user. For example, if the remote real-time hub is installed in a mobile device, it will render those controls a different way that if the remote-real time hub is installed in a PC computer.
  • the remote client's Real-Time hub will render the content using the best way it can. i.e. if the receiving device is a mobile device, it will render the controls using mobile control instead typical desktop control.
  • the invention will use the established XMPP connection. To specify the changes that have been made by a given device, it will use XML like the definition of the layout.
  • the receiver will handle this command and using the methods for grabbing and setting the content of the application's user control, will set the state giving user the chance of watching the changes the remote user did in real-time.
  • the Real-Time hub will send over the established connection a special command which will be processed by the real-time hub of the other end.
  • the first step (1 ) is to build the request which is going to be used as the starting point of the communication.
  • the Real-Time hub will firstly request the user to select the elements of the application which are selected to be controlled remotely (4). Achieving this can be done using the different methods which were explained before. After grabbing the components, the hub will form the device- neutral language describing the interface. Secondly, the Real-Time Hub will also grab the contents of those controls (5), using the same mechanism it used to grab the components themselves. Having all data needed to negotiate a connection, they will be sent through the network.
  • the connection will be established by receiving the remote peer the request sent by the host.
  • the remote client can either accept or deny the remote control request.
  • This connection will be negotiated using the Communications server as a gateway. Once both data are ready, they are serialized into a standard XMPP information-query message and sent to the remote peer.
  • the remote Real-Time hub will render the controls which were sent by the first user (6), and will track the changes the user will do in those controls (7). Every time a change is made in some of the controls sent, the Real-Time Hub will send those changes over the network, and the other end will show those changes (8).
  • a typical use case of the prototype can be described in the following steps: 1. Some user is writing a document regarding a reservation he is doing in a hotel for the holidays with his friend. He needs the personal details of one of his friends.
  • the user starts the Real-Time Hub in the Windows PC, and selects the friend he wants to give remote control of one of his applications.
  • the Real-Time Hub will give the user the option to choose the controls he wants to share.
  • the Real-Time Hub will grab the contents of the word document using a specific plugin installed in Word application and will build the XML which is going to be sent to the mobile device.
  • the Real-Time Hub of the mobile device receive the request of remote- control and the user accepts the request.
  • the Real-Time Hub of the mobile device will render the document with its contents using the limitations of the device itself.
  • the Real-Time Hub of the PC Device will receive the changes made in the mobile device and using the plugin installed will set those changes in the real application.
  • Every device can adapt them to his capacities.
  • a mobile device can render those controls in a very different way a desktop client should do.
  • remote control scenarios require the remote device to have strong processing power, and not always the control scheme is adapted to the mobile device itself.
  • the invention can use existing public services to transfer the elements and content in the remote-control phase.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

A method and a system to remotely share an application between electronic devices. In the method of the invention, said electronic devices have equal or different capabilites and is characterised in that it comprises: - selecting, a host user of a given electronic device, a remote user of a different electronic device to remotely share said application; - selecting, said host user, at least one user interface component of said application to share with said remote user; - grabbing, said given electronic device, current status of said at least one user interface component and content of said application; - adapting, said given electronic device, said current status and said content to a device neutral language representation; and - sending said current status and said content to said different electronic device by means of a communication link. The system of the invention is arranged to implement the method of the invention.

Description

A method and a system to remotely share an application between electronic devices
Field of the art
The present invention generally relates, in a first aspect, to a method to remotely share an application between electronic devices, said electronic devices having equal or different capabilites, and more particularly to method that allows giving remote control of only a part of the desktop by sending to the remote user elements of the application chosen by the host user, said elements being described in a neutral language and being displayed to the remote user in an adapted way according to the characteristics of the remote electronic device.
A second aspect of the invention relates to a system arranged to implement the method of the first aspect.
Prior State of the Art
Nowadays collaborative work is a trend. In some cases users may need that other user (or several users) performs some kind of interaction (enter data, validate a form, etc.) in order to finish a determined task. Usually this interaction has to be carried out instantaneously or in real time.
One of the most common interactions is when one user shares a part of his desktop to another user. For instance, a user could be using his PC to book a flight for himself and a friend, and at some step, personal information of his friend could be required. Then he could share with his friend that part of the input form, so he could fill in from his mobile.
Although most typical use cases are performed in a PC, they aren't limited exclusively to it and can be extended to other devices such as tablets or smartphones. In case of heterogeneous devices, such as PCs or mobile phones, some kind of content adaptation or transcoding (i.e. resizing an input box) will be needed.
Other of the main factors of this kind of interactions is "control". For instance, the user who is sharing ("the sharer") should have the full control of what he is sharing and until when this sharing will be active. The same concept could be applied for real time collaboration in a document or presentation. In this case the user could just predefine the page or paragraph which he would like to give control to his friend. Currently there are several alternatives which are used to partially solve the situations depicted above.
One of the typical approaches is the usage of remote desktop access technologies, such as the RDP protocol [1 ]. Remote Desktop Protocol (RDP) is a proprietary protocol developed by Microsoft, which concerns providing a user with a graphical interface to another computer. This approach has a couple of important drawbacks. In first place it requires that the devices have any kind of permanent connection with the servers. If the connection breaks, then the remote access is stalled or closed until it is reconnected again. On the other hand, it works with a full desktop environment, not just with a set of Ul elements. Additionally, the user experience in small screen devices is very poor, since the virtual desktop needs to be rescaled to the smaller screen of a smartphone.
There are also other virtualization solutions at application level, such as Citrix XenApp [2]: it is an on-demand application delivery solution that enables any Windows application to be virtualized, centralized, and managed in the datacenter and instantly delivered as a service to users anywhere on any device. However this virtualization- based alternative has the same problems as the previous one.
As an overview of the current methods depicted above, it is possible to conclude that: all of them work at full desktop or application level, a permanent network connection is required and the user experience regarding the access of some of the application from small screen devices can be very poor.
Other proposals:
- "COMPUTER REMOTE CONTROL SYSTEM". US005949412A. Robert A. Huntsman, Boise, Id. This invention generally relates to the field of remotely controlling a first computer from a second computer, where the first computer has a graphical user interface (GUI) and the second computer may have a dissimilar user interface and a dissimilar operating environment from the first computer. In this case, the way of transmitting the remote desktop or application is by means of image screenshots. Therefore no layout adaptation or resizing is performed to be optimized for different computing devices such as smartphones. Moreover, the remote control is limited to the level of the whole desktop or applications, not to single Ul elements (i.e. a set of input boxes) as it is defined in the current invention.
- "REMOTE DESKTOP ACCESS" US 2009 0235342 Todd R. Manion et al. Describes the Remote Desktop Protocol implemented by Microsoft. In this case, a virtual operating system experience on the second computer reproduces virtually the same as the real operating system experience on the first computer. It reproduces the whole remote desktop, not just a subset of input forms that could be selected by the user.
- "MISSION CONTROL, VNC OVER XMPP". Mission control is a software application for Linux Operating system which uses XMPP to transport the VNC data of a typical desktop sharing session. As the rest of the methods of desktop sharing it uses images or an image-based protocol to transfer the elements of the desktop. Indeed, it is only available to PC devices and it cannot be used on mobile devices or via web sites.
- "X WINDOW SYSTEM PROTOCOL, VERSION 1 1 ". This protocol can be used to share the elements of the desktop between different peers. Indeed this is the protocol used in Unix systems to layout the graphical desktop. Although it doesn't use an image based protocol, it doesn't allow sharing the control between different users and it doesn't support content adaptation for each device.
- "TECHNIQUES FOR PROVIDING ACCESSIBILITY OPTIONS IN REMOTE
TERMINAL SESSIONS" US7676549B2. Brendan McKeon, Makarand V Patwardhan, Nadim Y.Abdo. This invention from Microsoft tries to improve the accessibility of the Remote Desktop Protocol by sending single elements of the Ul instead of sending the whole desktop. Those elements are sent through the network using bitmap images. Once the recipient has the images, it can apply some transformations in order to improve the Accessibility of the components which were sent. However this method is restricted to Windows Platform, as it sends data which is tightly related to Windows technologies. As the information sent is composed by images and low-level Microsoft Windows desktop data, the receiver should be able to render natively those Windows components, making impossible for other devices, like mobile phones, to use this method.
The main problems with the existing solutions are:
- Most of current solutions work at level of the whole desktop or a specific application, not at a single Ul element (i.e. a set of input boxes).
- The technical approach used by the existing solutions is mainly based in the optimized transmission of images from the remote PC through a network. However this approach presents several drawbacks. First of all, the amount of data that will be sent through the network is high, and the user experience can be substantially affected by the network's latency. Second, it is not possible to alter the layout of the Ul elements that are being transferred from the remote PC, so the possibilities for adaptation of the content to devices with smaller screen (smartphone) are very limited. This leads to that user experience in small screen devices is very poor, since the virtual desktop needs to be rescaled to the smaller screen of a smartphone.
- Some virtualization-based solutions require that the devices have any kind of permanent connection with the servers. If the connection breaks, then the remote access is stalled or closed until it is reconnected again.
Description of the Invention
It is necessary to offer an alternative to the state of the art which covers the gaps found therein, particularly related to the lack of proposals which really allow working with a single User Interface element of an application and not with the whole desktop.
To that end, the present invention provides, in a first aspect a method to remotely share an application between electronic devices, said electronic devices having equal or different capabilites.
On contrary to the known proposals, the method of the invention, in a characteristic manner it further comprises:
- selecting, a host user of a given electronic device, a remote user of a different electronic device to remotely share said application;
- selecting, said host user, at least one user interface component of said application to share with said remote user;
- grabbing, said given electronic device, current status of said at least one user interface component and content of said application;
- adapting, said given electronic device, said current status and said content to a device neutral language representation; and
- sending said current status and said content to said different electronic device by means of a communication link
Other embodiments of the method of the first aspect of the invention are described according to appended claims 2 to 10, and in a subsequent section related to the detailed description of several embodiments.
A second aspect of the present invention concerns to a system to remotely share an application between electronic devices, said electronic devices having equal or different capabilites. In the system of the second aspect of the invention, on contrary to the known systems mentioned in the prior state of the art section, and in a characteristic manner it comprises using a real-time hub module in said electronic devices in order to:
- grab current status of at least one user interface component and content of said application, said at least one user interface component selected by a host user;
- adapt said current status and said content to a device neutral language representation; and
- send said current status and said content to a different electronic device of a remote user selected by said host user by means of a communication link.
The system of the second aspect of the invention is adapted to implement the method of the first aspect.
Other embodiments of the system of the second aspect of the invention are described according to appended claims 12 to 14, and in a subsequent section related to the detailed description of several embodiments.
Brief Description of the Drawings
The previous and other advantages and features will be more fully understood from the following detailed description of embodiments, with reference to the attached drawings which must be considered in an illustrative and non-limiting manner, in which:
Figure 1 shows the overall view of the invention architecture according to an embodiment of the present invention.
Figure 2 shows three different approaches to grab the components and the contents of the controls selected by the host user, according to an embodiment of the present invention.
Figure 3 shows the XML schema followed by the XMPP message structure that transports the description and the state of the host control, according to an embodiment of the present invention.
Figure 4 shows a concrete example of the XMPP message structure that transports the description and the state of the host control, according to an embodiment of the present invention.
Figure 5 shows the traffic that flows in real-time through XMPP network when connection between the host user and the remote user has been established, according to an embodiment of the present invention. Figure 6 shows the XML schema followed by the XMPP message structure that transports the modifications introduced by the remote user, according to an embodiment of the present invention.
Figure 7 shows a concrete example of the XMPP message structure that transports the modifications introduced by the remote user, according to an embodiment of the present invention.
Figure 8 shows the general steps of the method, according to an embodiment of the present invention.
Detailed Description of Several Embodiments
The invention consists of a system that allows a real-time interaction of a remote user with selected components of an application placed in the computer of a given user. In addition to the real-time interaction, the invention also contemplate the remote user could have a different device with different capabilities. If the destination device is different from the source, the contents should be adapted to be displayed in the low-capable device.
The host will select the user interface components (i.e. A web form input box or a board of a chess game) he wants to share with his guest, and then the state of those components will be grabbed, adapted and send through the network, allowing his guest to interact with those components in real time.
The overall view of the invention architecture was shown in Figure 1.
The different devices (101 ) which a user may have are the main participants in the idea proposed in this invention. These devices can be any kind of computing elements such as desktop computers, mobile phones, tablets, netbooks, etc. The user of a given device might want to give remote control of one of his applications (103) (or a part of it) for a short period of time to a user using a different device.
To achieve this purpose, in each device it is necessary to have a key element of this invention which is called Real-Time Hub (104). The Real-Time hub is able to do five main tasks:
- Grabbing the status of the application the user wants to give remote control to a remote user. This implies two kinds of procedures, firstly let the user select the elements of the Ul and lastly, grab the current status of those elements.
- Packing the description of the elements and the contents of those elements to a device-neutral language representation. - Establish and maintaining the connection between different Real-Time Hubs. Through this connection, data needed to achieve the real-time remote-control will be sent.
- Once data is received from the remote peer, this real-time hub will modify the content of the controls involved in the session.
- If the Real-Time hub plays the role of the receiving one, it will display the controls and their content in the best way the device can show them. Allowing the user to modify their contents.
When the user decides to start the real-time communication, he sends a request to the remote device using the real-time hub (104). To establish the communication, the hub will use a standard XMPP mechanism to inform the remote user the intention of the local user to start a remote control-session.
During the connection, the communications server (102) acts as a connection point between the peers, it will coordinate the first steps in the communication setup and will act as a contact server. Thanks to this functionality, the users will be able to log on to the service and manage the contacts they will be using the service with. Moreover, the communication server will route the XMPP messages from the host client to the remote client.
The main tasks of the communications server can be described as follows: - Maintain the contacts list of each user. At the beginning of the connection process the user will select one of his contact to communicate with
- Route the messages from one user to another. Both users will use this server to send messages to each other.
As the invention is built over a standard protocol with the standard message as the carrier, the communication server can be any standard XMPP server with the appropriate extensions installed.
The connection process can be described in five steps:
- The real-time hub will ask the communications server the contacts of the current user
- The real-time hub will present the user the set of contacts which are available to start a remote-control session
- The user will select the contact who he wants to share the contents of his applications
- The real-time hub will try to establish the connection with the remote real-time hub by using the communications server as a helper. - The remote real-time hub, will inform the remote user the will of the local user. If the communication is accepted by the remote user, the hub will begin by asking the user for the controls that are going to be remote controlled. The user will select them using his mouse and selecting them from the applications themselves. Once the controls are selected, the Real-Time Hub will try to grab the components and the contents of them using three different approaches, as shown in Figure 2:
- Using Application Specific Plugins. This method is used when the application allows some kind of extensibility, giving the possibility of writing custom plugins to extend its functionality. Not only is needed the application to be extended with custom plugins, but also the application should allow the plugin to grab and set its state. Examples of this method would be a Microsoft Office Plugin, or a Firefox browser plugin. By using this method, the real-time hub will communicate with the plugin and ask for the description of the selected elements and their current contents. The Real- Time hub will also communicate with the plugin to set the content of the controls when the remote user changes its contents.
- Using Operating system mechanism to look into and modify the content of running applications. In the case of applications which don't support any extensible mechanism, the process of grabbing its state will be done by these operating system mechanisms. In the case of Windows Platform, this will be done using the operating system messages which are supported within the OS. In example, we can grab and set the content of a "notepad" application without needing the notepad to provide anything, just using standard Windows messages. In this case, the real-time hub will grab the description of the elements and their contents by itself. It will also change their content by itself.
- Using the Communications hub Application Programming Interface. The solution implies a set of procedure calls implemented in the Device-dependant communications hub which will be used by the third-party application. These third party applications will implement the methods required by the provided interface and will able to send the state of the application themselves, and without requiring extra work by anybody else. In this case, the Real-Time Hub will communicate with the extension point provided by the application developer directly. This scenario is similar to the first one, but developer of the application will be the one who will implement the communication mechanism. The real-time hub will use this method to grab the description of the elements and to grab and set their contents. Once the real-time hub has obtained the description of the elements and their content, it will pack the data in a standard container ready to be sent through the network.
This way of connection between the Real-Time Hub and the Applications is used not only to grab the controls and their contents but also to set the content later when the remote user changes their values.
The invention uses the XMPP message structure as a container to transport the description and the state of the host control. The application state data is where the host application serializes its state, and it is the data which uses the receiver to layout the component and allow the remote user to interact with the control which was sent.
In order to describe the elements, the base language chosen to describe the layout is XML following the schema shown Figure 3.
The content of the controls are placed inside the description XML. This data is packed inside a standard XMPP IQ message and then sent to the remote client.
The purpose of this device-neutral language is to allow each real-time hub to adapt the contents to their best way to show them to the user. For example, if the remote real-time hub is installed in a mobile device, it will render those controls a different way that if the remote-real time hub is installed in a PC computer.
This is a clear advantage over other methods of transferring desktop elements, given that those methods use images to send the desktop content.
Once the content is received, the remote client's Real-Time hub will render the content using the best way it can. i.e. if the receiving device is a mobile device, it will render the controls using mobile control instead typical desktop control.
With the connection in an established state both peer will interchange commands about the things they are doing with the interface. All this traffic flows in real-time through the XMPP network.
To interchange the contents of the controls, the invention will use the established XMPP connection. To specify the changes that have been made by a given device, it will use XML like the definition of the layout.
The receiver will handle this command and using the methods for grabbing and setting the content of the application's user control, will set the state giving user the chance of watching the changes the remote user did in real-time.
Once one of the users of the process decides to end the remote-control session, he will use the real-time hub to end the connection. In this step, the Real-Time hub will send over the established connection a special command which will be processed by the real-time hub of the other end.
As a general overview, the steps of the protocol can be described as shown in Figure 7:
- The first step (1 ) is to build the request which is going to be used as the starting point of the communication. For that purpose, the Real-Time hub will firstly request the user to select the elements of the application which are selected to be controlled remotely (4). Achieving this can be done using the different methods which were explained before. After grabbing the components, the hub will form the device- neutral language describing the interface. Secondly, the Real-Time Hub will also grab the contents of those controls (5), using the same mechanism it used to grab the components themselves. Having all data needed to negotiate a connection, they will be sent through the network.
- In the second step (2), the connection will be established by receiving the remote peer the request sent by the host. In this step, the remote client can either accept or deny the remote control request. This connection will be negotiated using the Communications server as a gateway. Once both data are ready, they are serialized into a standard XMPP information-query message and sent to the remote peer.
- Once the communication is established (3), the environment is ready for interchanging the data between the peers. In this step, the remote Real-Time hub will render the controls which were sent by the first user (6), and will track the changes the user will do in those controls (7). Every time a change is made in some of the controls sent, the Real-Time Hub will send those changes over the network, and the other end will show those changes (8).
- Lastly when one of the peers decides to end the communication, it will use the
Real-Time Hub to end the session (9).
A prototype has been built using the following elements:
- A computer with Windows OS. The applications which are going to be remote controlled are Microsoft Word, Mozilla Firefox and a Chess game.
- A mobile device with Android OS.
- A Linux Server with a XMPP server installed.
A typical use case of the prototype can be described in the following steps: 1. Some user is writing a document regarding a reservation he is doing in a hotel for the holidays with his friend. He needs the personal details of one of his friends.
2. The user starts the Real-Time Hub in the Windows PC, and selects the friend he wants to give remote control of one of his applications.
3. The Real-Time Hub will give the user the option to choose the controls he wants to share.
4. The User will select the Word document he was writing.
5. The Real-Time Hub will grab the contents of the word document using a specific plugin installed in Word application and will build the XML which is going to be sent to the mobile device.
6. The Real-Time Hub of the mobile device receive the request of remote- control and the user accepts the request.
7. The Real-Time Hub of the mobile device will render the document with its contents using the limitations of the device itself.
8. When the user of the mobile device changes the contents of the document, fill in his personal details, the contents are sent over the network.
9. The Real-Time Hub of the PC Device will receive the changes made in the mobile device and using the plugin installed will set those changes in the real application.
10. Finally, the mobile device user will close the selection after making the changes he wanted to make.
From the point of view of the final user, he will see how the contents of the elements he selected at the beginning are changed by a remote user using a different device which, for the time being, wasn't able to manage.
Advantages of the invention
- Using this invention, one user can give remote control only a small part of his desktop. Typical scenarios of remote control imply giving a remote user full control of the entire user desktop.
- As the elements are described in a neutral language, the resources needed to transfer the elements and the contents of those elements are very small in terms of size and processing power.
- Thanks to this method of sending the shared controls, every device can adapt them to his capacities. A mobile device can render those controls in a very different way a desktop client should do. Usually remote control scenarios require the remote device to have strong processing power, and not always the control scheme is adapted to the mobile device itself.
- The method using to remote control the user interface elements are very low in terms of size, so each device should transfer a very small amount of network data during the remote control. Usually, remote desktop protocols transfer a lot of data as all the desktop is sent over the network.
- As the technologies used in server side are standard, it doesn't need any special deployment. The invention can use existing public services to transfer the elements and content in the remote-control phase.
A person skilled in the art could introduce changes and modifications in the embodiments described without departing from the scope of the invention as it is defined in the attached claims.
ACRONYMS
ADSL Asymmetric Digital Subscriber Line
GSM Global System for Mobile Communications
HTTP Hypertext Transfer Protocol
HTTPS Hypertext Transfer Protocol Secure
IMEI International Mobile Equipment Identity
IMSI International Mobile Subscriber Identity
MBB Mobile Broadband
MSISDN Mobile Subscriber ISDN Number
SI M Subscriber Identity Module
SMS Short Message Service
Ul User Interface
UMTS Universal Mobile Telecommunication System
REFERENCES
[1] http://en.wikipedia.org/wiki/Remote_Desktop_Protocol [2] http://en.wikipedia.org/wiki/Citrix_XenApp
[3] http://www.google.com/patents/about?id=7dAoAAAAEBAJ, "Computer remote control system" [4] http://www.google.com/patents/about?id=rdPIAAAAEBAJ, "REMOTE DESKTOP ACCESS"
[5] http://gnomejournal.org/article/84/telepathy-empathy-and-mission-control-5-in- gnome-228, Mission Control using Telepathy
[6] http://tools.ietf.org/search/rfc1013. X Window System RFC

Claims

Claims
1. - A method to remotely share an application between electronic devices, said electronic devices having equal or different capabilites, characterised in that it comprises:
- selecting, a host user of a given electronic device, a remote user of a different electronic device to remotely share said application;
- selecting, said host user, at least one user interface component of said application to share with said remote user;
- grabbing, said given electronic device, current status of said at least one user interface component and content of said application;
- adapting, said given electronic device, said current status and said content to a device neutral language representation; and
- sending said current status and said content to said different electronic device by means of a communication link.
2. - A method as per claim 1 , comprising displaying, said different electronic device, at least part of said application according to said current status and said content received, said at least part of said application displayed being adapted to characteristics of said different electronic device.
3.- A method as per claim 2, comprising sending back in real time modifications performed by said remote user to said application to said host user by means of said communication link.
4. - A method as per claim 3, comprising sending, said host user, a request to said remote user by means of an Extensible Messaging and Presence Protocol, or XMPP, mechanism in order to establish a connection through said communication link.
5. - A method as per claim 4, comprising performing said grabbing by using specific plugins of said application, wherein said specific plugins extend functionality of said application and said plugins grab and set said status and/or said content.
6. - A method as per claim 4, comprising performing said grabbing by using operating system mechanisms applied to said application, said operating system mechanisms being able to grab and set said status and/or said content.
7. - A method as per claim 4, comprising performing said grabbing by using a communications hub Application Programming Interface, or API, implemented by a developer of said application, said communications hub API being able to grab and set said status and/or content.
8. - A method as per claim 5, 6 or 7, comprising packing said status and content in a standard container, said standard container being defined by a XMPP message structure
9. - A method as per claim 8, comprising using Extensible Markup Language, or XML, in said standard container in order to perform said adaptation of said current status and said content, said standar container containing the following fields described in XML:
- control type: it defines the type of said user interface component;
- interaction type: it defines actions to do to said user interface component; and - content: it defines said content of said application.
10. - A method as per claim 9, comprising defining said modifications performed by said remote user to said application in a XMPP message structure using XML.
1 1 . - A system to remotely share an application between electronic devices, said electronic devices having equal or different capabilites, characterised in that it comprises using a real-time hub module in said electronic devices in order to:
- grab current status of at least one user interface component of said application and content of said application, said at least one user interface component selected by a host user;
- adapt said current status and said content to a device neutral language representation; and
- send said current status and said content to a different electronic device of a remote user selected by said host user by means of a communication link.
12. - A system as per claim 1 1 , wherein said communication link is established between real-time hub modules of said electronic devices using a standard XMPP mechanism.
13. - A system as per claim 12, wherein a communications server connected to said real-time hub modules coordinates part of said communication link establishment and routes messages between said electronic devices.
14. - A system as per claim 13, wherein said real-time hub module is in charge of sending back in real time modifications performed by said remote user to said application to said host user by means of said communication link.
15. - A system as per claim 13, wherein said messages are expressed in a XMMP standard messages and said neutral language is XML.
PCT/EP2012/068916 2011-09-29 2012-09-26 A method and a system to remotely share an application between electronic devices WO2013045470A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
ES201131578 2011-09-29
ESP201131578 2011-09-29

Publications (1)

Publication Number Publication Date
WO2013045470A1 true WO2013045470A1 (en) 2013-04-04

Family

ID=46980934

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2012/068916 WO2013045470A1 (en) 2011-09-29 2012-09-26 A method and a system to remotely share an application between electronic devices

Country Status (1)

Country Link
WO (1) WO2013045470A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5949412A (en) 1996-01-22 1999-09-07 Extended Systems, Inc. Computer remote control system
US20090235342A1 (en) 2008-03-13 2009-09-17 Microsoft Corporation Remote desktop access
US7676549B2 (en) 2005-05-27 2010-03-09 Microsoft Corporation Techniques for providing accessibility options in remote terminal sessions

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5949412A (en) 1996-01-22 1999-09-07 Extended Systems, Inc. Computer remote control system
US7676549B2 (en) 2005-05-27 2010-03-09 Microsoft Corporation Techniques for providing accessibility options in remote terminal sessions
US20090235342A1 (en) 2008-03-13 2009-09-17 Microsoft Corporation Remote desktop access

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
IVAN MARSIC: "Real-time collaboration in heterogeneous computing environments", PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON INFORMATION TECHNOLOGY: CODING AND COMPUTING (ITCC 2000)., 27 March 2000 (2000-03-27) - 29 March 2000 (2000-03-29), pages 222 - 227, XP002690635, ISBN: 0-7695-0540-6, Retrieved from the Internet <URL:http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=844214&contentType=Conference+Publications&sortType%3Dasc_p_Sequence%26filter%3DAND(p_IS_Number%3A18266)%26rowsPerPage%3D50> [retrieved on 20130117] *

Similar Documents

Publication Publication Date Title
US11700137B2 (en) Collaborative access to virtual desktops
EP2854376B1 (en) Transmission method, device and system for media stream
CN105378659B (en) Client device is set to be able to access that the method and system of remote desktop
EP2649502B1 (en) Methods and systems for a remote desktop session utilizing a http handler and a remote desktop client common interface
EP2649500B1 (en) Methods and systems for accessing and controlling a remote desktop of a remote machine in real time by a web browser at a client device via http api utilizing a transcoding server
EP2533493B1 (en) Proximity session mobility extension
US10999405B2 (en) Method for processing access requests and web browser
EP2514139B1 (en) System and method of multi-media conferencing between universal plug and play (upnp) enabled telephony devices and wireless area network (wan) devices
CN103581265A (en) Remote access method and system
US20140129695A1 (en) Connected devices
US10817137B2 (en) Method and system for communication between web browsers, using a unified communication environment
US20210273981A1 (en) Method for establishing a communication connection which is suitable for transmitting media streams between a first rtc client and a second rtc client
CN105847332A (en) Desktop virtualization method, client device and server-side device
CN111935276A (en) Remote host access method, device and equipment
US20070133511A1 (en) Composite services delivery utilizing lightweight messaging
WO2009114429A2 (en) Policies for session types
WO2019119280A1 (en) Service processing method, cloud server, and terminal device
CN104780591B (en) Business interoperability implementation method, device, system and relevant device between terminal device
KR20170111609A (en) Image service system and method thereof
WO2013045470A1 (en) A method and a system to remotely share an application between electronic devices
US20150120805A1 (en) Bi-directional Channel-based Progress Indicator
CN112565458B (en) Platform remote control method and device, storage medium and electronic equipment
CN113573002A (en) Method and system for realizing remote control in video call
EP4047833A1 (en) Load balancing system, load balancing method, and carrier means
CN114679486B (en) Communication interaction method and system, application terminal, client terminal and transfer server terminal

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12769074

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12769074

Country of ref document: EP

Kind code of ref document: A1