GB2547634A

GB2547634A - Software system for displaying a remote desktop

Info

Publication number: GB2547634A
Application number: GB1601966.3A
Authority: GB
Inventors: B Hill Michael
Original assignee: Dolphin Computer Access Ltd
Current assignee: Dolphin Computer Access Ltd
Priority date: 2016-02-03
Filing date: 2016-02-03
Publication date: 2017-08-30
Also published as: GB201601966D0

Abstract

A software system is described for displaying on a monitor of a local computer at least a part of a desktop of the local computer and at least a part of a desktop image of a remote computer and for permitting the display of the desktop image of the remote computer to be modified on the monitor of the local computer dependent on meta data transmitted from the remote computer in addition to the image of the desktop of the remote computer. The software system may include a part installed on the remote computer and a part installed on the local computer. The system may also include a screen magnifier. The screen magnifier may be capable of magnifying the local desktop image and the remote desktop image at the same or different magnifications. The part of the software system installed on the remote computer may include means for transmitting the remote desktop image together with messages incorporating the metadata relating to the remote desktop image. The part of the software system installed on the remote computer may include means for disabling font-aliasing or animations insofar as it pertains to the image transmitted to the local computer.

Description

SOFTWARE SYSTEM FOR DISPLAYING A REMOTE DESKTOP

This invention relates to a software system for displaying and manipulating a remote desktop image on a local computer.

In this specification the following terms are defined as follows:

Local desktop - the image (a pixel bitmap) of a user’s desktop on the user’s local computer. The local desktop is normally the same size as the resolution of the user’s monitor, but it can be a different size. Where a user’s computer is provided with more than one monitor, the local desktop is usually the resolution of the union of the monitor resolutions, but it can be a different size.

Local computer - the computer directly in use by the user and which, in practice, is required to display a magnified image for the user to view.

Remote computer - a computer that is connected to an interactive whiteboard, (classroom) display panel or projector. It is the desktop of this computer the user wishes to view remotely.

Remote desktop - the image (pixel bitmap) of the desktop on the remote computer. The remote desktop may not be the same size as the local desktop or as the monitor(s) of the local computer.

Local monitor-the physical display device(s) forming part of the local computer, for example a built-in panel display or one or more external devices. That is, the local monitor may be made up of one or more physical display devices. The resolution (number of pixels) of the local monitor is generally fixed and may be the same size as the local desktop, but this is not essential.

Image - a picture, usually represented by a two-dimensional array of pixels (often known as a bitmap).

Viewport - An area on a monitor that is used to show an image. The image may include portions of the local or remote desktop, or camera image and may be magnified, reduced, orthe image may be processed in otherways. The viewport has attributes such as size, position on a monitor, an image to display, data that indicates which portion of the image to display, and data that indicates how to display it.

Viewport position - the location on the viewport image that is shown in the viewport. If the viewport is showing a magnified portion of the image, the position indicates where in the source image the portion to show is located.

Meta data - data that is relevant, but in addition, to the remote desktop image that is sent from the remote computer to the local computer.

There is a need to provide a visually impaired user on the screen of their own local computer with a magnified version of at least part of the screen of a remote computer and to be able to track what is happening on the screen of the remote computer in real time, for example to follow a mouse pointer, document caret or other display changes. This is coupled with a need on the part of the visually impaired user to be able to use their own local computer at the same time, for example by splitting the screen display between magnified portions of the screen display transmitted from the remote computer and the display generated on the screen of the local computer, or by quickly switching between the screen display transmitted from the remote computer and that generated on the local computer. The visually impaired user additionally does not want the remote desktop transmitted from the remote computer to occupy any portion of the desktop generated on the local computer so that they can use their normal applications software.

The visually impaired user may be, for example, a student wishing to view a teacher’s interactive whiteboard in a classroom, the whiteboard being viewed via a large screen monitor or a projector attached to a computer, or an employee wishing to view, for example, a PowerPoint presentation displayed on a large screen monitor or by way of a projector attached to a computer.

Screen magnifiers are well known and typically work by displaying a magnified portion of a user’s desktop on the monitor of the user’s computer. The magnified portion is often known as a viewport. Because it is magnified, only a portion of the original desktop is viewable at any one time and the user is therefore provided with various methods to control which portion is displayed, including the use of a mouse, keyboard or touchscreen. The screen magnifier also typically follows what the user is doing on the local desktop, for example moving the magnified area to follow a flashing beam cursor as the user types. Some screen magnifiers have additional visual enhancements to improve the quality of the enlarged image. Some of these enhancements employ additional data from the local computer to redraw some magnified screen elements. However, until the present invention there has been no facility to reproduce these functions in a remote desktop in a viewport on a local computer.

Existing methods for providing on a user’s computer a display of part of a whiteboard or the like include the user employing a camera having a zoom facility for capturing an image of a part of the whiteboard or the like. The camera is typically used in conjunction with software on the user’s computer which displays the image in an application window on part of the screen of the user’s computer. Alternatively, the camera may have a separate video out connector so the user can employ a monitor switching solution to choose whether to view their computer desktop or the camera image on their computer monitor. As a further alternative, virtual network computing applications allow a user to connect remotely to another computer and view its desktop in an application window.

Cameras can have difficulties with lighting (such as interference from fluorescent lighting) and camera wobble. If the whiteboard is an LCD screen, cameras can suffer serious interference from the whiteboard’s own raster refresh. A camera cannot react to events on the whiteboard, so the user has to manually move the magnified image to locate items of interest. With a camera, the image quality depends on the focussing ability of the camera and the relative position of the camera to the whiteboard. Line of sight is required and this can be difficult in a classroom environment. Monitor switching is neither very portable, nor very practical. A virtual network application can provide a better image than a camera because it can transmit an exact pixel representation to the local computer. However, the remote desktop is typically displayed in a window on the desktop of the local computer, making it difficult to view side-by-side with other applications. Avirtual network application also only provides an image; there is no additional information that could be used to help the user track what is going on or to render an enhanced image.

It is therefore an object of the present invention to provide a software system which overcomes, or at least ameliorates the above problems.

According to the present invention there is provided a software system for displaying on a monitor of a local computer at least a part of a desktop of the local computer and at least a part of a desktop image of a remote computer and for permitting the display of the desktop image of the remote computer to be modified on the monitor of the local computer dependent on meta data transmitted from the remote computer in addition to the image of the desktop of the remote computer.

The software system may include a part installed on the remote computer and a part installed on the local computer.

The part of the software system installed on the local computer may include a screen magnifier. The screen magnifier may be capable of magnifying the local desktop image and the remote desktop image at the same or different magnifications.

The part of the software system installed on the remote computer may include means for transmitting the remote desktop image, or part thereof (such as changes thereto), together with messages incorporating the meta data relating to the remote desktop image.

That part of the software system installed on the remote computer may include means for disabling font anti-aliasing and or for disabling animations relating to the remote desktop image, or part thereof, at least insofar as it pertains to the image transmitted to the local computer.

The meta data transmitted from the remote computer to the local computer may include mouse location and/or focus location.

That part of the software system installed on the local computer may move the viewport position of the remote desktop image in accordance with the location of the mouse pointer and/or the focus position of the remote desktop image. Alternatively or additionally, that part of the software system installed on the local computer may allow a user of the local computer to move the location of the viewport of the remote desktop image.

That part of the software system installed on the local computer may include means for reading text under the mouse pointer on the remote desktop image and/or for using font information provided as part of the meta data to re-render the magnified remote desktop image.

For a better understanding of the present invention and to show more clearly how it may be carried into effect reference will now be made, byway of example, to the accompanying figure which shows a monitor of a local computer employing part of one embodiment of a software system according to the present invention.

The software system according to the present invention includes software installed on a local computer to manipulate a local desktop image and to receive and manipulate a remote desktop image. There is shown in the figure a monitor of a local computer in which the screen image is split by that part of the software system according to the present invention provided on the local computer. As shown in the figure, the screen image is split vertically, with the left-hand side of the screen showing a magnified portion of the local desktop (indicated by the letter A) and the right-hand side of the screen showing a magnified portion of a remote desktop (indicated by the letter B). It will be noted the screen may be split in different ways, for example horizontally, and the two portions may have different proportions. In any event, in order to display both the local and remote desktops simultaneously, two viewports are created, one for the local desktop and the other for the remote desktop. Each desktop retains its original size (number of pixels), but of course only part of each desktop can be seen at any one time. Each viewport may have a position, magnification level and other image enhancements independently of the other as will be explained in more detail hereinafter.

The use of two viewports side-by-side also permits the overall image to be split in a configuration employing two monitors in such a way that one of the viewports is displayed on one of the monitors and the adjacent viewport is displayed on the other of the monitors. In this way, one of the monitors displays one of the local and remote desktops and the other of the monitors displays the other of the local and remote desktops.

The software system according to the present invention also includes software installed on the remote computer to transmit the remote desktop image over a network or other suitable communication method to the local computer. Although communications between the local and remote computers will typically be over a wired or wireless network, alternative communications systems may be used if desired, such as a direct connection between the local and remote computers. The precise means of communication is not important, but should ideally have a relatively low latency to allow for real time updates. It is not essential that the remote computer is running the same operating system as the local computer.

There are several open documented standards available for transmitting a desktop image from one computer to another over a network. One of these is VNC which uses the RFB (Remote Frame Buffer) protocol, which is an adaptation of RFC6143. Such standards are well known and documented and the VNC system is also available as open source and supported on multiple operating systems.

The remote desktop also transmits to the local computer additional technical meta data relating to the transmitted image in synchronism with the remote desktop image itself, which meta data is used by the screen magnifier software on the local computer to enhance user experience of the remote desktop image at the local computer. The RFB protocol mentioned hereinbefore also allows for custom messages to be embedded in the transmission of the desktop image and such custom messages can be used to transmit the additional meta data.

On start-up, the part of the software system according to the present invention installed on the remote computer first makes any required changes to the configuration of the remote computer to improve the user experience on the local computer. These include disabling font anti-aliasing and/or disabling animations. The software then opens an incoming network connection and waits for a connection from the local computer. The connection is typically based on one of the known remote desktop protocols. Once a connection is established, the remote computer begins sending remote desktop image updates to the local computer using the remote desktop protocol. The remote computer also uses the relevant APIs (typically the operating system’s accessibility APIs) to capture the mouse and focus location information which forms part of the meta data. The software uses a RFB (Remote Frame Buffer) custom encoded message to transmit this information along with the remote desktop image to the local computer.

The mouse position is used to display a mouse cursor in the remote desktop viewport on the local computer so the user can see where the mouse cursor is on the desktop of the remote computer. The mouse position may also be used for other purposes, for example to automatically move the viewport position to track the mouse when the mouse position changes and/or to display highlighting to make the position of the mouse easier to see.

The focus location may include one or more of the position of the I-beam (flashing) cursor, the line of text it is on, control or button rectangle, focus rectangle, window rectangle and application window rectangle. Focus position information can be used to automatically move the viewport position to track the focus and to further enhance the location with visual highlighting.

The remote computer can support more than one local computer at the same time. This is accomplished by transmitting the same information to each local computer after every change in the remote desktop image or in the focus. The RFB protocol mentioned above supports multiple local computers (clients) and this system is well understood by the skilled person.

Thus, when the desktop image changes or the focus moves, the remote computer transmits image and metadata updates to the or each local computer.

On shutdown, the local computer closes the RFB channel and disconnects from the remote computer. Once the or each local computer has disconnected, the remote computer can idle until a local computer connects.

That part of the software system on the local computer connects the local computer with the remote computer. For example, settings within the software may specify the network location of the remote computer. Once connection is established, as explained above that part of the software system on the remote computer transmits the remote desktop image to the local computer together with the meta data in real time.

In practice, the magnification system forming part of the software system according to the present invention installed on the local computer needs to be able to replace the local desktop image with content other than a magnified local desktop, while maintaining a copy of the local desktop image and allowing applications to run normally. This is accomplished by capturing the original local desktop image, performing an image enlargement of a portion of it and displaying the result in the local viewport. The method of doing this may vary between different operating systems, but is well documented and generally involves redirecting all normal drawing output to an off-screen bitmap and then drawing a magnified image to the output (the local viewport).

In Microsoft Windows XP this is accomplished by using a display driver in conjunction with the Microsoft Driver Chain Manager. The chaining display driver redirects standard windows output to an off-screen surface by changing the destination surface for all drawing calls. It then draws custom content directly to the monitor under control of an application.

In Windows 8, magnifiers typically hook the IDXGISwapChain::Present() call in the desktop window manager process. When PresentQ is called, the content of the back buffer is replaced with an image which can be anything the magnifier software wants to show. The magnifier can use the Desktop Duplication API to capture the original desktop image. Some care has to be taken to avoid hooking the call to Present() that is used by the Desktop Duplication API, but this is not a problem for the skilled person and can be done, for example, by checking whether the swapchain belongs to a real output or not by calling GetContainingOutputQ on the swapchain object.

Initially, the magnification software on the local computer only magnifies the local desktop. Once connection to the remote computer has been established using the RFB protocol, for example following an option in the software on the local computer, a key press or other input method, the magnification viewports on the local computer can be reorganised (for example by the locally installed part of the software system) such that the viewport showing the local desktop uses a first portion of the monitor and the viewport showing the remote desktop uses a second portion of the monitor, as shown in the figure. The local magnification software can then draw a magnified portion of the local desktop and a magnified portion of the remote desktop in the respective viewport.

The magnification software typically follows the following repetitive process: 1. Get any changes to the real remote or local desktop image. 2. Calculate a new viewport location depending on any focus changes. 3. Generate a magnified image from a portion of the remote or local desktop. 4. Display the magnified remote or local desktop image on the monitor.

Magnification typically includes a mode where the image is not magnified, or may even be shrunk. This provides the user with an overview of the entire remote or local desktop or image that is being magnified.

Known screen readers and magnifiers may implement an off-screen model. This is a database of screen elements generated by monitoring the API calls made by applications that draw to the screen. Methods of hooking API calls and creating an off-screen model are well known, although they vary across operating systems. The off-screen model can provide additional contextual information about text on the screen. This allows that part of the software system according to the present invention on the local computer to provide additional support for the user, such as the ability to read text under the mouse pointer even on the remote desktop or the use of font information to enhance the magnified image by re-rendering the image when it has been magnified (see, for example, U.S. Patent No. 8280193). A conventional off-screen model consists of one or more methods of monitoring application rendering. In the case of Microsoft Windows, this can be by hooking operating system API calls, either in user mode or in the display driver. Various methods for this are well known. A second component is a database which is updated using information from the hooks to maintain an accurate representation of the text on the screen. Typically, the application hooks are in a different process to the off-screen model, so a degree of inter-process communication is required as is well understood. In the software system according to the present invention, the application hooks may be implemented that part of the software system according to the present invention installed on the remote computer. The information they generate is then sent to the local computer as part of the meta data stream to the local computer, where the off-screen model is updated.

That part of the software system according to the present invention on the local computer can then use the off-screen model information in the same way it might use a local off-screen model, for example to read screen content or to draw larger text or a larger image when it magnifies.

For this to work, all relevant information required to update the off-screen model needs to be packaged and transmitted over the communication channel to the local computer. Most API calls do not require complex arguments and can easily be packed into messages for transmission. The only complication may be font information if detailed font data is required. In this case, either information about each font needs to be transmitted, orfonts need to be synchronised between the remote and local computers. In either case, the means for accomplishing this is well known.

As explained above, the additional meta data can be used by the part of the software system according to the present invention and installed on the local computer to:

Automatically follow a mouse pointer and focus position by moving the viewport position.

Apply visual effects, such as colour enhancements and/or cross hairs, to the focus position.

Re-render the text with a higher resolution version to eliminate jagged pixel boundaries from the magnified image.

The meta data may include further information. For example, where the software system of the present invention is employed in the context of a teacher and student, the teacher may wish to restrict the ability of the student to see specific areas of the remote desktop, such as a certain application or a certain window.

The locally-installed software keeps track of the focus position and mouse position for each viewport separately. When local changes to the focus occur the locally-installed software may move the viewport showing the local desktop appropriately to track the focus. When focus change messages arrive from the remote computer as part of the meta data, the locally-installed software moves the viewport location that is showing the remote desktop.

The basic processing loop for the magnification is as follows: 1. Get any changes to the local desktop image. 2. Calculate a new viewport location depending on any focus changes for the local desktop. 3. Generate a magnified image from a portion of the local desktop. 4. Get any changes to the remote desktop image. 5. Calculate a new viewport location depending on any focus changes for the remote desktop. 6. Generate a magnified image from a portion of the remote desktop. 7. Combine both magnified images into a single image and display it on the monitor.

Parts of the processing loop can be skipped if there is no change to a desktop or no focus change. It is also possible for the local desktop and remote desktop processing to be run asynchronously in two separate threads provided the method of combining the result to the monitor is synchronised.

The locally-installed software also allows the user to manually move the location of the viewport around both the local desktop and the remote desktop, to change the magnification level and to change other attributes (such as the colours and/or image smoothing method). If the local computer is a touch screen device, updating of a respective viewport can be accomplished by the user touching the viewport and employing standard gestures, such as drag to move the viewport location and pinching to change the magnification. In the case of a mouse, the locally-installed software can allow the user to switch the mouse pointer to either viewport by using a keypress or other user input device. While in the remote desktop viewport the user is not able to interact with the content, but it is possible to move the mouse pointer around the desktop in a similar manner to the local desktop.

The user may wish to switch quickly between viewing the local desktop and the remote desktop. To accomplish this, the locally-installed software is able to rearrange the viewports and not to display one of them, while maintaining the connection to the remote computer and keeping the updates to the remote desktop image in memory. This enables the user to switch quickly back to viewing the remote desktop without waiting for a connection to be made to the remote computer.

The part of the software system on the local computer additionally allows the user of the local computer to independently control the attributes of both the local and remote desktop viewports. For example, magnification levels and other visual enhancements may be controlled independently. Attributes may be controlled in a number of ways including the use of key presses to choose which viewport to control, and then the use of additional keys to change the settings for that viewport. Alternatively or additionally, touch screen input may be employed. For example the local software may determine which viewport the user wishes to control depending on which viewport is initially touched on the local monitor.

The user is able to choose whether to allow remote computer activity as represented by the meta data to control the viewport displaying the remote desktop. If the user chooses not to allow this, the user is able on the local computer to move around the remote desktop independently of any focus changes made on the remote computer.

Where the remote desktop image is larger than the viewport on the local computer, then not all of the remote desktop will fit within the viewport even at 1x magnification. In this case, the user can still move the viewport position relative to the image of the remote desktop to view different portions of the remote desktop image. Additionally in this case, the user may zoom the image to less than 1x magnification in order to show the entire remote desktop image in the viewport. This may affect readability of the image depending on the magnification and the image reduction algorithm employed, but is useful to allow the user to orient themselves relative to the remote desktop as a whole.

Independent on the content of the meta data, the part of the software system on the local computer may additionally allow the user to: switch between multiple remote computers; and/or freeze the remote desktop image, that is to prevent updates to the remote desktop image taking effect, to allow the user to navigate around a static image of the remote desktop; and/or capture and save the remote desktop image to a local file.

In addition to the above-described meta data, that part of the software system on the remote computer may allow the user of that computer to: take over the remote desktop image displayed on the local computer, for example to allow a teacher to direct a pupil to a specific part of the remote desktop image on the local computer irrespective of the settings on the local computer; and/or restrict synthesised mouse/keyboard input on the local computer to a portion of the remote desktop image, for example to permit a teacher to allow a pupil to interact with a document or text entry field in a controlled manner; and/or restrict synthesised mouse/keyboard input on the local computer to a specific application or set of applications, for example to permit a teacher to allow a pupil interact with a specific application in a controlled manner.

The software system according to the present invention may incorporate a chat facility to allow a user of the local computer to send a message to a user of the remote computer and vice versa. The chat facility allows a teacher to ask a question of a pupil or to check whether the pupil using the local computer requires assistance without disturbing other pupils. The teacher may enable or disable messages from the pupil.

The following references may be helpful to the skilled person: RFB protocol - https://tools.ietf.org/html/rfc6143 VNC protocol - https://en.wikipedia.org/wiki/RFB_protocol NeatX protocol - open source remote X Windows -https://en.wikipedia.org/wiki/Neatx Microsoft magnification engine - U.S. Patent No. 8542253 Microsoft magnification API - https://msdn.microsoft.com/en-us/library/windows/ desktop/ms692162(v=vs.85).aspx Windows Driver Chain manager - https://msdn.microsoft.com/en-us/library/ms971320.aspx Desktop Duplication API - https://msdn.microsoft.com/en-gb/library/windows/ desktop/hh404487(v=vs.85).aspx

Off-screen model - General information on assistive technology including off screen models - Book: Assistive Technology for Visually Impaired and Blind People - Marion Hersh, Michael A Johnson. ISBN 978-1-84628-867-8 Method and apparatus for magnifying computer screen display - U.S. Patent No. 8280193

Claims

1. A software system for displaying on a monitor of a local computer at least a part of a desktop of the local computer and at least a part of a desktop image of a remote computer and for permitting the display of the desktop image of the remote computer to be modified on the monitor of the local computer dependent on meta data transmitted from the remote computer in addition to the image of the desktop of the remote computer.

2. A software system as claimed in claim 1 and including a part installed on the remote computer and a part installed on the local computer.

3. A software system as claimed in claim 1 or 2, wherein the part of the software system installed on the local computer includes a screen magnifier.

4. A software system as claimed in claim 3, wherein the screen magnifier is capable of magnifying the local desktop image and the remote desktop image at the same or different magnifications.

5. A software system as claimed in any preceding claim, wherein the part of the software system installed on the remote computer includes means for transmitting the remote desktop image, or part thereof (such as changes thereto), together with messages incorporating the meta data relating to the remote desktop image.

6. A software system as claimed in any preceding claim, wherein that part of the software system installed on the remote computer includes means for disabling font anti-aliasing and/or for disabling animations relating to the remote desktop image, or part thereof, at least insofar as it pertains to the image transmitted to the local computer.

7. A software system as claimed in any preceding claim, wherein the meta data transm itted from the remote computer to the local computer includes mouse location and/or focus location.

8. A software system as claimed in any preceding claim, wherein that part of the software system installed on the local computer moves the viewport position of the remote desktop image in accordance with the location of the mouse pointer and/or the focus position of the remote desktop image.

9. A software system as claimed in any preceding claim, wherein that part of the software system installed on the local computer allows a user of the local computer to move the location of the viewport of the remote desktop image.

10. A software system as claimed in any preceding claim, wherein that part of the software system installed on the local computer includes means for reading text under the mouse pointer on the remote desktop image and/or for using font information provided as part of the meta data to re-render the magnified remote desktop image.