WO2016142713A1

WO2016142713A1 - System for connecting external displays to a mobile device

Info

Publication number: WO2016142713A1
Application number: PCT/GB2016/050656
Authority: WO
Inventors: Peter SKAN
Original assignee: Displaylink (Uk) Limited
Priority date: 2015-03-10
Filing date: 2016-03-10
Publication date: 2016-09-15
Also published as: GB2536253B; GB201504065D0; GB2536253A

Abstract

Disclosed herein is a method of providing display output from a computing device adapted to run an operating system and applications to a display interface device over a general- purpose data transmission channel using a display application. The display application runs on the computing device and initiates a user-level service, which is arranged to provide display output to the general-purpose data transmission channel during operation of one or more other applications. The application instructs the operating system to create a display buffer entity configured to receive display output from software running on the device. The user-level service then retrieves display output from the display buffer entity and outputs display data based on the retrieved display output for transmission to the display interface device over the general-purpose data transmission channel.

Description

System for Connecting External Displays to a Mobile Device

As mobile computing devices such as smartphones, tablet computers and laptops become more common and more powerful and hot-desking in a work environment also becomes more common, users increasingly wish to use their mobile devices as their main computing devices. This is advantageous as it provides increased portability and convenience. However, especially in the case of small mobile devices such as smartphones and tablets, it is not always practical to use them as primary computing devices due to the fact that their inbuilt screens are very small. It is therefore desirable to connect them to external display devices such as monitors, televisions and even projectors. However, with many mobile devices this is not possible due to limitations in their operating systems and available connectors; most modern mobile devices are only provided with USB connectors.

To allow connection of an external display to a mobile device by USB would ordinarily require modifications to the operating system of the mobile device. This requires the co-operation of the device manufacturer and the operating system designers (and possibly mobile telephony service providers), and may in fact require the co-operation of multiple device manufacturers and designers in order to ensure that there is a uniform connection interface provided for external displays.

The present invention seeks to alleviate some of these problems.

Accordingly, in a first aspect of the invention, there is provided a method of providing display output from a computing device adapted to run an operating system and applications to a display interface device over a general-purpose data transmission channel using a display application, the method comprising: initiating, by the display application running on the computing device a user-level service, the service arranged to provide display output to the general-purpose data transmission channel during operation of one or more other applications; instructing the operating system to create a display buffer entity configured to receive display output from software running on the device; and retrieving, by the user-level service, display output from the display buffer entity and outputting display data based on the retrieved display output for transmission to the display interface device over the general-purpose data transmission channel.

The term "application" preferably refers to a software program (or collection of software programs/modules) that is separate from the operating system. Typically such an application is a user-level program, e.g. it runs with user-level privileges rather than as part of the operating system.

The display application preferably includes the user-level service. Installation and/or operation of the display application and service preferably do not require modification of the operating system and/or do not require installation of any operating system level software components.

The term "service" preferably refers to a software entity that can remain operational to provide functionality during operation of other applications (for example, when the application that launched the service is dormant or not foregrounded), e.g. by way of suitable multitasking facilities provided by the operating system. As for user-level applications, a user-level service is typically a service that runs with user-level privileges rather than as part of the operating system.

The display buffer entity preferably represents an image data buffer (e.g. a frame buffer), and may comprise an image data buffer or a reference to an image data buffer. The display buffer entity is preferably arranged to receive display output from one or more other applications running on the device. The term "other applications" as used herein preferably refers to applications other than the display application itself (and may include e.g. the Android home screen application and other parts of the user front end). Alternatively or additionally, the display buffer entity may be arranged to receive display output from the display application itself.

The operating system is preferably an Android operating system, in which case the display buffer entity preferably comprises an Android Virtual Display.

Preferably, the instructing step comprises configuring the operating system to mirror a display output generated for a display of the device (e.g. an integrated display) to the display buffer entity or Virtual Display. The term "mirror" preferably refers to rendering and/or compositing the same source image data (or at least a portion thereof) to both the device display and the display buffer entity (Virtual Display), in effect creating a copy of at least a portion of display output rendered for the device display on the Virtual Display. In one example, the entire screen display may be mirrored. Alternatively, only a selected portion of display output may be mirrored to the display buffer entity (e.g. one or more selected screen regions or windows).

The instructing step preferably comprises configuring a compositor module of the operating system (e.g. Android SurfaceFlinger or the like) to output display data to the display buffer entity or Virtual Display. The instructing step is preferably performed by the display application, and may be performed by the service or by another part of the application.

The service is preferably configured as a foreground service. The term "foreground service" preferably refers to a service configured to run in a manner such that it is regarded as important by the operating system (e.g. a high-priority service), e.g. compared to other (background) services, and/or is not subject to termination by the operating system (or at least has a higher priority to be kept running), for example when the system is low on memory. In a particular example the service is an Android foreground service. Preferably, the method comprises, at the user-level service, receiving a notification from the operating system that display output is available from the display buffer entity (e.g. that a rendered display frame is available), and retrieving the display output from the display buffer entity in response to the notification.

The method may comprise processing the retrieved display output prior to transmission via the general purpose transmission medium. The processing may include one or more of: encoding the display output, compressing the display output, and encrypting the display output. Accordingly, the display application may comprise an encoding module for performing the processing step.

The display output or processed display output is preferably provided to a channel interface module arranged to interface with the general-purpose data transmission channel.

The display application preferably includes the channel interface module.

The method preferably comprises obtaining an access permission for access to the general-purpose data transmission channel, and preferably providing the permission to the channel interface module to enable the channel interface module to send data to the general-purpose data transmission channel. For example, the operating system may request the necessary access permission(s) from the user when the display interface device is connected to the data transmission channel or at another time.

The channel interface module preferably provides the display data to an operating system interface component adapted to interface with the general-purpose data transmission channel; the operating system interface component may comprises a device driver for the general-purpose data transmission channel.

For efficiency, one or both of the encoding module and the channel interface module of the display application may comprise device native code (e.g., code native to the computing device). The user-level service may comprise virtual machine code for execution by a virtual machine of the computing device (e.g. Java or Dalvik virtual machine code in the case of Android).

Preferably, the method comprises starting the display application and/or user- level service in response to the display interface device being connected to the computing device via the general-purpose data transmission channel. For example, the application may be registered with the operating system such that the operating system launches the application automatically on connection of the display interface device, and the application then starts the user-level service. The display application and/or user-level service are preferably terminated in response to disconnection of the display interface device (or of the last such connected device).

Preferably, the application or user-level service initiates the instructing and receiving steps in response to detecting connection of a display to the display interface device. Additionally, an encoder instance and/or channel interface instance may be created in response to detecting connection of the display, and may be specific to the display.

The method may comprise providing display output to a plurality of external displays connected to one or more display interface devices. The method may then further comprise creating a respective display buffer entity for each connected display, and optionally creating respective encoder and/or channel interface instances for each connected display.

The general-purpose data transmission channel is preferably arranged for the transmission of data including data other than display data (or more generally, data other than audio/visual data). The general-purpose data transmission channel is preferably not designed for, or limited to, transmission of any specific types of data (e.g. image data), but rather provides a data transport mechanism independent of the type of data.

The general-purpose data transmission channel preferably comprises a Universal Serial Bus, USB. The USB channel preferably uses USB protocols operating over a USB physical connection. However, other general-purpose transmission media may be used, such as serial or parallel peripheral buses, wired or wireless network connections, etc., and any suitable combination of transmission protocol and physical medium may be utilised.

Preferably, the method comprises receiving the display data at the display interface device, optionally processing the data (e.g. to decompress/decrypt etc.), and generating a display signal based on the received display data, and outputting the display signal to a display. The display interface device is preferably connected to the display by a special- purpose channel for transmission of a display signal, for example an HDMI, DVI, DisplayPort or VGA connection.

The computing device is preferably a mobile device, such as a smartphone or tablet computer.

In a further aspect of the invention, there is provided a computing device adapted to run an operating system and applications and having means for providing display output to an external display using a general-purpose data transmission channel, the device comprising: means for running a display application including a user-level display service, the display service arranged to provide display output to the general-purpose data transmission channel during operation of one or more other applications; means for instructing the operating system to create a display buffer entity configured to receive display output from software running on the computing device; and means for receiving, by the user-level service, display output from the display buffer entity and outputting display data based on the received display output to the general-purpose data transmission channel for transmission to the external display.

The computing device preferably further comprises means for performing any method as set out above. The invention also provides a system comprising a computing device as set out above; and a display interface device connected to, and adapted to receive display data from, the computing device over a general-purpose data transmission channel and to output a display signal to at least one display based on the received display data. The system may include one or more displays connected to the display interface device for receiving the display signal.

The invention further provides a computer-readable medium comprising software code adapted, when executed on a data processing apparatus, to perform any method as set out herein.

In a further aspect of the invention, there is provided an application, or a computer- readable medium comprising software code for an application, adapted, when executed on a data processing device, to provide display output to a display interface device over a general-purpose data transmission channel, the application comprising: software code for instructing the operating system to create a display buffer entity configured to receive display output from software running on the device; and software code implementing a user- level display service, the display service arranged to provide display output to the general- purpose data transmission channel during operation of one or more other applications, the user-level display service arranged to: retrieve display output from the display buffer entity; and output display data based on the retrieved display output for transmission to the display interface device over the general-purpose data transmission channel. The code for the instructing step may be part of the user-level display service.

The application preferably further comprises one or more of: software code implementing an encoding module arranged to process the retrieved display output, the processing preferably including one or more of encoding, compressing and encrypting the display output; and software code implementing a channel interface module arranged to receive the display output or processed display output, and output the display output or the processed display output to the general-purpose data transmission channel, preferably via an operating system driver. As set out previously, the channel interface module and/or encoding module may comprise device-native code, and the display service may comprise virtual machine code.

The computer readable medium or application preferably further comprises software code for performing any method as set out above.

The user-level service may also be provided as an independent aspect of the invention. Thus, in a further aspect of the invention, there is provided a computer-readable medium comprising software code, adapted, when executed on a data processing device, to implement a user-level service for providing display output to a display interface device over a general-purpose data transmission channel, the software code for the user-level service arranged to perform a method including the steps of: instructing the operating system to create a display buffer entity configured to receive display output from software running on the device; retrieving display output from the display buffer entity; and outputting display data based on the retrieved display output for transmission to the display interface device over the general-purpose data transmission channel. That method and a device having means for performing that method may be provided as further independent aspects of the invention, and the method may include any of the further features as set out above with regard to the first aspect of the invention.

Where reference is made herein to means for performing various actions or functions, those means may be in the form of one or more processors (and associated memory) configured to execute software code for performing the actions or functions.

More generally, the invention also provides a computer program and a computer program product for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein, and a computer readable medium having stored thereon a program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein. The invention also provides a signal embodying a computer program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein, a method of transmitting such a signal, and a computer product having an operating system which supports a computer program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein.

The invention extends to methods and/or apparatus substantially as herein described with reference to the accompanying drawings. Any feature in one aspect of the invention may be applied to other aspects of the invention, in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa. Furthermore, features implemented in hardware may generally be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly.

Preferred features of the present invention will now be described, purely by way of example, with reference to the accompanying drawings, in which:-

Figure 1 shows a simplified schematic of a display system including a mobile device connected to an external display;

Figure 2 shows a schematic diagram of communications between a display application and an operating system of the mobile device;

Figures 3A and 3B illustrate operation of the display application;

Figure 4 illustrates a software architecture of the system;

Figure 5 illustrates a method for providing direct access to a USB channel; and Figure 6 illustrates a hardware architecture of the system.

Overview

Embodiments of the invention provide an application for use on an Android device that transmits display output to an external display via a general-purpose transmission channel, in preferred embodiments a Universal Serial Bus (USB) connection. The display application typically (though not necessarily) mirrors the output presented on an integrated display of the mobile device. The display output is sent to the external display via an adapter connected to the mobile device over USB. The adapter in turn is connected to one or more external displays and provides a display signal to those display(s).

An embodiment is illustrated in overview in Figure 1. Figure 1 shows an Android smartphone 11 (which includes a device-internal display typically with touch input functionality) running the display application 10 connected to an adapter in the form of a Display Interface Device 12 using a general-purpose connection, which in this embodiment is a USB cable 14. The Display Interface Device 12 is in turn connected to a display 13 via a special-purpose display connection 15, such as an HDMI (High-Definition Multimedia Interface), DVI (Digital Visual Interface), DisplayPort or VGA (Video Graphics Array) connection.

Due to the limited bandwidth of USB, the display application 10 installed on the smartphone 11 is preferably arranged to compress the display data prior to transmitting it and the Display Interface Device 12 is arranged to decompress it as well as converting it from the general-purpose USB format to a display-specific format for transmission down the connection 15 between the Display Interface Device 12 and the display 13. Other processing may also be performed at the display application and/or Display Interface Device as described in more detail below.

The Display Interface Device 12 may be in the form of an external adapter connectable via cables to both the mobile device and the display, or may be directly connected to one of the devices (e.g. as a dongle). In one example the Display Interface Device is in the form of a dock to which the mobile device can be connected. The Display Interface Device may also be integrated into the mobile device or display. Thus connections 14 and/or 15 may take the form of pluggable cables (USB, HDMI etc.) or may comprise direct socket-to-socket connections (e.g. in a dock or dongle) or internal connection pathways in the case of an integrated solution.

The display application 10 is preferably a user-level application. Thus, it is separate from the operating system (and preferably does not require system privileges to run). Note that Android distinguishes between user applications and system applications, though both may be considered user-level applications in the above sense ("system applications" in Android are not part of the operating system per se but are stored in a special system application area or partition and generally cannot be uninstalled without root access; they may also have access to certain permissions not available to user applications).

In one embodiment, the display application is implemented as an Android system application in the above sense. Alternatively, it could be implemented as a user application, in which case it may advantageously be signed with the manufacturer's platform key. Implementation as a system application or application signed with a platform key may in some Android implementations allow the application to gain the necessary privileges to capture display data and send it to the display. Alternatively, the application may gain the necessary privileges by requesting and receiving the user's permission to capture display output. In either case, the application will then operate according to the process described below.

The display application preferably includes all the relevant functionality described herein and can be added to the mobile device (e.g. during manufacture or original software installation), or can be subsequently installed, without requiring any changes to be made to the operating system software. For example, the display application may be distributed via an app store such as Google Play.

Figure 2 illustrates the communication between the display application (on the left) and the operating system of the mobile device (on the right). The application includes a Service 202, which is preferably an Android Service implemented in Java; an Encoder 204, which may be in the form of a user-space driver implemented in C++; and native code 206 providing an interface for accessing the USB. Among other components, the operating system includes an Android USB Manager 208 and an Android Display Manager 210, which provides a method for creating virtual displays including Virtual Display 212. The operating system also includes USB driver 214.

Operation

Figures 3A and 3B illustrate the typical operation of the Display Application from connection of the Display Interface Device with an attached display (Step S1) to disconnection (Step S19), with reference to Figure 2 (references in square parentheses are to the data flows and components depicted in Figure 2).

At Step S1 , the Display Interface Device with a connected display is attached to the mobile device by plugging in a USB cable. The mobile device is aware of the connection due to the conventional handshaking signals associated with such a connection and the Android USB manager prompts the user for permission for the application to handle the device. When the user gives permission, the Android operating system will automatically launch the Display Application (Step S2), which up until this point was not running. At Step S3, the Application automatically starts the Service [202]. The Service then runs continuously to gather display data and forward this over the USB link to the display as described in more detail below. Preferably, the service is run as an Android foreground service. Foreground services are typically used in applications such as music players. The use of a foreground service is advantageous because it means that the display will not be disrupted when the user launches a new application or changes to a different application. It also means that (unlike Android background services) the Service is unlikely to be terminated by the operating system unless the operating system is very short of memory, which will improve the experience for the user.

This example assumes that the Display Interface Device is attached while the mobile device is operational. If the mobile device is booted with the Display Interface Device attached, the Application will launch automatically upon booting and the process will continue as herein described.

Upon launch, the Application is supplied with an attachment notification (an "Intent") by the Android USB Manager and it forwards this to the Service (Step S4). This indicates that the application has been granted the necessary permissions to access the USB Display Interface Device. From this point on, in this embodiment, the same Service would handle all further attachments or detachments of Display Interface Devices to or from the USB. Each time a subsequent Display Interface Device is attached an Intent will be forwarded to the existing Service, and no further instance of the Service is launched.

The Service then opens access to the Display Interface Device (Step S5) by making a call [221] to the Android USB Manager. At Step S6, the Android USB Manager returns to the Service for use by the Service an open file descriptor and appropriate device descriptors [222]. Appropriate permissions may also be returned at this point. This returns control to the application.

At Step S7, the Service passes the file and device descriptors on to the USB Access Interface [206]. It then creates an instance of the Encoder (Step S8) for the newly attached Display Interface Device and/or display and determines the resolution of the attached display. At Step S9, the Service requests [223] a Virtual Display from the Android Display Manager (through the relevant Android API). The operating system then creates the Virtual Display (Step S10).

A Virtual Display is an Android entity to which the Android graphics subsystem (including in particular the Android compositor, known as "SurfaceFlinger") can render display output. The Virtual Display represents a buffer to which graphical data is rendered; the buffer may be managed by an Android BufferQueue instance.

The new Virtual Display is preferably public, which means that any application can use it (e.g. to create windows on it). This means that the system is not limited to specific predetermined applications but can be used by any application on the mobile device.

If the Display Interface Device was not connected to a display when it was attached to the mobile device, the Display Application and Service will be launched according to the process described above. However, the Service will preferably not create the Encoder or request a Virtual Display until a display is connected to the Display Interface Device.

When applications send display data to the Virtual Display, the data is composited into a frame. This process is under the control of the operating system, as shown in Figure 2. Specifically, the compositing is typically performed by SurfaceFlinger, the Android Compositor. SurfaceFlinger defines a set of layers which are rendered into a composited frame, where the content of the layers is obtained from input surfaces rendered by applications or the operating system.

In a preferred embodiment, the compositor, i.e. in this example Android SurfaceFlinger, is configured to mirror the mobile device's internal display output to the created Virtual Display (and hence to the external display), for example allowing users to view normal mobile device applications on a larger display. To achieve this, the compositor is configured to use the same layers - and hence the same input surfaces - for compositing the Virtual Display as are used to composite the internal (physical) display. By composing both screens separately from the same input surfaces, scaling can be optimised to the respective displays. An alternative approach is to use the composed output for the internal display as input for the Virtual Display composition (e.g. in circumstances where scaling issues are less important).

Instead of mirroring, applications may place specific content on the Virtual Display e.g. by creating an Android Presentation on the Virtual Display. Furthermore, the system preferably supports both mirrored and non-mirrored modes, and can switch between those modes. For example, if mirroring is enabled, and specific (non-mirrored) content is placed on the Virtual Display by an application, then mirroring ceases and the Virtual Display then shows that unique content instead. In the non-mirrored use case, different output can be provided on the device-internal screen and on the external display.

For example, the external display may be used to show content for an application (e.g. video or game output), while the internal device screen (which typically has touch input functionality) can be used as a control interface for the application, displaying control buttons, media transport controls, menus or the like.

Regardless of whether the output frame for the Virtual Display is rendered in mirrored or non-mirrored mode, when the frame is complete, a notification [224] to this effect is sent by the operating system to the Service (Step S11) and the Service then accesses the contents of the frame buffer that contains the display data on the Virtual Display. It then passes [225] this to the Encoder that it created at Step S8 (Step S12). In this embodiment, the Encoder compresses the frame (Step S13). Any suitable image compression techniques may be applied. For example, since device displays often contain portions that do not change for periods of time, the Encoder may encode/compress changed portions of a frame but skip unchanged portions. The Encoder may alternatively or additionally perform other processing or encoding tasks, for example changing the image data representation (e.g. to change pixel encoding format or rescale the image). The Encoder then passes [226] the encoded/compressed data to the Native USB Access interface code [206], which in turn (Step S14) sends it to the kernel USB Device Driver using the USB file system [227]. The USB file system and the kernel USB Device Driver are both provided by the operating system.

The USB Device Driver communicates with the Display Interface Device in order to transmit the compressed display data (Step S15) to the Display Interface Device and the Display Interface Device in turn handles communication with the display.

At Step S16, the compressed data is received by the Display Interface Device. It is then decompressed and otherwise manipulated as appropriate. This may include scaling, rotation, division for multiple displays or any other appropriate processing. It will also be converted from the general-purpose USB format in which it was received to the display- specific signal format that the display is configured to use. For example, after processing, the display data may be converted to a digital signal suitable for transmission over HDMI or DVI, or may be converted by a digital-to-analogue converter to an analogue video signal, e.g. for transmission over an analogue VGA, component or composite display connection. The signal is then sent on to the display over the relevant physical connection. Steps S11- S16 are repeated for as long as the display remains connected and in use.

If a display is disconnected from an attached Display Interface Device, the Encoder and Virtual Display associated with that display are destroyed, but the Service preferably continues to run even if the associated Display Interface Device has no displays connected to it, in order to detect subsequent re-connections.

When the Display Interface Device is disconnected at Step S17, the Android USB Manager sends another Intent to the Service. Assuming that only one Display Interface Device was attached, the Service is then automatically terminated (Step S18), as is the Application (Step S19). If there is more than one Display Interface Device connected then the Service and Application will continue running until the last one is disconnected.

Software and hardware architecture

Figure 4 provides an overview of the software architecture of an embodiment. As shown, the display application 10 includes the Display Service 202, Encoder 204 and native code USB Access Interface 206. The Display Service communicates with the Android operating system 402 to set up the USB connection, create a Virtual Display and receive display rendered to the Virtual Display by the Android graphics stack (including SurfaceFlinger). After processing by Encoder 204, the USB Access Interface 206 transmits processed display data to the USB stack / device driver in the Linux kernel 404 of the Android operating system. The USB driver outputs the data over the physical USB connection to the Display Interface Device 12 for transmission to display 13. The application 10 is thus preferably self-contained and makes use only of standard Android functionality, and can accordingly be installed and run on the mobile device without modification of the operating system (OS).

Figure 5 illustrates the process of enabling direct access to the system USB driver

214, showing in particular how the Display Application interacts with the Android USB Manager in order to allow native code to access USB devices.

As illustrated, the Display Service 202 interacts with the Android USB manager 510 to open the USB device (flow 501) and the USB manager returns open file and device descriptors 512 (flow 502). The Display Service 202 then passes the descriptors 512 to native code USB Access Interface 206 (flow 503). The USB Access Interface can then use the descriptors to access the system USB driver 214 directly. The Display Application 10 preferably additionally obtains the relevant permissions to access the USB connection as described previously.

The disclosed methods may be implemented in software, hardware, firmware, combination thereof, or any other method that may be appropriate to a specific embodiment. A simplified hardware architecture for an exemplary embodiment is depicted in Figure 6.

As shown in Figure 6, the mobile device 11 includes one or more processors 606 for running the operating system and applications, along with volatile / random access memory 602 and persistent storage 604 (e.g. FLASH memory) for storing OS and application software and data. Display data may be rendered to buffers in memory 602 by applications and/or the OS. Display output is supplied to an internal display 608. Wireless interfaces 612 such as Wi-Fi, Bluetooth and GSM interfaces provide regular communications functionalities of the device.

USB interface 610 is connected to a corresponding USB interface 614 in Display

Interface Device 12 via USB connection 14 (e.g. a USB cable). The Display Interface Device includes one or more processors 616 for receiving, decompressing and otherwise processing display data, persistent storage 618 (e.g. ROM) for storing device firmware/software, and volatile / random access memory 620 for storing data, including received display data. Processed display data is output to display interface 622 for transmission to a display via a suitable display connection such as HDMI, DisplayPort or VGA. Devices 11 and 12 may typically include other components (e.g. I/O interfaces etc.) which for the purpose of clarity have not been shown.

Further features

Some further features and advantages of embodiments of the invention are discussed below.

Embodiments of the invention provide a method for connecting a monitor or display to a mobile device via USB, comprising an application that:

• interacts with the operating system of the mobile device but does not require any changes to the operating system;

• is capable of handling display data even when the user switches to use a different application;

• gains the necessary permission from the user to use its native code to access the display;

· accesses the operating system in order to gain access to display data; and

• transfers the said display data to the display.

Use of an application is beneficial since applications are easier to distribute and deploy without requiring modification to other system components. However, applications do not necessarily run constantly on a mobile device once they have been launched. For example, on smartphones and tablet computers, it is likely that only one application will be active at any one time while other applications are dormant. Embodiments overcome this by use of a user-level foreground service to handle display data.

The application is preferably implemented in such a way that it does not require changes to the operating system because this will facilitate deployment of the application and also make it unnecessary for manufacturers to change their own implementations. This is in turn advantageous as it does not require co-ordination across multiple device manufacturers.

Embodiments preferably use USB as this is a simple connection topology that takes advantage of technology already available on many mobile devices.

For improved performance, embodiments use device-native code (e.g. written in C++) to access the USB device. However, an Android application may not normally have the required privilege to do this. Preferred embodiments overcome this by obtaining the necessary permission together with file and device descriptors from the Android USB Manager and then using native code for all subsequent USB operations. Preferably, the application is arranged such that it is able to handle display data in a secure manner, e.g. by encryption. This is beneficial because it will allow the system to be used for the display of protected content. Display data is preferably compressed before being transmitted to the display to reduce bandwidth requirements. The application is preferably launched automatically upon the connection of a Display Interface Device (or alternatively on connection of a display), and therefore does not run constantly. This will result in reduced power use, which is especially beneficial on a mobile device as such devices are normally battery-powered. In particular, if a mobile device is not connected to an external display it is more likely to be running on battery power, thus making it more important that the application is not running unnecessarily under these circumstances. This approach can also reduce unnecessary use of processing power and other resources on the mobile device. If the application is launched automatically rather than being launched deliberately by the user, this is also more convenient for the user.

The system is preferably able to support multiple displays. This is beneficial because it will allow for situations where the user wishes to use multiple displays, for example on a desk in an office. The use of multiple displays is a proven method of improving productivity and if the user is using the mobile device in place of a conventional computer it is beneficial to allow for such improvements.

A single Service preferably handles all the displays. This is beneficial because it reduces the overhead caused by the launch and operation of multiple instances of the same Service.

Embodiments also provide a system comprising:

• A Mobile Device running an application as set out above;

• One or more displays capable of connecting to the mobile device and displaying data transmitted by the mobile device;

• An intermediate device, which may be a docking station or adapter (the "Display Interface Device" as mentioned above), that is connected to the mobile Device and also to one or more displays.

Preferably the Display Interface Device is arranged so as to decompress compressed data and perform any other necessary processing such as decryption, scaling, rotation etc. before transmitting the display data to the display. Decompression is performed if the data has been sent in a compressed form, and it is beneficial for additional processing to be carried out by a Display Interface Device rather than on the mobile device in order to reduce the load on the mobile device. Preferably, the Display Interface Device is further arranged so as to convert data to a display-specific format such as VGA, HDMI or DisplayPort. The Display Interface Device may be integral to the display.

Disclosed embodiments enable use of USB connected displays with Android by means of a user-level service and make it possible for a user to mirror their Android device's screen (or selected portions thereof) on another display even when alternating between different applications and the home screen.

Disclosed embodiments can be used to display content on one or more external screens, such as when showing photos or a presentation or for demonstrations. The external display(s) may be larger than the device-internal screen for ease of working and/or viewing by several people.

Embodiments allow support for USB displays to be added to mobile devices merely by installing an application program. The application runs the display driver code as a foreground service to ensure that it continues to run even when the user switches to a different application. The fact that no modifications to the OS are necessary makes it possible to distribute the service as an "app", which is much easier to deploy.

It will be understood that the present invention has been described above purely by way of example, and modification of detail can be made within the scope of the invention.

For example, the disclosed embodiments involve a mobile device such as a smartphone running an Android operating system, connected to a Display Interface Device via a USB cable. The Display Interface Device is then connected to a display, which in this example is an LCD monitor, using a display-specific cable (e.g. VGA, DVI or HDMI).

It will be appreciated that this embodiment is by way of example only, and embodiments can include any computing device (mobile or non-mobile, e.g. a mobile telephone, tablet, video streaming device or games console etc.), running Android or any other operating system (e.g. iOS) having appropriate facilities.

While a wired USB connection is used in the disclosed embodiments, it will be appreciated that the display data may be transferred to the Display Interface Device by any communications media, such as wireless or optical links, and using any suitable communications protocols, such as IEEE1394, Ethernet, Thunderbolt, 802.11 wireless, etc. Similarly, the connection between the Display Interface Device and displays can use any suitable wired or wireless communication medium/protocol.

While the use of Public Virtual Displays has been proposed, where only a single application requires access to an external display, that application itself could contain a display service (and other software elements) as described above that creates a Private Virtual Display for its own use. Such an embodiment would otherwise operate as described above.

Claims

1. A method of providing display output from a computing device adapted to run an operating system and applications to a display interface device over a general-purpose data transmission channel using a display application, the method comprising:

initiating, by the display application running on the computing device a user-level service, the service arranged to provide display output to the general-purpose data transmission channel during operation of one or more other applications;

instructing the operating system to create a display buffer entity configured to receive display output from software running on the device; and

retrieving, by the user-level service, display output from the display buffer entity and outputting display data based on the retrieved display output for transmission to the display interface device over the general-purpose data transmission channel.

2. A method according to claim 1 , wherein the application includes the user-level service.

3. A method according to claim 1 or 2, wherein installation and/or operation of the display application and service do not require modification of the operating system and/or do not require installation of any operating system level software components.

4. A method according to any of the preceding claims, wherein the display buffer entity comprises an image data buffer or a reference to an image data buffer.

5. A method according to any of the preceding claims, wherein the operating system is an Android operating system.

6. A method according to claim 5, wherein the display buffer entity comprises an Android Virtual Display.

7. A method according to any of the preceding claims, wherein the instructing step comprises configuring the operating system to mirror a display output generated for a display of the device to the display buffer entity or Virtual Display.

8. A method according to any of the preceding claims, wherein the instructing step comprises configuring a compositor module of the operating system to output display data to the display buffer entity or Virtual Display.

9. A method according to any of the preceding claims, comprising configuring the service as a foreground service.

10. A method according to any of the preceding claims, comprising, at the user-level service, receiving a notification from the operating system that display output is available from the display buffer entity, and retrieving the display output from the display buffer entity in response to the notification.

1 1. A method according to any of the preceding claims, comprising processing the retrieved display output prior to transmission via the general purpose transmission medium, the processing preferably comprising one or more of: encoding the display output, compressing the display output, and encrypting the display output.

12. A method according to claim 1 1 , wherein the display application comprises an encoding module for performing the processing step.

13. A method according to any of the preceding claims, comprising providing the display output or processed display output to a channel interface module arranged to interface with the general-purpose data transmission channel.

14. A method according to claim 13, wherein the application includes the channel interface module.

15. A method according to claim 13 or 14, comprising obtaining an access permission for access to the general-purpose data transmission channel, and preferably providing the permission to the channel interface module to enable the channel interface module to send data to the general-purpose data transmission channel.

16. A method according to any of claims 13 to 15, comprising providing, by the channel interface module, the display data to an operating system interface component adapted to interface with the general-purpose data transmission channel, preferably wherein the operating system interface component comprises a device driver for the general-purpose data transmission channel.

17. A method according to any of claims 12 to 16, wherein one or both of the encoding module and the channel interface module of the display application comprise device native code.

18. A method according to any of the preceding claims, wherein the user-level service comprises virtual machine code for execution by a virtual machine of the device.

19. A method according to any of the preceding claims, comprising starting the display application and/or user-level service in response to the display interface device being connected to the computing device via the general-purpose data transmission channel.

20. A method according to any of the preceding claims, comprising terminating the display application and/or user-level service in response to disconnection of the display interface device.

21. A method according to any of the preceding claims, wherein the user-level service initiates the instructing and receiving steps in response to detecting connection of a display to the display interface device.

22. A method according to any of the preceding claims, comprising providing display output to a plurality of external displays connected to one or more display interface devices, and preferably further comprising creating a respective display buffer entity for each connected display, and optionally creating respective encoder and/or channel interface instances for each connected display.

23. A method according to any of the preceding claims, wherein the general-purpose data transmission channel comprises a Universal Serial Bus, USB.

24. A method according to any of the preceding claims, comprising receiving the display data at the display interface device, generating a display signal based on the received display data, and outputting the display signal to a display.

25. A method according to claim 24, wherein the display interface device is connected to the display by a special-purpose channel for transmission of a display signal, the special- purpose channel preferably comprising an HDMI, DVI, DisplayPort or VGA connection.

26. A method according to any of the preceding claims, wherein the computing device is a mobile device, optionally a smartphone or tablet computer.

27. A computing device adapted to run an operating system and applications and having means for providing display output to an external display using a general-purpose data transmission channel, the device comprising:

means for running a display application including a user-level display service, the display service arranged to provide display output to the general-purpose data transmission channel during operation of one or more other applications;

means for instructing the operating system to create a display buffer entity configured to receive display output from software running on the computing device; and means for receiving, by the user-level service, display output from the display buffer entity and outputting display data based on the received display output to the general- purpose data transmission channel for transmission to the external display.

28. A computing device according to claim 27, further having means for performing a method as set out in any of claims 1 to 26.

29. A system comprising:

a computing device as set out in claim 27 or 28; and

a display interface device connected to, and adapted to receive display data from, the computing device over a general-purpose data transmission channel and to output a display signal to a display based on the received display data;

the system optionally further comprising one or more displays connected to the display interface device for receiving the display signal.

30. A computer-readable medium comprising software code adapted, when executed on a data processing apparatus, to perform a method as set out in any of claims 1 to 26.

31. A computer-readable medium comprising software code for an application adapted, when executed on a data processing device, to provide display output to a display interface device over a general-purpose data transmission channel, the application comprising:

software code for instructing the operating system to create a display buffer entity configured to receive display output from software running on the device; and

software code implementing a user-level display service, the display service arranged to provide display output to the general-purpose data transmission channel during operation of one or more other applications, the user-level display service arranged to:

retrieve display output from the display buffer entity; and

output display data based on the retrieved display output for transmission to the display interface device over the general-purpose data transmission channel.

32. A computer-readable medium according to claim 31 , wherein the application further comprises one or more of:

software code implementing an encoding module arranged to process the retrieved display output, the processing preferably including one or more of encoding, compressing and encrypting the display output;

software code implementing a channel interface module arranged to receive the display output or processed display output, and output the display output or the processed display output to the general-purpose data transmission channel.

33. A computer readable medium according to claim 31 or 32, further comprising software code for performing a method as set out in any of claims 1 to 26.