WO2009143294A2 - Methods and systems for using external display devices with a mobile computing device - Google Patents
Methods and systems for using external display devices with a mobile computing device Download PDFInfo
- Publication number
- WO2009143294A2 WO2009143294A2 PCT/US2009/044741 US2009044741W WO2009143294A2 WO 2009143294 A2 WO2009143294 A2 WO 2009143294A2 US 2009044741 W US2009044741 W US 2009044741W WO 2009143294 A2 WO2009143294 A2 WO 2009143294A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- resource
- computing device
- mobile computing
- virtual
- output data
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/363—Graphics controllers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1415—Digital output to display device ; Cooperation and interconnection of the display device with other functional units with means for detecting differences between the image stored in the host and the images displayed on the displays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/14—Solving problems related to the presentation of information to be displayed
- G09G2340/145—Solving problems related to the presentation of information to be displayed related to small screens
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/06—Consumer Electronics Control, i.e. control of another device by a display or vice versa
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/16—Use of wireless transmission of display information
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/14—Display of multiple viewports
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/1813—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast for computer conferences, e.g. chat rooms
- H04L12/1827—Network arrangements for conference optimisation or adaptation
Definitions
- the present disclosure relates to methods and systems for using external display devices with a mobile computing device.
- output data associated with a resource executing on the mobile computing device may be displayed on the external display device.
- the output data may include a user interface appropriate for the display of the external display device.
- Execution of the resource may be transferred from the mobile computing device to the external display device.
- the mobile computing device may be repurposed to display a virtual input device for interacting with a resource with output data being displayed on an external display device.
- GUI graphical user interface
- mobile devices are often designed to be small enough to fit in a pocket, have relatively small displays, and some (often limited) means for data input such as telephone keypad, full keyboard, touch screen, stylus, etc.
- GUI graphical user interface
- limiting factors include the small display size and meager computational capacity.
- Various methods have been developed to run some desktop applications on mobile devices, but the results typically range from cumbersome and clumsy to not feasible or cost-prohibitive.
- the most common examples include rewriting applications, in whole or in part, to accommodate mobile device limitations, providing limited or "lite” versions of applications, scaling the application's graphics to fit on the mobile device display, and using zooming and scrolling to turn the mobile device display into a "virtual lens” to view sub-regions of the application at a time.
- a method for displaying, on an external display device connected to a mobile computing device, output data associated with at least one resource includes the step of receiving, by a virtual graphics driver on the mobile computing device, an identification of an external display device connected to the mobile computing device.
- the method includes the step of selecting, by a window management service on the mobile computing device according to the identification, at least one resource to display on the external display device.
- the method includes the step of positioning, by the window management service, output data associated with the at least one resource in a virtual screen space.
- the method includes the step of transmitting, by the virtual graphics driver, the output data to the external display device according to the virtual screen space.
- the method includes the step of allocating, by the window management service, a screen space corresponding to the external display device in the virtual screen space according to the identification. In some embodiments, the method includes the step of allocating, by the window management service, the screen space corresponding to the external display device according to a resolution of the external display device. In some embodiments, the method includes the step of removing, by the window management service, output data associated with the at least one resource from a position in the virtual screen space corresponding to a native display of the mobile computing device. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a type of the resource.
- the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a user association between the resource and a cell in a grid display layout. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a policy associated with cells in a grid display layout. In some embodiments, the method includes the step of re-positioning, by the window management service, the output data in the virtual screen space according to an alteration of a grid display layout. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a user command to display the output data in an empty cell.
- the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a policy associated with a dynamic display layout. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a default position associated with the resource. In some embodiments, the method includes the step of receiving, by the window management service, information associated with a user interaction with the output data. In some embodiments, the method includes the step of re-positioning, by the window management service, the output data according to the information associated with the user interaction.
- the method includes the step of re-positioning, by the window management service, the output data to a position in the virtual screen space corresponding to a native display of the mobile computing device or a different external display device. In some embodiments, the method includes the step of obscuring, by the window management service, output data associated with a different resource. In some embodiments, the method includes the step of re-sizing, by the window management service, the output data associated with the resource.
- a mobile computing device for displaying output data associated with at least one resource on an external display device.
- the mobile computing device also includes a window management service that selects, according to an identification of the external display device, the at least one resource to display on the external display device and positions output data associated with the at least one resource in a virtual screen space.
- the mobile computing device also includes a virtual graphics driver that transmits the output data to the external display device according to the virtual screen space.
- the window management service allocates a screen space corresponding to the external display device in the virtual screen space according to the identification.
- the window management service allocates a screen space corresponding to the external display device in the virtual screen space according to a resolution of the external display device.
- the window management service removes output data associated with the at least one resource from a position in the virtual screen space corresponding to a native display of the mobile computing device.
- a method for displaying, on an external display device connected to a mobile computing device, a user interface for a resource includes the step of receiving from the resource, by a window management service on the mobile computing device, a first user interface for display on a native display of the mobile computing device.
- the method also includes the step of transmitting, by a virtual graphics driver on the mobile computing device, the first user interface to the native display.
- the method also includes the step of transmitting, by the window management service, an identification of the external display device to the resource.
- the method also includes the step of receiving from the resource, by the window management service, a second user interface according to the identification, the second user interface being distinct from the first user interface.
- the method also includes the step of transmitting, by the virtual graphics driver, the second user interface to the external display device.
- the method includes the step of receiving from the resource, by the window management service, the second user interface optimized for the external display device. In more embodiments, the method includes the step of transmitting, by the window management service, at least one property of the external display device to the resource. In additional embodiments, the method includes the step of transmitting, by the window management service, an indication that a display on the external display device has a larger resolution or size than the native display. In many embodiments, the method includes the step of receiving from the resource, by the window management service, a default user interface in the absence of a user interface optimized for the external display device.
- the method includes the step of receiving from the resource, by the window management service, a communication that a user interface optimized for the external display device is unavailable. In some embodiments, the method includes the step of transmitting, by the window management system, an identification of the second user interface and an instruction to display the second user interface to the external display device.
- a method for transferring execution of a virtualized resource from a mobile computing device to an external display device also includes the step of receiving, by a server from the mobile computing device, an identification of an external display device connected to the mobile computing device.
- the method also includes the step of identifying, by the server, a virtual machine being delivered to the external display device.
- the method also includes the step of disconnecting, by the server, the virtualized resource from a virtual machine being delivered to the mobile computing device.
- the method also includes the step of connecting, by the server, the virtualized resource to the virtual machine being delivered to the external display device.
- the virtualized resource is a virtualized application.
- the method includes the step of transferring, by the server, information related to the state of execution of the virtualized resource from the virtual machine being to delivered to the mobile computing device to the virtual machine being delivered to the external display device.
- a method of repurposing a mobile computing device as a virtual input device for a first resource being displayed on an external display device also includes the step of evaluating, by the server, input capabilities of the mobile computing device and input needs of the first resource.
- the method also includes the step of delivering according to the evaluation, by the server to the mobile computing device, a second resource that generates a virtual input device.
- the method includes the step of delivering, by the server to the mobile computing device, the second resource that generates a virtual keyboard, virtual track pad, virtual pointing device, virtual camera, virtual microphone, virtual point-of- sale device, virtual speaker, virtual headphone jack, or any combination thereof.
- the method includes the step of delivering, by the server to the mobile computing device, the second resource that allows user interaction with output data of the first resource.
- the method includes the step of delivering, by the server to the mobile computing device, the second resource that transmits an identification of a user interaction with the virtual input device to the first resource.
- the method includes the step of delivering, by the server to the mobile computing device, the second resource that generates at least one user interaction element. In some embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that redirects user interactions with the virtual input device to the input capabilities of the mobile computing device. In many embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that correlates a location of a touch input with a displayed user interaction element. In various embodiments, the method includes the step of delivering, by the server to the mobile computing device, a third resource that generates another virtual input device in conjunction with the virtual input device of the second resource.
- the method includes the step of delivering, by the server to the mobile computing device, the second resource for display in conjunction with the first resource on a single display surface. In various embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that generates a virtual input device for a second external device connected to the mobile computing device.
- a system for repurposing a mobile computing device as a virtual input device for a first resource being displayed on an external display device also includes a server that evaluates input capabilities of the mobile computing device and input needs of the first resource and delivers to the mobile computing device, according to the evaluation, a second resource that generates a virtual input device.
- the second resource generates a virtual keyboard, virtual track pad, virtual pointing device, virtual camera, virtual microphone, virtual point-of- sale device, virtual speaker, virtual headphone jack, or any combination thereof.
- the second resource allows user interaction with output data of the first resource.
- the second resource transmits an identification of a user interaction with the virtual input device to the first resource.
- the second resource generates at least one user interaction element.
- the second resource redirects user interactions with the virtual input device to the input capabilities of the mobile computing device.
- the second resource correlates a location of a touch input with a displayed user interaction element.
- the server delivers a third resource that generates another virtual input device in conjunction with the virtual input device of the second resource.
- the second resource is displayed in conjunction with the first resource on a single display surface.
- the second resource generates a virtual input device for a second external device connected to the mobile computing device.
- FIG. IA is a block diagram depicting an embodiment of a network environment comprising client machines in communication with remote machines;
- FIGs. IB and 1C are block diagrams depicting embodiments of computers useful in connection with the methods and systems described herein;
- FIG. 2A is a block diagram depicting one embodiment of a system for displaying a plurality of resources in a user-configurable display layout on an external display device;
- FIG. 2B is a block diagram depicting a system for mapping a display of one or more resources to one or more display devices;
- FIG. 2C is a screen shot depicting one embodiment of a system for displaying a plurality of resources in a user-configurable display layout on an external display device, the user-configurable display layout divided into a grid;
- FIG. 2D is a screen shot depicting one embodiment of a system for displaying a plurality of resources in a user-configurable, dynamic display layout on an external display device;
- FIG. 3 A is a block diagram depicting one embodiment of a plurality of screen spaces provided by a mobile computing device attached to one or more external display devices;
- FIG. 3B is a block diagram depicting one embodiment of a mobile computing device providing a plurality of screen spaces
- FIG. 3 C is a block diagram depicting one embodiment of a logical representation of a plurality of screen spaces managed by a virtual graphics driver
- FIG. 4A is a block diagram depicting one embodiment of a graphical user interface for the addition of and configuration of additional screen spaces
- FIG. 4B is a block diagram depicting one embodiment of a graphical user interface for customizing a grid display layout
- FIG. 4C is a screen shot depicting one embodiment of a customizable grid display layout
- FIG. 4D is a screen shot depicting one embodiment of unstacked displays on a mobile computing device
- FIG. 4E is a flow diagram depicting one embodiment of the steps taken in a method for displaying, by an external display device, output data generated by a plurality of resources executing on a mobile computing device;
- FIG. 5 A is a block diagram depicting one embodiment of a system for displaying, by an external display device, a first user interface to a resource distinct from a second user interface to the resource displayed by a mobile computing device executing the resource;
- FIG. 5B is a block diagram depicting a system for mapping a display of one or more user interfaces to one or more display devices
- FIG. 5C is a screen shot depicting one embodiment of a system for displaying, by an external display device, a first user interface to an application distinct from a second user interface to the application displayed by a mobile computing device executing the application;
- FIG. 6 is a flow diagram depicting an embodiment of the steps taken in a method for displaying, by an external display device, a first user interface to a resource distinct from a second user interface to the resource displayed by a mobile computing device executing the resource.
- FIG. 7A is a block diagram depicting one embodiment of a system for executing and displaying a plurality of resources on a client computing device
- FIG. 7B is a block diagram depicting one embodiment of a system for executing a plurality of resources on a server and displaying them on a client computing device;
- FIG. 8A is a block diagram of one embodiment of a system for executing a plurality of resources from a mobile computing device on a remote server and displaying results on a client computing device;
- FIG. 8B is a flow diagram depicting one embodiment of the steps taken in a method for executing a plurality of resources from a mobile computing device on a remote server and displaying results on a client computing device;
- FIG. 9A is a block diagram depicting one embodiment of a server for displaying and activating disconnected sessions with a mobile computing device and/or a client computing device;
- FIG. 9B is a flow diagram depicting an embodiments of the steps taken in a method for receiving output data generated by the at least one previously disconnected session;
- FIG. 9C is a flow diagram depicting one embodiment of the steps taken in a method for displaying and activating disconnected sessions with a mobile computing device and/or a client computing device;
- FIG. 1OA is a block diagram depicting an embodiment of a system for generating a computing environment on a remote machine for display on a local machine;
- FIG. 1OB is a block diagram depicting one embodiment of a system in which an agent integrates applications from various sources into a computing environment
- FIG. 1 IA is a flow diagram depicting one embodiment of the steps taken in a method for generating a computing environment on a remote machine for display on a local machine;
- FIG. 1 IB is a flow diagram depicting one embodiment of the steps taken in a method for displaying, on the local machine, a desktop environment that is generated by the remote machine;
- FIG. 12A is a block diagram depicting an embodiment of a system for generating a computing environment incorporating output data from multiple remote machines for display on a local machine;
- FIG. 12B is a screen shot depicts one embodiment of a system for generating a computing environment incorporating output data from multiple remote machines for display on a local machine;
- FIG. 13 is a flow diagram depicting one embodiment of a method for generating a computing environment incorporating output data from multiple remote machines for display on a local machine;
- FIG. 14A is a block diagram depicting one embodiment of a system for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device;
- FIGS. 14B and 14C are screen shots depicting embodiments of a mobile computing device displaying a virtual input device for interaction by a user of the mobile computing device with a display of output data generated by an execution of a resource;
- FIG. 15 is a flow diagram depicting one embodiment of a method for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device;
- FIG. 16 is a flow diagram depicting one embodiment of a method for translating a user interaction with an input/output device of a first type into a command for processing by a user interface supporting interaction with an input/output device of a second type.
- FIGs. IA- 1C relate to general computing devices and networks
- FIGs. 2A-4E relate to methods and systems for displaying resources executing on a mobile computing device in a user configurable layout on an external display;
- FIGs. 5A-6 relate to dynamic switching between user interfaces
- FIGs. 7A-7B relate to methods and systems for displaying on a client computing device, output data generated by resources executed on the client computing device or a server;
- FIGs. 8A-8B relate to methods and systems for displaying on a client computing device, output data generated by resources executed on a mobile computing device;
- FIGs. 9A-9C relate to methods and systems for handling disconnected sessions between one or more of computing devices (included to support workspace control/smooth roaming aspects of a mobile computing device communicating with a server); and
- FIGs. 10A- 13 relate to methods and systems for generating a computing environment on one or more remote machines and displaying them on a local machine (included to provide support for combining Nirvana techniques with reverse seamless subject matter).
- FIGs. 14A-15 relate to methods and systems for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device.
- FIG. 16 relates to methods and systems for translating a user interaction with an input/output device of a first type into a command for processing by a user interface supporting interaction with an input/output device of a second type.
- the network environment comprises one or more clients 102a- 102n (also generally referred to as local machine(s) 102, client(s) 102, client node(s) 102, client machine(s) 102, client computer(s) 102, client device(s) 102, endpoint(s) 102, or endpoint node(s) 102) in communication with one or more servers 106a-106n (also generally referred to as server(s) 106 or remote machine(s) 106) via one or more networks 104.
- a client 102 has the capacity to function as both a client node seeking access to resources provided by a server and as a server providing access to hosted resources for other clients 102a-102n.
- FIG. IA shows a network 104 between the clients 102 and the servers 106
- the network 104 can be a local-area network (LAN), such as a company Intranet, a metropolitan area network (MAN), or a wide area network (WAN), such as the Internet or the World Wide Web.
- LAN local-area network
- MAN metropolitan area network
- WAN wide area network
- a network 104' may be a private network and a network 104 may be a public network.
- a network 104 may be a private network and a network 104' a public network.
- networks 104 and 104' may both be private networks.
- the network 104 may be any type and/or form of network and may include any of the following: a point to point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network and a wireline network.
- the network 104 may comprise a wireless link, such as an infrared channel or satellite band.
- the topology of the network 104 may be a bus, star, or ring network topology.
- the network 104 may be of any such network topology as known to those ordinarily skilled in the art capable of supporting the operations described herein.
- the network may comprise mobile telephone networks utilizing any protocol or protocols used to communicate among mobile devices, including AMPS, TDMA, CDMA, GSM, GPRS or UMTS.
- AMPS AMPS
- TDMA Time Division Multiple Access
- CDMA Code Division Multiple Access
- GSM Global System for Mobile communications
- GPRS Global System for Mobile communications
- UMTS Universal Mobile communications
- the system may include multiple, logically-grouped servers 106.
- the logical group of servers may be referred to as a server farm 38.
- the servers 106 may be geographically dispersed.
- a server farm 38 may be administered as a single entity.
- the server farm 38 comprises a plurality of server farms 38.
- the servers 106 within each server farm 38 can be heterogeneous - one or more of the servers 106 can operate according to one type of operating system platform (e.g., WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Washington), while one or more of the other servers 106 can operate on according to another type of operating system platform (e.g., Unix or Linux).
- operating system platform e.g., WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Washington
- the servers 106 of each server farm 38 do not need to be physically proximate to another server 106 in the same server farm 38.
- the group of servers 106 logically grouped as a server farm 38 may be interconnected using a wide-area network (WAN) connection or a metropolitan-area network (MAN) connection.
- WAN wide-area network
- MAN metropolitan-area network
- a server farm 38 may include servers 106 physically located in different continents or different regions of a continent, country, state, city, campus, or room. Data transmission speeds between servers 106 in the server farm 38 can be increased if the servers 106 are connected using a local-area network (LAN) connection or some form of direct connection.
- LAN local-area network
- Server 106 may be a file server, application server, web server, proxy server, appliance, network appliance, gateway, application gateway, gateway server, virtualization server, deployment server, SSL VPN server, or firewall.
- a server 106 provides a remote authentication dial-in user service, and is referred to as a RADIUS server.
- a server 106 may have the capacity to function as either an application server or as a master application server.
- a server 106 is a blade server.
- a server 106 executes a virtual machine providing, to a user or client computer 102, access to a computing environment.
- a server 106 may include an Active Directory.
- the server may include an Active Directory.
- the server 106 may be an application acceleration appliance.
- the server 106 may provide functionality including firewall functionality, application firewall functionality, or load balancing functionality.
- the server 106 comprises an appliance such as one of the line of appliances manufactured by the Citrix Application Networking Group, of San Jose, CA, or Silver Peak Systems, Inc., of Mountain View, CA, or of Riverbed Technology, Inc., of San Francisco, CA, or of F5 Networks, Inc., of Seattle, WA, or of Juniper Networks, Inc., of Sunnyvale, CA.
- a server 106 executes an application on behalf of a user of a client 102.
- a server 106 executes a virtual machine, which provides an execution session within which applications execute on behalf of a user or a client 102.
- the execution session is a hosted desktop session.
- the execution session provides access to a computing environment, which may comprise one or more of: an application, a plurality of applications, a desktop application, and a desktop session in which one or more applications may execute.
- a client 102 communicates with a server 106.
- the client 102 communicates directly with one of the servers 106 in a server farm 38.
- the client 102 executes a program neighborhood application to communicate with a server 106 in a server farm 38.
- the server 106 provides the functionality of a master node.
- the client 102 communicates with the server 106 in the server farm 38 through a network 104. Over the network 104, the client 102 can, for example, request execution of various applications hosted by the servers 106a-106n in the server farm 38 and receive output of the results of the application execution for display.
- only the master node provides the functionality required to identify and provide address information associated with a server 106b hosting a requested application.
- the server 106 provides the functionality of a web server.
- the server 106a receives requests from the client 102, forwards the requests to a second server 106b and responds to the request by the client 102 with a response to the request from the server 106b.
- the server 106 acquires an enumeration of applications available to the client 102 and address information associated with a server 106' hosting an application identified by the enumeration of applications.
- the server 106 presents the response to the request to the client 102 using a web interface.
- the client 102 communicates directly with the server 106 to access the identified application.
- the client 102 receives output data, such as display data, generated by an execution of the identified application on the server 106.
- the server 106 or a server farm 38 may be running one or more applications, such as an application providing a thin-client computing or remote display presentation application.
- the server 106 or server farm 38 executes as an application any portion of the CITRIX ACCESS SUITE by Citrix Systems, Inc., such as the METAFRAME or CITRIX PRESENTATION SERVER and/or any of the MICROSOFT WINDOWS Terminal Services manufactured by the Microsoft Corporation.
- the application is an ICA client, developed by Citrix Systems, Inc. of Fort Lauderdale, Florida.
- the server 106 may run an application, which, for example, may be an application server providing email services such as MICROSOFT EXCHANGE manufactured by the Microsoft Corporation of Redmond, Washington, a web or Internet server, or a desktop sharing server, or a collaboration server.
- any of the applications may comprise any type of hosted service or products, such as GOTOMEETING provided by Citrix Online Division, Inc. of Santa Barbara, California, WEBEX provided by WebEx, Inc. of Santa Clara, California, or Microsoft Office LIVE MEETING provided by Microsoft Corporation of Redmond, Washington.
- a client 102 may execute, operate or otherwise provide an application, which can be any type and/or form of software, program, or executable instructions such as any type and/or form of web browser, web-based client, client-server application, a thin-client computing client, an ActiveX control, or a Java applet, or any other type and/or form of executable instructions capable of executing on client 102.
- the application may be a server-based or a remote -based application executed on behalf of the client 102 on a server 106.
- the server 106 may display output to the client 102 using any thin-client or remote-display protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft.
- ICA Independent Computing Architecture
- the application can use any type of protocol and it can be, for example, an HTTP client, an FTP client, an Oscar client, or a Telnet client.
- the application comprises any type of software related to voice over internet protocol (VoIP) communications, such as a soft IP telephone.
- VoIP voice over internet protocol
- the application comprises any application related to real-time data communications, such as applications for streaming video and/or audio.
- the client 102 and server 106 may be deployed as and/or executed on any type and form of computing device, such as a computer, network device or appliance capable of communicating on any type and form of network and performing the operations described herein.
- FIGs. IB and 1C depict block diagrams of a computing device 100 useful for practicing an embodiment of the client 102 or a server 106. As shown in FIGs. IB and 1C, each computing device 100 includes a central processing unit 121, and a main memory unit 122. As shown in FIG.
- a computing device 100 may include a storage device 128, an installation device 116, a network interface 118, an I/O controller 123, display devices 124a- n, a keyboard 126 and a pointing device 127, such as a mouse.
- the storage device 128 may include, without limitation, an operating system, software, and a client agent 120.
- each computing device 100 may also include additional optional elements, such as a memory port 103, a bridge 170, one or more input/output devices 130a-130n (generally referred to using reference numeral 130), and a cache memory 140 in communication with the central processing unit 121.
- the central processing unit 121 is any logic circuitry that responds to and processes instructions fetched from the main memory unit 122.
- the central processing unit 121 is provided by a microprocessor unit, such as: those manufactured by Intel Corporation of Mountain View, California; those manufactured by Motorola Corporation of Schaumburg, Illinois; those manufactured by Transmeta Corporation of Santa Clara, California; the RS/6000 processor, those manufactured by International Business Machines of White Plains, New York; or those manufactured by Advanced Micro Devices of Sunnyvale, California.
- the computing device 100 may be based on any of these processors, or any other processor capable of operating as described herein.
- Main memory unit 122 may be one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the microprocessor 121, such as Static random access memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM), Dynamic random access memory (DRAM), Fast Page Mode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM), Extended Data Output DRAM (EDO DRAM), Burst Extended Data Output DRAM (BEDO DRAM), Enhanced DRAM (EDRAM), synchronous DRAM (SDRAM), JEDEC SRAM, PClOO SDRAM, Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), SyncLink DRAM (SLDRAM), Direct Rambus DRAM (DRDRAM), or Ferroelectric RAM (FRAM).
- SRAM Static random access memory
- BSRAM SynchBurst SRAM
- DRAM Dynamic random access memory
- FPM DRAM Fast Page Mode DRAM
- EDRAM Enhanced
- the main memory 122 may be based on any of the above described memory chips, or any other available memory chips capable of operating as described herein.
- the processor 121 communicates with main memory 122 via a system bus 150 (described in more detail below).
- FIG. 1C depicts an embodiment of a computing device 100 in which the processor communicates directly with main memory 122 via a memory port 103.
- the main memory 122 may be DRDRAM.
- FIG. 1C depicts an embodiment in which the main processor 121 communicates directly with cache memory 140 via a secondary bus, sometimes referred to as a backside bus.
- the main processor 121 communicates with cache memory 140 using the system bus 150.
- Cache memory 140 typically has a faster response time than main memory 122 and is typically provided by SRAM, BSRAM, or EDRAM.
- the processor 121 communicates with various I/O devices 130 via a local system bus 150.
- FIG. 1C depicts an embodiment of a computer 100 in which the main processor 121 communicates directly with I/O device 130b via HYPERTRANSPORT, RAPIDIO, or INFINIBAND communications technology.
- FIG. 1C also depicts an embodiment in which local busses and direct communication are mixed: the processor 121 communicates with I/O device 130a using a local interconnect bus while communicating with I/O device 130b directly.
- I/O devices 13 Oa- 13 On may be present in the computing device 100.
- Input devices include keyboards, mice, trackpads, trackballs, microphones, and drawing tablets.
- Output devices include video displays, speakers, inkjet printers, laser printers, and dye-sublimation printers.
- the I/O devices may be controlled by an I/O controller 123 as shown in FIG. IB.
- the I/O controller may control one or more I/O devices such as a keyboard 126 and a pointing device 127, e.g., a mouse or optical pen.
- an I/O device may also provide storage and/or an installation medium 116 for the computing device 100.
- the computing device 100 may provide USB connections (not shown) to receive handheld USB storage devices such as the USB Flash Drive line of devices manufactured by Twintech Industry, Inc. of Los Alamitos, California.
- the computing device 100 may support any suitable installation device 116, such as a floppy disk drive for receiving floppy disks such as 3.5-inch, 5.25-inch disks or ZIP disks, a CD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, tape drives of various formats, USB device, hard-drive or any other device suitable for installing software and programs.
- the computing device 100 may further comprise a storage device, such as one or more hard disk drives or redundant arrays of independent disks, for storing an operating system and other related software, and for storing application software programs such as any program related to the client agent 120.
- any of the installation devices 116 could also be used as the storage device.
- the operating system and the software can be run from a bootable medium, for example, a bootable CD, such as KNOPPIX, a bootable CD for GNU/Linux that is available as a GNU/Linux distribution from knoppix.net.
- a bootable CD such as KNOPPIX
- KNOPPIX a bootable CD for GNU/Linux that is available as a GNU/Linux distribution from knoppix.net.
- the computing device 100 may include a network interface 118 to interface to the network 104 through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (e.g., 802.11, Tl, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or some combination of any or all of the above.
- standard telephone lines LAN or WAN links (e.g., 802.11, Tl, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or some combination of any or all of the above.
- LAN or WAN links e.g., 802.11, Tl, T3, 56kb, X.25, SNA, DECNET
- broadband connections e.g., ISDN, Frame Relay
- Connections can be established using a variety of communication protocols (e.g., TCP/IP, IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), RS232, IEEE 802.11, IEEE 802.11a, IEEE 802.11b, IEEE 802.1 Ig, CDMA, GSM, WiM ax and direct asynchronous connections).
- the computing device 100 communicates with other computing devices 100' via any type and/or form of gateway or tunneling protocol such as Secure Socket Layer (SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida.
- SSL Secure Socket Layer
- TLS Transport Layer Security
- Citrix Gateway Protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida.
- the network interface 118 may comprise a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device 100 to any type of network capable of communication and performing the operations described herein.
- the computing device 100 may comprise or be connected to multiple display devices 124a-124n, which each may be of the same or different type and/or form.
- any of the I/O devices 13 Oa- 13 On and/or the I/O controller 123 may comprise any type and/or form of suitable hardware, software, or combination of hardware and software to support, enable or provide for the connection and use of multiple display devices 124a-124n by the computing device 100.
- the computing device 100 may include any type and/or form of video adapter, video card, driver, and/or library to interface, communicate, connect or otherwise use the display devices 124a-124n.
- a video adapter may comprise multiple connectors to interface to multiple display devices 124a-124n.
- the computing device 100 may include multiple video adapters, with each video adapter connected to one or more of the display devices 124a-124n.
- any portion of the operating system of the computing device 100 may be configured for using multiple displays 124a-124n.
- one or more of the display devices 124a-124n may be provided by one or more other computing devices, such as computing devices 100a and 100b connected to the computing device 100, for example, via a network. These embodiments may include any type of software designed and constructed to use another computer's display device as a second display device 124a for the computing device 100.
- a computing device 100 may be configured to have multiple display devices 124a-124n.
- an I/O device 130 may be a bridge between the system bus 150 and an external communication bus, such as a USB bus, an Apple Desktop Bus, an RS-232 serial connection, a SCSI bus, a Fire Wire bus, a Fire Wire 800 bus, an Ethernet bus, an AppleTalk bus, a Gigabit Ethernet bus, an Asynchronous Transfer Mode bus, a HIPPI bus, a Super HIPPI bus, a SerialPlus bus, a SCI/LAMP bus, a FibreChannel bus, or a Serial Attached small computer system interface bus.
- an external communication bus such as a USB bus, an Apple Desktop Bus, an RS-232 serial connection, a SCSI bus, a Fire Wire bus, a Fire Wire 800 bus, an Ethernet bus, an AppleTalk bus, a Gigabit Ethernet bus, an Asynchronous Transfer Mode bus, a HIPPI bus, a Super HIPPI bus, a SerialPlus bus, a SCI/LAMP bus, a FibreChannel bus, or a
- a computing device 100 of the sort depicted in FIGs. IB and 1C typically operates under the control of operating systems, which control scheduling of tasks and access to system resources.
- the computing device 100 can be running any operating system such as any of the versions of the MICROSOFT WINDOWS operating systems, the different releases of the Unix and Linux operating systems, any version of the MAC OS for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating systems for mobile computing devices, or any other operating system capable of running on the computing device and performing the operations described herein.
- Typical operating systems include, but are not limited to: WINDOWS 3.x, WINDOWS 95, WINDOWS 98, WINDOWS 2000, WINDOWS NT 3.51, WINDOWS NT 4.0, WINDOWS CE, WINDOWS XP, and WINDOWS VISTA, all of which are manufactured by Microsoft Corporation of Redmond, Washington; MAC OS, manufactured by Apple Computer of Cupertino, California; OS/2, manufactured by International Business Machines of Armonk, New York; and Linux, a freely-available operating system distributed by Caldera Corp. of Salt Lake City, Utah, or any type and/or form of a Unix operating system, among others.
- the computer system 100 can be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile telephone or other portable telecommunication device, media playing device, a gaming system, mobile computing device, or any other type and/or form of computing, telecommunications or media device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.
- the computer system 100 may comprise a device of the IPOD family of devices manufactured by Apple Computer of Cupertino, California, a PLAYSTATION 2 , PLAYSTATION 3, or PERSONAL PLAYSTATION PORTABLE (PSP) device manufactured by the Sony Corporation of Tokyo, Japan, a NINTENDO DS, NINTENDO GAMEBOY, NINTENDO GAMEBOY ADVANCED or NINTENDO REVOLUTION device manufactured by Nintendo Co., Ltd., of Kyoto, Japan, or an XBOX or XBOX 360 device manufactured by the Microsoft Corporation of Redmond, Washington.
- PSP PERSONAL PLAYSTATION PORTABLE
- the computing device 100 may have different processors, operating systems, and input devices consistent with the device.
- the computing device 100 is a TREO 180, 270, 600, 650, 680, 70Op, 70Ow, or 750 smart phone manufactured by Palm, Inc.
- the TREO smart phone is operated under the control of the PalmOS operating system and includes a stylus input device as well as a five-way navigator device.
- the computing device 100 is a mobile device, such as a
- JAVA-enabled cellular telephone or personal digital assistant such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the iml 100, all of which are manufactured by Motorola Corp. of Schaumburg, Illinois, the 6035 or the 7135, manufactured by Kyocera of Kyoto, Japan, or the i300 or i330, manufactured by Samsung Electronics Co., Ltd., of Seoul, Korea.
- the computing device 100 is a mobile device manufactured by Nokia of Finland, or by Sony Ericsson Mobile Communications AB of Lund, Sweden.
- the computing device 100 is a Blackberry handheld or smart phone, such as the devices manufactured by Research In Motion Limited, including the Blackberry 7100 series, 8700 series, 7700 series, 7200 series, the Blackberry 7520, or the Blackberry Pearl 8100.
- the computing device 100 is a smart phone, Pocket PC, Pocket PC Phone, or other handheld mobile device supporting Microsoft Windows Mobile Software.
- the computing device 100 can be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile telephone, any other computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.
- the computing device 100 is a digital audio player.
- the computing device 100 is a digital audio player such as the Apple IPOD, IPOD Touch, IPOD NANO, and IPOD SHUFFLE lines of devices, manufactured by Apple Computer of Cupertino, California.
- the digital audio player may function as both a portable media player and as a mass storage device.
- the computing device 100 is a digital audio player such as the DigitalAudioPlayer Select MP3 players, manufactured by Samsung Electronics America, of Ridgefield Park, NJ, or the Motorola m500 or m25 Digital Audio Players, manufactured by Motorola Inc. of Schaumburg, IL.
- the computing device 100 is a portable media player, such as the Zen Vision W, the Zen Vision series, the Zen Portable Media Center devices, or the Digital MP3 line of MP3 players, manufactured by Creative Technologies Ltd.
- the computing device 100 is a portable media player or digital audio player supporting file formats including, but not limited to, MP3, WAV, M4A/AAC, WMA Protected AAC, AIFF, Audible audiobook, Apple Lossless audio file formats and .mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats.
- the computing device 100 comprises a combination of devices, such as a mobile phone combined with a digital audio player or portable media player.
- the computing device 100 is a Motorola RAZR or Motorola ROKR line of combination digital audio players and mobile phones.
- the computing device 100 is an iPhone smartphone, manufactured by Apple Computer of Cupertino, California.
- FIG. 2A a block diagram depicts one embodiment of a system for displaying a plurality of resources on one or more display devices.
- the system includes a mobile computing device 102 that may communicate with one or more external display devices 202a-n.
- the embodiment in FIG. 2A shows a mobile computing device 102 with a native display 201, although devices without native displays may be used in other embodiments.
- the mobile computing device 102 executes a plurality of resources 204a-n (collectively, 204).
- the window management system 206 and virtual graphics driver 208 manage the locations and sizes of the display of output data associated with each of the plurality of resources in a user-configurable display layout.
- the mobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to an external display device 202. In some of these embodiments, the mobile computing device 102 transmits the output data upon establishing a connection with the external display device 202. In various embodiments, the mobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to the device's native display 201. In many embodiments, the mobile computing device 102 transmits the output data associated with certain of the plurality of resources 204 to the native display 201 and transmits the output data associated with other of the plurality of resources 204 to the external display devices 202a-n.
- the mobile computing device 102 executes a plurality of resources 204.
- the mobile computing device 102 is a client 102 as described above in connection with FIGs. 1A-1C.
- the mobile computing device 102 displays the output data associated with a resource 204a in a plurality of resources 204a-n executed by the mobile computing device 102.
- the mobile computing device 102 displays the output data associated with each of the plurality of resources 204.
- a resource in the plurality of resources 204 may include, without limitation, a data file, an executable file, configuration files, an application, a desktop environment (which may itself include a plurality of applications for execution by the user), a computing environment image (such as a virtual machine image), and/or operating system software or other applications needed to execute a computing environment image.
- the mobile computing device 102 includes a window management service 206 allowing an external display device 202 to display the output data associated with each of a plurality of resources 204 executed on the mobile computing device 102.
- the window management service 206 allows multiple resources running on the mobile computing device 102 to be viewed on an external display device 202 or the native display 201 at substantially the same time, as opposed to allowing the output of one resource 204 to be viewed exclusively on the native display 201 or external display device 202.
- the window management service 206 in conjunction with a virtual graphics driver 208, manages the display layout of the windows displayed on the external display device 202 and the native display 201.
- the virtual graphics driver 208 is a driver-level component that manages a virtual screen frame buffer storing output data that will be displayed by the native display 201 on the mobile computing device 102 or an external display device 202.
- the window management service 206 in conjunction with the virtual graphics driver 208, manages the boundaries and size of a screen space used to display output data and on which display device the output data is displayed.
- an external display device 202 receives output data associated with each of the plurality of resources 204 and displays the output data in a user- configurable display layout.
- the external display device 202 includes a dock to which the mobile computing device 102 connects.
- the external display device 202 includes a receiver for communicating with the mobile computing device 102 wirelessly, for example, via BLUETOOTH, Wi-Fi or other networking protocols, as described above in connection with FIGs. IA- 1C.
- the external display device 202 is a display device 124 as described above in connection with FIG. lB-lC.
- FIG. 2B a block diagram depicts one embodiment of a system for mapping the display of one or more resources 204 of the mobile computing device 102 on one or more display devices 201 and/or 202.
- the window management service 206 manages a virtual screen space 210.
- the virtual screen space 210 may map to the native display 201 and one or more external display devices 202.
- the window management service 206 may position output data associated with the user interfaces of one or more resources 204 on the virtual screen space 210 to specify where each user interface will be displayed.
- the window management service 206 positions the output data according to a user preference.
- the window management service 206 positions the output data according to a policy.
- the window management service 206 positions the output data based on the resource 204 associated with the output data.
- the window management service 206 communicates with the virtual graphics driver 208 to transmit output data associated with user interfaces of resources 204 to the native display 201 and one or more external display devices 202.
- the window management service 206 may transmit output data and associated coordinates from the virtual screen space 210 to the virtual graphics driver 208.
- the virtual graphics driver 208 stores the output data in a virtual screen frame buffer.
- the virtual graphics driver 208 transmits the entries in the virtual screen frame buffer to the native display 201 and external display devices 202.
- the virtual graphics driver 208 transmits an entry in the virtual screen frame buffer to a native display 201 or an external display device 202 based on the position of the entry in the frame buffer.
- a screen shot depicts one embodiment of a user- configurable display layout in which the external display device 202 displays the output data of the resources 204.
- the output data of the resources 204 is displayed in a grid display layout.
- the grid display layout may include one or more cells in an arrangement.
- a cell may display output data associated with a resource. In some embodiments, more than one cell displays output data associated with the same resource.
- the cells are uniformly sized, whereas in other embodiments, the cells have different sizes.
- the cells may be arranged in any configuration. In some embodiments, the cells may be arranged in rows, columns, or both.
- a cell may have a descriptor associated with the cell's position in the grid. The descriptor may indicate the position of a cell within a row. In the embodiment depicted in FIG.
- the cell for resource 204a may have the descriptor "1-1," the cell for resource 204b may have the descriptor "1-2,” the cell for resource 204c may have the descriptor "1-3,” the cell for resource 204d may have the descriptor "2-1,” and the cell for resource 204d may have the descriptor "2-2.”
- the cells may be numbered, e.g. "Cell 1,” “Cell 2,” etc. However, any system of choosing descriptors known to those of ordinary skill in the art may be used.
- the window management service 206 configures a grid display layout according to the resources 204 being displayed on the native display 201 or the external display device 202. In some embodiments, the service 206 configures a grid display layout according to the number of resources 204 being displayed. In other embodiments, the service 206 configures a grid display layout according to the size or amount of content in the user interfaces of the resources 204. For example, if an external display device 202 will display four resources with comparable amounts of content, the window management service 206 may configure a grid display layout with four uniform cells.
- the window management service 206 may configure a grid display layout with three uniform cells in a first row and a single cell in a second row.
- the single cell in the second row may be three times as wide as the cells in the first row.
- the window management service 206 may configure a grid display layout to reserve a cell for displaying information about the resources being displayed, such as a menu of the resources.
- the window management service 206 may configure a grid display layout to reserve a cell for allowing a user to configure the grid display layout.
- a screen shot depicts one embodiment of a user- configurable display layout providing a dynamic display layout in which the external display device 202 displays the output data associated with the plurality of resources 204.
- windows on the external display device 202 that display output data for resources 204 may be dynamically positioned and sized.
- the window management service 206 may position a user interface for a resource at a default position and with a default size chosen according to a policy, the resource 204, or any other method.
- the window management service 206 may order overlapping user interfaces such that higher-order user interfaces obscure lower-order user interfaces.
- the window management service 206 may transmit output data to the virtual graphics driver 208 reflecting the obfuscation.
- the user may re-position or re-size a window by, for example, clicking and dragging the window or a window edge.
- the virtual graphics driver 208 may detect the user's change to the window, and transmit information about the user's change to the window management service 206.
- the window management service 206 may process the change and transmit updated output data to the virtual graphics driver 208.
- the user moves the user interface for a resource 204 to any location on a native display 201 or external display device 202.
- the user moves the user interface for a resource 204 to a different display device.
- the updated output data indicates that one user interface's size has been increased or location has been adjusted to obscure another user interface.
- the updated output data indicates that one user interface's size has been decreased or location has been adjusted such that more of another user interface shall be visible.
- FIG. 3A a block diagram depicts one embodiment of virtual screen spaces 210 provided by a mobile computing device attached to one or more external display devices.
- the mobile computing device 102 includes a virtual graphics driver 208 and a virtual screen 210.
- the virtual screen 210 includes a plurality of virtual screen spaces 310 and 312a-n.
- Virtual screen space 310 may be a native display screen space for the native display 201 on the mobile computing device 102.
- the other virtual screen spaces 312a-n may be extended screen spaces that correspond to the displays of external display devices 202.
- the window management service 206 and virtual graphics driver 208 manage the virtual screen 210.
- the virtual graphics driver 208 uses a virtual screen frame buffer to manage the mobile computing device's native display 201 and change the native display's 201 screen resolution. In another embodiment, the virtual graphics driver 208 uses a virtual screen frame buffer to manage an extended screen space 312 and to change a resolution of the extended screen 312.
- the virtual graphics driver 208 allocates and manages a plurality of virtual screen spaces 310, 312a-n and virtual screen frame buffers.
- each virtual screen space and virtual screen frame buffer has a resolution independent of the other screen spaces and frame buffers.
- output data associated with each of the plurality of resources 204 can reside within any of the virtual screen spaces 310, 312a-n.
- each of the extended screen spaces 312a-n is associated with at least one external display device 202, depending on the capabilities of the device.
- the window management service 206 and the virtual graphics driver 208 allocate and manage the display, on a plurality of external display devices 202, of output data associated with a plurality of resources.
- output data associated with a resource 204a displays on a mobile computing device 102
- output data associated with a resource 204b displays on one external display device 202a
- output data associated with a resource 204c display on another external display device 202b.
- the window management device 206 identifies one of the external display devices 202 for displaying output data generated by a resource 204a based upon a type of the resource 204a.
- the window management service 206 may determine that a type of resource rendering a video may display on a television screen, while a type of resource rendering a word processing application may render on a display of a laptop computer.
- FIG. 3B a block diagram depicts one embodiment of a mobile computing device 102 providing a virtual screen 210 with virtual screen spaces 310, 312a-n of varying resolutions.
- the virtual screen 210 includes a native display screen space 310 corresponding to the native display 201 of the mobile computing device 102 with a resolution of 320 pixels x 240 pixels.
- the virtual screen 210 also includes an extended screen 312a corresponding to the display of an external display device 202 with a resolution of 1024 pixels x 768 pixels, an extended screen 312n-l corresponding to the display of an external display device 202 with a resolution of 800 pixels x 600 pixels, and an extended screen 312n corresponding to the display of an external display device 202 with a resolution of 640 pixels x 480 pixels.
- the virtual screen 210 may include a native display screen space 310 and any number of extended screens 312 of any resolution.
- the entire virtual screen space 210 may be mapped into a single virtual screen frame buffer, although embodiments that map into multiple buffers may be used.
- FIG. 3C a block diagram depicts one embodiment of a logical representation of a plurality of virtual screen spaces managed by a virtual graphics driver.
- the virtual graphics driver 208 manages multiple virtual screen spaces with different resolutions in a virtual screen frame buffer.
- the native display 201 of the mobile computing device is the primary display and the external display device 202, corresponding to the extended screen 312a, is a secondary display.
- output data associated with resources 204 on the native display screen space 310 will be displayed on the native display 201 and output data associated with resources 204 on the extended screen space 312a will be displayed on the external display device 202 associated with the extended screen spaces 312a.
- FIG. 4A a block diagram depicts one embodiment of a graphical user interface for customizing a dynamic display layout.
- the user selects a screen space and sets parameters to configure the space.
- the user selects a screen space from interface element 401.
- the user may select a screen space by selecting a tab, icon, button, or radio button associated with a screen space, selecting a screen space from a drop-down menu, typing the name of a screen space into a text box, or any other methods known to persons of ordinary skill in the art.
- the user may create a new screen space.
- the user may create a new screen space by selecting the "Add New Display” button 403, and an interface element 401 corresponding to the new screen space may be created and added to the graphical user interface.
- the user may name the new screen space, assign an external display device 202 corresponding to the space, and configure the screen space according to the methods described below.
- the graphical user interface may include interface elements the user manipulates to set parameters for the space.
- the graphical user interface includes interface elements 402 and 404 for setting the resolution and mode of the screen space, respectively, and in many embodiments, the graphical user interface may include interface elements for setting any other parameters such as z-order or default position.
- the user sets the resolution by selecting a radio button from an interface element 402 corresponding to a predetermined resolution or selecting a radio button to customize the width and height of the screen's resolution.
- the user may set the resolution by adjusting a slider to a desired resolution, selecting a thumbnail from a group of thumbnails representing displays of the screen space at different resolutions, or clicking and dragging a boundary of an image representing the screen space to change the resolution.
- the user sets the mode by selecting a radio button from an interface element 404 indicating whether the user wishes the screen space to be a primary display, an extended display, a display that mirrors the primary display, or a disabled display, although any other modes may be used.
- the user may set the mode by selecting an icon representing the mode, selecting the mode from a drop-down menu, or any other method.
- a block diagram depicts one embodiment of a graphical user interface for customizing a grid display layout.
- the user may select the screen space to customize according to any of the methods described in FIG. 4A.
- the graphical user interface allows users to configure the number, size, and position of cells within a grid in which the output data for each of the plurality of resources 204 is displayed.
- the user may add cells to the grid by selecting the user interface element 406, an "Add Cell” button. Each time the user selects the "Add Cell” button, a new numbered cell appears in the grid, and the user may delete a cell by selecting the cell and the user interface element 407, a "Delete Cell” button.
- the user may configure the position, resolution, or both of each cell.
- the user may configure the position by dragging and dropping the cell to the desired location.
- the user may configure the position by entering a numerical value corresponding to the desired position for the cell. For example, the user may select a cell and input the position "2-1" to indicate the cell should appear in the second row and first column of the grid.
- the remaining cells may automatically readjust their positions in response to the user's selection of a position for the new cell.
- the user may configure the resolution of each cell.
- the user configures the resolution by dragging and dropping a boundary of a cell to the desired size.
- the user configures the resolution by selecting from a group of resolutions.
- the embodiment shown in FIG. 4B depicts a user interface element 408 in the form of radio buttons corresponding to a group of resolutions, including a radio button that permits the user to customize the resolution of the cell.
- a user may select a resolution by selecting a check box, clicking a button, choosing a resolution from a drop-down menu, or any other known method of inputting information.
- a user may configure the grid display layout prior to the display of output data by the external display device 202. For example, the user may choose a setting associated with a preferred cell position and resolution for each resource in the plurality of resources 204 in a user profile. Upon execution of the resource, the window management service 206 may consult the user profile to determine how to display the user interface of the resource 204. In other embodiments, the user may configure the grid display layout during the display of output data by the external display device 202. For example, upon execution of a resource 204 by the mobile computing device 102, the user may choose or modify a setting associated with a preferred cell position and resolution for the resource 204.
- a user may associate output data for a resource 204 with a cell in the grid display layout.
- the user may select a cell on the grid display layout and associate a resource with the cell such that output data of the resource appears in the selected cell.
- the user may associate a plurality of resources with the same cell and prioritize the resources to determine which output data will be displayed in the cell if more than one of the associated resources are being executed.
- the user may associate a resource with a cell by dragging the output data generated by the resource to the cell in the grid display layout.
- the window management service 206 may store the association between the cell and the resource 204.
- a screen shot depicts one embodiment of a customized grid display layout.
- the grid display layout includes output data for a calendar 420, file explorer 422, and e-mail 424 corresponding to cells 1, 2, and 3 in the grid layout of FIG. 4B.
- no resources have been associated with the interface element 430 corresponding to cell 4, an empty cell.
- empty cells include a user interface element 430 for selecting a resource whose output data should display in the cell.
- the user interface element 430 enumerates at least one resource 204 whose output data is available for display on the external display device 202.
- the window management system 206 upon selection of a resource 204 by a user via the user interface element 430, receives an identification of the selected resource 204.
- the window management system 206 may associate the selected resource with the cell corresponding to the user interface element 430.
- the window management system 206 in communication with the virtual graphics driver 208, manages an association between the selected resource and the cell corresponding to the user interface element 430 to output data associated with the resource to the cell in the customizable grid display layout.
- a screen shot depicts one embodiment of a dynamic display layout provided by a mobile computing device 102 displaying the output data generated by each of a plurality of resources 204 on the native display 201 of the mobile computing device 102.
- the output data for all the resources may be displayed on the native display 201 because the mobile computing device 102 is not attached to any external display devices.
- the dynamic layout includes four windows 440, 442, 444, and 446 that contain output data generated by four resources executing on the mobile computing device 102.
- a window 440 for a resource 204 may be dynamically displayed on the native display 201 of the mobile computing device 102 when the resource 204 is first executed.
- the user may dynamically resize the window 440 by, for example, dragging and dropping a boundary of a window 440 to the desired location.
- the user may move the window 440 to any position on the native display 201.
- the user may move the window 440 to an external display device 202 connected to the mobile computing device 102.
- the user may minimize the window 440 on the native display 201 of the mobile computing device 102.
- the user may change focus from a window displaying output data associated with one resource 204a to a window displaying output data associated with another resource 204b. The user may change focus by selecting the window containing output data for the desired resource 204b.
- the output data generated by resource 204b may obscure output data generated by other resources 204 executing on the mobile computing device 102.
- the windows 440-448 can be adjusted in any manner and/or form on any display, either the native display 201 or external display devices 202, as apparent to one of ordinary skill in the art.
- a flow diagram depicts one embodiment of the steps taken in a method for displaying, by an external display device 202, output data generated by a resource 204 executing on a mobile computing device 102.
- the method includes the step of executing, by a mobile computing device 102, a plurality of resources 204 (step 402).
- the method includes the step of displaying, by the external display device 202, in a user-configurable display layout, the received output data associated with each of the plurality of resources 204 (step 404).
- a mobile computing device Referring still to FIG. 4E, and in greater detail, a mobile computing device
- the external display device displays, in a user-configurable grid display layout, the received output data generated by each of the plurality of resources 204 (step 404). As described above in connection with FIGS. 2A-2D, in various embodiments, the mobile computing device 102 determines for each of the plurality of resources 204 whether to transfer output data associated with the resource 204 to the external display device 202.
- a window management service 206 manages the size, position, resolution, or any other characteristic of a resource 204 in the virtual screen space 210.
- a window management service 206 executing on the mobile computing device 102, and in communication with a virtual graphics driver 208, identifies the resources 204 whose output data the external display device 202 should display.
- the window management service 206 receives, from the virtual graphics driver 208, an identification of an external display device 202 attached to the mobile computing device 102.
- the window management service 206 evaluates each of the plurality of resources 204 to determine whether the external display device 202 should display the output data generated by each of the plurality of resources 204.
- the window management service 206 evaluates a setting associated with each of the plurality of resources 204 to determine whether the external display device 202 should display the output data generated by each of the plurality of resources 204. In yet another embodiment, the window management service 206 evaluates a user- specified preference setting associated with each of the plurality of resources 204 to determine whether the external display device 202 should display the output data generated by each of the plurality of resources 204.
- the window management service 206 instructs the virtual graphics driver 208 to forward output data generated by a resource 204 executing on the mobile computing device to the external display device 202 for display. In another embodiment, the window management service 206 instructs the virtual graphics driver 208 to forward the output data generated by the resource 204 to the external display device responsive to determining that a setting associated with the resource 204 indicates that the output data generated by the resource 204 should be displayed on the external display device 202.
- a plurality of settings are associated with the resource 204, each of the plurality of settings indicating whether the output data generated by the resource 204 should be displayed on each of a plurality of types of external display device 202; for example, a first setting associated with the resource 204 may indicate that the generated output data should be displayed on the external display device 202 if the external display device 202 is a television screen and a second setting associated with the resource 204a may indicate that the generated output data should not be displayed on the external display device 202 if the external display device 202 has a display below a certain size and/or resolution.
- the window management service 206 instructs the virtual graphics driver 208 to have the output data for a resource 204 displayed on the external display device 202 in a grid display layout. In another embodiment, the window management service 206 instructs the virtual graphics driver 208 to have the output data for a resource 204 displayed on the external display device 202 in a dynamic display layout. In still another embodiment, the window management service 206 evaluates a setting associated with each of the plurality of resources 204 to identify a display layout according to which the external display device 202 should display the output data generated by each of the plurality of resources 204.
- the window management service 206 evaluates a user- specified preference setting associated with each of the plurality of resources 204 to identify a display layout according to which the external display device 202 should display the output data generated by each of the plurality of resources 204; for example, and in some embodiments, a user specifies a subset of the plurality of resources 204 for which the external display device 202 should display output data.
- the virtual graphics driver 208 forwards output data generated by a resource 204 executing on the mobile computing device to the external display device 202 for display. In another embodiment, the virtual graphics driver 208 instructs an operating system of the mobile computing device 102 to forward output data generated by a resource 204 executing on the mobile computing device to the external display device 202 for display. In still another embodiment, the virtual graphics driver 208 forwards an identification of a display layout to the external display device 202. In yet another embodiment, the virtual graphics driver 208 instructs an operating system of the mobile computing device 102 to forward an identification of a display layout to the external display device 202.
- FIG. 5 A a block diagram depicts one embodiment of a system for displaying, by an external display device 202, a second user interface for a resource 204 distinct from a first user interface for the resource 204 displayed by the mobile computing device 102 that is executing the resource 204.
- the system includes a window management service 206, a virtual graphics driver 208, and resources 204 that may provide more than one user interfaces 502a-n.
- the resource 204 may provide a user interface 502a optimized for the native display 201 on the mobile computing device 102.
- the resource 204 may provide a user interface 504b optimized for display on the external display device 202.
- the resource 204 may provide user interfaces 502a-n optimized for display on different external display devices 202.
- the window management service 206 provides functionality to allow resources to dynamically change user interfaces 502 to adapt to the external display device 202 that will display output data generated by the resource.
- the window management service 206 indicates to a resource 204 the properties of the external display device 202 that will display the resource's output data.
- the resource 204 provides a user interface 502b-n appropriate for the properties of the external display device 202, which may differ from the user interface 502a provided to the native display 201 of the mobile computing device.
- the mobile computing device 101 may execute a plurality of resources 204, and each resource may provide a plurality of user interfaces 502 optimized for display on the native display 201 of a mobile computing device 102 or on an external display device 202.
- FIG. 5B a block diagram depicts one embodiment of a system for mapping a first user interface 502a on the native display 201 of the mobile computing device 102 to a second user interface 502b on one or more external display devices 202.
- the virtual graphics driver 208 maps all the screen spaces 210a-n from the display of the mobile computing device 102 and/or the external display devices 202 into one virtual screen space 210.
- the virtual graphics driver 208 communicates the coordinates related to the virtual screen space 210 to the window management service 206.
- the window management service 206 positions user interfaces of one or more resources 204 onto the virtual screen space 210.
- the window management services 206 uses information provided by the virtual graphics driver 208 to position the user interfaces on an external display device 202.
- the user interface 502b for the resource 204 displayed by an external display device 202 is substantially same as the user interface 502a displayed on the native display 201.
- the user interface 502b displayed by an external display device 202 is distinct from the user interface 502a displayed on the native display 201.
- the user interface for any resource 204 displayed on an external display device 202 is distinct from the user interface 510 displayed on a different external display device 202.
- a screen shot depicts one embodiment of a system for displaying, by an external display device 202, a second user interface 502b for a resource 204 distinct from a first user interface 502a for the resource 204 displayed by a mobile computing device 102 that is executing the resource 204.
- a resource 204 may generate a plurality of user interfaces 502a and 502b, each of which may be optimized for use with a different type of display device.
- a first user interface 502a of the plurality of user interfaces may be optimized for use with the native display 201 of a mobile computing device 102.
- the second user interface 502b of the plurality of user interfaces may be optimized for use with an external display device 202.
- the first user interface 502a may be displayed on the native display 201 of the mobile computing device 102 until the mobile computing device 102 is attached to an external display device 202.
- the second user interface 502b may be displayed on the external display device 102.
- the mobile computing device 102 may cease to display the first user interface 502a upon attachment, and in other embodiments, the mobile computing device 102 may continue to display the first user interface 502a as the external display device 102 displays the second user interface 502b.
- a flow diagram depicts an embodiment of the steps taken in a method for displaying, by an external display device, a first user interface to an application distinct from a second user interface to the application displayed by a mobile computing device executing the application.
- the method includes the step of generating, by a resource executing on a mobile computing device, a first user interface for display on the mobile computing device (step 610).
- the method includes the step of receiving, by the resource, from a window management service, an identification of an external display device (step 612).
- the method includes the step of generating, by the resource, a second user interface for display on the external display device, responsive to receiving the identification of the external display device (step 614).
- a resource executing on a mobile computing device generates a first user interface for display on the mobile computing device (step 610).
- the resource 204 generates a user interface 502a optimized for use with the mobile computing device 102.
- the resource receives, from a window management service, an identification of an external display device (step 612).
- the resource receives the identification of the external display device 202 via a virtual graphics driver 208.
- the resource In response to receiving the identification of the external display device 202, the resource generates a second user interface 502b for display on the external display device (step 614).
- the window management service 206 receives an identification of an external display device 202 to which the mobile computing device 102 has attached.
- the window management service 206 transmits, to the resource, an application programming interface (API) call indicating to the resource that the external display device 202 is available.
- API application programming interface
- the window management service 206 may inform the resource 204 executing on the mobile computing device of the identification of the attached external display device 202.
- the window management service 206 indicates to the resource 204 that the external display device 202 has a display with a larger resolution or size than the native display 201 on the mobile computing device 102. In further embodiments, the window management service 206 allows the resource to change its user interface to an interface more optimized for use with the external display device 202.
- the window management service receives, from the resource, an identification of a user interface associated with the attached external display device.
- the window management service 206 receives the identification via a virtual graphics driver 208.
- the window management service 206 forwards the identification to the virtual graphics driver 208.
- the window management service 206 receives, from the resource, a user interface associated with the attached external display device 202. In another embodiment, when a user interface associated with the attached external display device is not available, the window management service 206 receives, from the resource, a default user interface. In still another embodiment, the window management service 206 receives from the resource a communication that a user interface associated with the attached external display device 202 is not available. [0130] In one embodiment, the resource 204 generates a user interface 502b optimized for use with an external display device 202 to which the mobile computing device 102 is attached.
- the resource 204 instructs an operating system on the mobile computing device 102 to display one of a plurality of user interfaces selected in response to receiving an identification of the external display device 202 upon which the user interface will be displayed.
- the resource 204 instructs the window management system 206 on the mobile computing device 102 to display one of the plurality of user interfaces selected responsive to receiving an identification of the external display device 202 upon which the user interface will be displayed.
- the window management system 206 transmits, to the external display device 202, an identification of a user interface 504 with an instruction to display the user interface 504.
- the window management system 206 transmits, to a virtual graphics driver 208, an identification of a user interface 504 with an instruction to display the user interface 502b on the external display device 202.
- FIG. 7 A a block diagram depicts one embodiment of a system for executing and displaying a plurality of resources 204a-204n (204 in general) on a client 102b.
- the system includes a client 102a, which is a mobile computing device, in connection with another client 102b.
- the client 102b includes a dock to which the mobile computing device 102a attaches.
- the client 102b includes a receiver for communicating with the mobile computing device 102a wirelessly, for example, via BLUETOOTH, Wi-Fi or other networking protocols, as described above in connection with FIGs. IA- 1C.
- the client 102b includes an external display 124 as described above in connection with FIG. 1B-1C.
- one or more resources 204 execute on the mobile computing device 102a.
- the mobile computing device 102a accesses hardware and/or software components of the client 102b to execute the resources 204.
- a version of a resource 204 is executed on the client 102b.
- the version of the resource 204 on the client 102b can be a local copy of the resource 204 installed on the client 102b.
- the version on the client 102b can offer more functionalities of the resource 204 than available on the mobile computing device 102a.
- the version on the client 102b can be another resource 204b producing the same output data as the resource 204 executing on the mobile computing device 102a.
- the mobile computing device 102a and the client 102b share hardware and software resources to execute one or more resources 204.
- Execution of the resource 204 may be transferred to the client 102b in many ways.
- the execution of the resource 204 is terminated on the mobile computing device 102a and the version of the resource 204 on the client 102b is started.
- information and/or data values related to the state of execution of the resource 204 on the mobile computing device 102a is transferred to the client 102b.
- no information and/or data values related to the state of execution of the resource 204 on the mobile computing device 102a is transferred to the client 102b. In such an embodiment, recent updates are not transferred to the client 102b when the execution is transferred from the mobile computing device 102a to the client 102b.
- the resource 204 is transferred to the client 102b when the execution is transferred from the mobile computing device 102a to the client 102b.
- information and/or data values related to the state of execution is also transferred when the resource 204 is transferred to the client 102b.
- the client 102b includes an operational or performance characteristic not provided by the mobile computing device 102a.
- the client 102b has a more powerful processor and/or larger memory than the processor and memory of the mobile computing device 102a.
- the client 102b provides an I/O device, display device, installation device, or other peripherals, such as a keyboard or printer not available to the mobile computing device 102a.
- the client 102b may provide a feature, a resource, or peripheral desired to be used by the user of the mobile computing device 102a For example, the user may want to access a file or an application provided on a remote machine available via a connection across a network.
- the client 102b provides access to machines on a network 104, such as those in machine farm 38, not available to the mobile computing device 102a, or to a user of the mobile computing device 102a.
- one or more resources 204 may execute in a virtual machine on the mobile computing device 102a.
- a virtual machine executing on the mobile computing device 102a provides access to a computing environment based at a remote location.
- an application program stored in the mobile computing device 102a executes to access data associated with the computing environment provided on the mobile computing device 102a.
- the mobile computing device 102a executes virtualization software, at least a portion of which is stored on the mobile computing device 102a.
- the mobile computing device 102a provides access to a computing environment by executing an operating system with access to one or more applications stored on the mobile computing device 102a, the operating system and the one or more applications having access to user data stored in the mobile computing device 102a.
- the mobile computing device 102a executes a virtual machine, responsive to data stored in the mobile computing device 102a. In another embodiment, the mobile computing device executes a virtual machine responsive to a policy stored in the mobile computing device 102a. In still another embodiment, the mobile computing device 102a executes a virtual machine that provides access to a requested resource or computing environment, the virtual machine executed responsive to a virtual machine image stored in the mobile computing device 102a. In yet another embodiment, the mobile computing device 102a transfers execution of the virtual machine to the client 102b.
- the client 102b connects to the mobile computing device 102a, executes a virtual machine, and provides access to a computing environment responsive to data stored in the mobile computing device 102a.
- the client 102b may mount the storage device of the mobile computing device 102a as a removable hard drive or storage element 128 of the client 102b.
- the mobile computing device 102a may be a plug and play device (PnP) of the client 102b, such that a PnP protocol such as that manufactured by Microsoft Corporation of Redmond, Washington, is used between the mobile computing device 102a and client 102b, such as via I/O devices 130a- 130n or a network interface 118.
- PnP plug and play device
- the client 102b provides access to a computing environment based on a portable computing environment provided in the mobile computing device 102a.
- the client 102b executes a virtual machine and a virtualization layer to execute the computing environment based on a virtualization software, virtual machine image, or user data.
- the client 102b includes a transceiver for accessing data stored in the mobile computing device 102a.
- a loading mechanism on the mobile computing device is not limited to:
- a actuates the establishment of the computing environment on the client 102b based on the portable computing environment stored in the mobile computing device 102a.
- the loading mechanism of the client 102b actuates the establishment of the computing environment.
- a loading mechanism on the mobile computing device 102a works in conjunction with the loading mechanism of the client 102b to establish the computing environment.
- FIG. 7B a block diagram depicts one embodiment of a system for executing resources 204 on a remote server 106 and displaying output data on a client computing device 102b.
- the system includes the mobile computing device 102a, a client 102b and a server 106.
- the mobile computing device 102a may be attached or connected to the client 102b in a plurality of possible ways as described with reference to FIG. 7A.
- the client 102b connects to a remote server 106 to request execution of a resource 204 by the server 106.
- the server 106 may have an operational or performance characteristic not present in the client 102b.
- the server 106 has more powerful hardware and/or software resources not available to the client 102b.
- the server 106 provides access to machines on a network 104, such as those in machine farm 38, not directly available to the client 102b.
- one or more resources 204 execute on the client 102b.
- the one or more resources 204 may include an application program to establish a remote connection with the server 106.
- the client 102b may execute, operate or otherwise provide the application, which can be any type and/or form of software, program, or executable instructions such as any type and/or form of web browser, web-based client, client-server application, a thin-client computing client, an ActiveX control, or a Java applet, or any other type and/or form of executable instructions capable of executing on the client 102.
- the application may be a server-based application executed on behalf of the client 102 on a server 106.
- the server 106 may communicate with the client 102 using a presentation layer protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Washington.
- ICA Independent Computing Architecture
- RDP Remote Desktop Protocol
- one or more resources 204 may execute in a virtual machine on the client 102b as described above in connection to FIG. 7A.
- the server 106 connects to the client 102b, executes a virtual machine, and provides access to a computing environment.
- the server 106 may transmit the virtual machine image to the client 102b.
- the server 106 transmits output data generated by the virtual machine image executed on the server 106 to the client 102b.
- the virtual machine image includes a version of each of the plurality of resources 204 of the mobile computing device 102a.
- Output data from the execution of the resources 204 are displayed in one or more ways.
- the output data generated by an execution of a resource 204a on the server 106 is displayed on the mobile computing device 102a.
- the output data generated by an execution of a resource 204 on the server 106 is displayed on the client 102b.
- the output data from the server 106 is displayed on the client 102b as a part of a computing environment.
- a resource 204 is executed locally on the mobile computing device 102a and displayed on the client 102b.
- output data is simultaneously displayed in the mobile computing device 102a and the client 102b.
- a block diagram depicts one embodiment of a system for executing resources for a mobile computing device remotely on a server and displaying output data generated by the resources on the mobile computing device, the client, or both.
- the system includes a mobile computing device 102a in communication with a client 102b over a connection 104.
- the system also includes a server 106 communicating with the mobile computing device 102a over a connection 104' and the client 102b over a connection 104".
- one or more resources 204 executes on the mobile computing device 102a.
- the one or more resources 204 may include an application program to establish a connection 104 with the client 102b and/or a connection 104' with the server 106.
- the mobile computing device 102a or the client 102b may execute, operate or otherwise provide the application, which can be any type and/or form of software, program, or executable instructions such as any type and/or form of web browser, web-based client, client-server application, a thin-client computing client, an ActiveX control, or a Java applet, or any other type and/or form of executable instructions capable of executing on the mobile computing device 102.
- the application may be a server-based or a remote -based application executed on behalf of the mobile computing device 102a or the client 102b on a server 106.
- the server 106 may communicate with the mobile computing device 102a or the client 102b using any presentation layer protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Washington.
- ICA Independent Computing Architecture
- RDP Remote Desktop Protocol
- one or more resources 204 may execute in a virtual machine on the mobile computing device 102a as described with respect to FIG. 7A.
- the mobile computing device 102a transfers execution of the virtual machine to the client 102b and/or the server 106.
- the mobile computing device 102a transfers execution of a resource 204 in a virtual machine to the client 102b and/or the server 106.
- the mobile computing device 102a may transmit to the server 106 an identification of the attached client 102b.
- the server 106 may identify a virtual machine on the client 102b.
- the server 106 may disconnect a resource 204 executing in a virtual machine on the mobile computing device 102a and connect the resource 204 to the virtual machine on the client 102b. In these embodiments, the server 106 may transmit information regarding the state of execution of the resource to the virtual machine on the client 102b. In other embodiments, the server 106 may transfer execution of a resource executing in a virtual machine by disconnecting the virtual machine from the mobile computing device 102a and connecting the virtual machine to the client 102b.
- the server 106 provides access to a computing environment to the client 102b.
- a computing environment is the XenDesktop distributed by Citrix Inc. of Ft. Lauderdale, Florida.
- the output of the resource 204 that is executed on the server 106 for the mobile computing device 102a is displayed as a part of the computing environment on the client 102b through the connection 104".
- the output of the resource 204 executing on the server 106 for the mobile computing device 102a is displayed on the client 102b through the connection 104" but not as a part of the computing environment.
- a flow diagram depicts one embodiment of the steps taken in a method for executing resources for a mobile computing device remotely on a server and displaying output from the resources on a client device.
- the method includes the step of connecting, by an agent on a mobile computing device 102a, to a server 106 via a first session to execute and display output data of a resource 204 executed on the server 106 (step 810).
- the method also includes the step of connecting by the agent on the mobile computing device 102a with a client 102b via a second session (step 812).
- the method further includes the step of establishing, by an agent executing on the client 102b a third session with the server 106 (step 814).
- the server 106 displays a computing environment on the client 102b.
- the computing environment includes a display of output generated by an execution, on a remote server 106, of the resource 204.
- an agent or application program on the mobile computing device 102a connects to a server 106 to establish the connection 104' (step 810).
- a first session is established between the mobile computing device 102a and the server 106 for the mobile computing device 102a to access resources 204 on the server 106.
- the mobile computing device 102a executes a resource 204 on the server 106 via the first session.
- the output of the execution is displayed on the mobile computing device 102a.
- the output of the execution is transmitted to a client 102b.
- the agent on the mobile computing device 102a connects with a client 102b via a second session over connection 104 (step 812).
- the second session is set up by the agent that sets up the first session.
- the second session is set up by an agent separate from the agent that sets up the first session.
- the first session and the second session function according to a substantially same protocol.
- the first session and the second session function according to separate protocols.
- the connection between the mobile computing device 102a and the client 102b may be chosen from a plurality of possible connections as described with reference to FIG. 5A.
- an agent on the client 102b establishes a third session between the client 102b and the server 106 over the connection 104" (step 814).
- the server 106 transmits to the client 102b a computing environment over the connection 104".
- the computing environment is a XenDesktop distributed by Citrix Systems Inc. of Ft. Lauderdale, Florida.
- the output data generated by the execution of the resource 204 is incorporated into a display of the computing environment as described in greater detail in connection with FIGs. 10-11.
- the output data generated by the execution of the resource 204 is transmitted from the server 106 to the client computing device 102b over the connection 104".
- the output data generated by the execution of the resource 204 is displayed on the client 102b separately from the computing environment.
- the resource 204 is executed on the mobile computing device 102a and output data generated by the execution of the resource 204 is transmitted to the server 106 over connection 104' to be forwarded to the client 102b over the connection 104" for display.
- execution of the resource is transferred or relocated to the client 102b.
- output data generated by the resource 204 is displayed inside the computing environment.
- the execution of the resource 204 is transferred to the server 106.
- the output data from the execution on the server 106 is transmitted to the mobile computing device 102a and/or the client 102b.
- FIG. 9 A depicts an embodiment of a system for displaying and activating disconnected sessions between a mobile computing device and a server.
- a block diagram depicts a system 900 including the mobile computing device 102 and the server 106, although any type of client may be used in place of the mobile computing device 102.
- the mobile computing device 102 includes an input module 908, a client process 910, a network module 912, and a display module 924.
- the input module 908 provides an interface for a user of the mobile computing device 102 to interact with the mobile computing device 102, for example, to request the remote execution of an application 916 in a session 918 from the server 106.
- a server 106 for displaying and activating disconnected sessions includes a network module 920, a data store 926, and a server process 922.
- the network module 920 receives authentication information associated with a user of a mobile computing device 102.
- the data store 926 enumerates at least one disconnected session associated with the user.
- the server process 922 transmits, to the mobile computing device 102, a graphical representation of the at least one disconnected session associated with the user.
- the server process 922 receives a request to connect to the at least one disconnected session.
- the server process 922 connects the mobile computing device 102 to the requested at least one disconnected session enumerated in the data store 926.
- a session 918 is a process, operating on the server 106, that supports the execution of one or more applications 916.
- execution of a session 918 includes execution of a desktop application 916 from which the execution of other applications 916 can be initiated.
- a session 918 is a session between a mobile computing device 102 and a hosted desktop provided by a virtual machine executing on a server 106.
- a desktop session 918' executes within the session 918.
- one or more applications 916 execute within a desktop session 918' executing within the session 918.
- a session 918' may execute within a second session 918.
- a virtual machine on a server 106 executes a session 918 and a second session 918' - such as a desktop session within which a plurality of applications 916 executes - executes within the session 918.
- the session 918 includes an instance of the execution of a single application 916.
- the input module 908 is, for example, a graphical user interface that provides one or more icons or menu selections for a user to select.
- at least one icon or menu selection represents a specific application 916 available for remote execution.
- selecting an icon or menu selection initiates the transmittal of a log-on request to the server 106 for access to that application 916.
- an icon or menu selection does not represent any specific application 916, but instead represents a general server 106 log-on procedure.
- the input module 108 is non-graphical user interface. In this embodiment, the user can enter a command to send a log-on request to server 106.
- Entering a command can include typing a predefined set of characters or depressing a specified key sequence on an input device (e.g., a keyboard or keypad).
- the log-on request includes user-provided authentication information.
- the input module 908 accepts the input of the user-provided authentication information, which can include any type of authentication information, including, without limitation, any of user name-password/PIN combinations, voice samples, one-time passcodes, biometric data, digital certificates, or smart card data.
- the input module 908 is in communication with additional hardware peripherals to facilitate acceptance of user authentication information.
- Information associated with a user may be transmitted from the mobile computing device and received by the server.
- the input module 908 accepts authentication information and provides it to the client process 910.
- the client process 910 manages the mobile computing device-side functionality of the remotely-executing session.
- the client process 910 forwards user input including the authentication information and requests for termination or disconnection of sessions 918 to the server 106.
- the server 106 receives the information associated with a user of a mobile computing device 102 and authenticates the user responsive to the information.
- the client process 910 processes data received from the server 106, for example, by forwarding graphical output data generated in a session 918, or a graphical representation of output data from a session 918, to the display module 924.
- the network module 912 provides for communication between a mobile computing device 102 and the server 106.
- the network module 920 provides communication functionality for the server 106.
- the network module 912 sends user input, such as authentication information and requests for access to, disconnection from, or termination of sessions 918 executing on the server 106.
- the network module 912 also receives output data from the sessions 918 and forwards the output data to the client process 910.
- the network module 912 encapsulates user input into, and reconstitutes session output data from, a predetermined protocol for transmission to the server 106.
- the network module 912 encrypts outgoing transmissions and decrypts incoming transmissions.
- the network module 920 receives authentication information associated with a user of a mobile computing device 102. In another embodiment, the network module 920 receives communications from the mobile computing device 102 over one or more data networks or links 921. In still another embodiment, the network module 920 transmits output data to the mobile computing device 102. In still even another embodiment, the network module 920 encrypts outgoing communications and decrypts incoming communications. In one embodiment, the network module 920 of the server 106 communicates with the network module 912 of a mobile computing device 102 over a network 104. In another embodiment, incoming communications, once decrypted or retrieved from a protocol (if necessary), are forwarded to a session 918 or to the server process 922, as appropriate.
- the network module 920 encapsulates outgoing communications in a protocol for transmission and retrieves incoming data from transmissions received according to a communications protocol.
- network module 920 uses at least one communication protocol to encapsulate data.
- a first communication protocol capable of encapsulating secondary protocols used in communications between the mobile computing device and the host service, ensures that data is maintained during a disrupted network connection.
- data communicated between the mobile computing device and the host service is buffered. When, for example, a mobile computing device roams between different access points in the same network, the buffered data is maintained during the temporarily disrupted network connection.
- buffered data when a mobile computing device switches between networks (e.g., from a wired network to a wireless network) the buffered data is maintained during the temporarily disrupted connection to the host service.
- buffered data can also be maintained, for example, when the network connection is disrupted due to a failure of a server side component (e.g., a failure of a server side proxy), due to a time-out in the system, or due to other reasons.
- the network module 920 provides session persistence and reliability by encapsulating secondary protocols within the first communication protocol.
- At least one previously disconnected application session already associated with the user is identified responsive to the authentication information.
- at least one previously disconnected desktop session already associated with the user is identified responsive to the authentication information.
- the mobile computing device 102 receives output data generated by the at least one previously disconnected session.
- an application executing within the at least one previously disconnected session generates the output data.
- the mobile computing device 102 generates a graphical representation of the at least one previously disconnected session using the received output data.
- the mobile computing device 102 receives a graphical representation of the output data.
- the mobile computing device 102 may receive and display a graphical representation representing at least one previously disconnected session associated with the user.
- the display module 924 displays the graphical representation of the at least one previously disconnected session to a user of the mobile computing device 102.
- output data is displayed to the user.
- a graphical user interface is displayed to the user with the received graphical representation.
- the received graphical representation incorporates a graphical user interface element.
- a text-based representation of output data generated by a disconnected session is displayed to the user.
- the display module 924 displays the output data generated by an application 916 or a session 918 from a remotely-executing session 918.
- the display module 924 forwards output data received from the client process 910 directly to a display device, such as the display device 124 described above in connection with FIGs. IB and 1C, or other suitable form of display device.
- the received output data is encrypted, encapsulated in a protocol, or both.
- the display module 924 first manipulates the output data so that the output data can be interpreted by a standard display adapter such as a computer video card.
- a user may request to connect to at least one previously disconnected session, and the request may be transmitted to the server.
- a user of the mobile computing device 102 selects a graphical representation to which to connect.
- the mobile computing device 102 transmits the request to connect responsive to selection of a corresponding graphical representation selected by a user.
- Output data generated by the at least one previously disconnected session associated with the user is received in response to the transmitted request.
- the output data generated by the session, and by applications executing within the session, are received and displayed as described below.
- the server process 922 manages the execution and termination of sessions 918 and the connections and disconnections of those sessions 918 to the mobile computing device 102.
- the server process 922 can initiate new sessions 918, disconnect a mobile computing device 102 from a session 918, detect a mobile computing device 102 disconnection from a session 918, locate a session 918 from which a user has disconnected, locate a session 918 to which a user of a mobile computing device 102 is connected to a client computing device 102b, receive a request to connect to a disconnected session, and connect a user to a disconnected session 918.
- the sessions 918 are configured with a user's personal preferences and authorization privileges.
- the output transmitter 924 transmits output data from a session 918 to a mobile computing device 102 through the network module 920.
- the output transmitter 924 intercepts the output data generated in a session 918 and determines which mobile computing device 102 is connected to the session 918. If the session 918 is connected to a mobile computing device 102, the output transmitter 924 transmits the output data to the connected device via the network module 920. In one embodiment, if the session 918 is not connected to a mobile computing device 102, the output transmitter 924 discards the output data and waits to receive future output data.
- the output transmitter 924 disregards all further output data until the output transmitter 924 receives notification that a session 918 has connected to a mobile computing device 102. In some embodiments, as described in further detail below, if the sessions 918 are not connected to a mobile computing device 102, the output transmitter 924 transmits, to a mobile computing device 102, a graphical representation of at least one disconnected session associated with the user.
- the output transmitter 924 stores the data until the output transmitter 924 receives notification that the session 918 has connected to a mobile computing device 102. In another embodiment, the output transmitter 924 attempts to send output data to a mobile computing device 102 until the server process 922 notifies the output transmitter 924 that the mobile computing device 102 is disconnected from the server 106. In still another embodiment, the output transmitter 924 determines which of the mobile computing device 102 or other client, if any, the session 918 is connected to by consulting the data store 926. In yet another embodiment, the server process 922 determines to which of the mobile computing device 102 or other client, if any, the session 918 is connected by consulting the data store 926 and transmits the determination to the output transmitter 924.
- At least one disconnected session already associated with the user is identified in response to the information.
- the at least one disconnected session already associated with the user continues to execute at least one application.
- the at least one disconnected session already associated with the user continues to execute a desktop session.
- a server 106 identifies the at least one disconnected session.
- a first session executing on a first server is identified.
- a second session executing on a second server is identified.
- one of the first session and the second session is identified as a disconnected session.
- a second session executing on the first server is identified.
- one of the first session and the second session is identified as a disconnected session.
- the data store 926 includes information related to sessions 918 initiated by users.
- the data store 926 is stored in volatile or non-volatile memory.
- the data store 926 is distributed through multiple servers. Table 1 shows the data included in a portion of an illustrative data store 926.
- the illustrative data store 926 in Table 1 includes data associating each session 918 with the user that initiated the session 918, an identification of the mobile computing device 102 or the client computing device 102b, if any, from which the user is currently connected to the server 106, and the internet protocol (IP) address of the mobile computing device 102 or the client computing device 102b.
- IP internet protocol
- the illustrative data store 926 also includes the status of each session.
- a session 918 status can be, for example, "active” (meaning a user is connected to the session 918), or "disconnected” (meaning a user is not connected to the session 918).
- a session status can also be set to "executing-disconnected” (meaning the user has disconnected from the session 918, but applications in the session 918 are still executing), or “stalled-disconnected” (meaning the user is disconnected and applications 916 in the session 918 are not executing, but their operational state immediately prior to the disconnection has been stored).
- the data store 926 further stores information indicating the applications 916 that are executing within each session 918 and data indicating a process associated with each application 916. In one embodiment, where the server 106 resides in a server farm, the data store 926 also includes the data in the last two rows of Table 1 that indicate on which server in the server farm each application 916 is or was executing, and the IP address of that server. In other embodiments, the data store 926 includes a status indicator for each application 916 in each session 918.
- Session 9 is associated with User 1, who is currently using terminal 1.
- Terminal one's IP address is 172.16.2.50.
- the status of Session 1 is active, and in Session 1 , a word processing program is being executed on Server A as process number 1.
- Server A's IP address is 172.16.2.55.
- Session 2 in Table 1 is an example of a disconnected session 918.
- Session 2 is associated with User 2, but Session 2 is not connected to a mobile computing device 102.
- Session 9 includes a database program that is executing on Server A, at IP address 172.16.2.55 as process number 3.
- Session 3 is an example of how a user can interact with sessions 918 operating on different servers 106.
- Session 3 is associated with User 1, as is Session 1.
- Session 3 includes a spreadsheet program that is executing on Server B at IP address 172.16.2.56 as process number 2, whereas the session 918 included in App Session 1 is executing on Server A.
- the server 106 also includes a rules source 928.
- the rules source 928 stores rules specifying a policy applicable to a user requesting access to a session 918, or to an application 916 in a session 918.
- the rules stored in the rules source 928 are specified at least in part by the system administrator.
- a user specifies at least some of the rules stored in the rules source 928.
- the user-specified rule(s) may be referred to as preferences.
- the rules source 928 can be stored in volatile or non- volatile memory or distributed through multiple servers.
- a rule stored in the rule source 928 might require or forbid automatic connection to disconnected sessions 918.
- a rule might require or forbid automatic connection to active sessions 918 currently connected to a different mobile computing device 102.
- a rule might make a connection contingent on the mobile computing device 102 residing within a secure network.
- a rule might only allow connection to sessions 918 after receiving user approval.
- a rule might only allow connection for a predetermined time after disconnection.
- a rule may only allow connection to sessions 918 that provide access to specific applications 916.
- the authentication module 930 authenticates a user that attempts to log on to the server 106.
- the authentication module 930 receives user-provided authentication information transmitted from the mobile computing device 102.
- the authentication module 930 authenticates the user based on user-provided authentication information.
- the authentication module 930 transmits, responsive to a successful authentication, the results of the authentication process (e.g., allow or deny access, the user's system ID, mobile computing device ID, user access permissions, etc.) to the server process 922.
- the results of the authentication process e.g., allow or deny access, the user's system ID, mobile computing device ID, user access permissions, etc.
- affirmative user input can include selecting an "Exit" option from a menu, clicking on an icon, or entering a termination command into a command- line interface.
- the server process 922 in response to receiving a termination request, terminates the execution of the session 918 and of any application 916 within that session 918 is halted.
- data related to the session 918 is also removed from the data store 926.
- disconnection either intentional or unintentional does not result in termination of sessions 918.
- the application or applications operating in a session 918 are executing on the server 106 and a connection to the mobile computing device 102 is not usually necessary to continue execution of the applications 916.
- the applications 916 continue to execute while waiting for a user to connect to the session 918.
- the server process 922 upon disconnection of a user, stalls the execution of the applications 916 operating in the session 918. In this embodiment, the server process 922 halts further execution of the applications 916, and the server process 922 stores the operational state of the application 916 and any data the application 916 is processing.
- the server process 922 can selectively stall execution of specific applications 916 after a user disconnects. For example, and in one embodiment, the server continues execution of an application 916 for a fixed time period, and if a user fails to connect within that time period, the server process 922 stalls the application 916. In yet another of these embodiments, the server process 922 stalls specified sessions 918 that cannot continue executing without user input. In other embodiments, the server process 922 updates a data record associated with the application 916 or with the session 918 to include an identification of the status of the application or session.
- the server process 922 continues execution of the application 916 while the session 918 remains disconnected from the mobile computing device 102. In other embodiments, if the user of a mobile computing device 102 disconnects from the server 106 and then connects to the server 106 while operating the mobile computing device 102, the client computing device 102b, or a second client computing device 102n (not shown), the server process 922 can connect the user to one or more previously initiated, non-terminated session(s) 918 associated with the user, and reinitiate execution of any stalled applications 916. [0180] In one embodiment, the server process 922 detects a disconnection. A user can intentionally and manually instruct the server to disconnect a session 918 from the mobile computing device 102.
- sessions 918 provide a menu option for disconnection (as distinguished from termination above) that a user can select.
- the server process 922 can also detect an unintentional disconnection.
- the network module 920 of the server 106 informs the server process 922 when a predetermined number of data packets transmitted by the network module 920 to a mobile computing device 102 have not been acknowledged by the mobile computing device 102.
- the mobile computing device 102 periodically transmits a signal to the server 106 to confirm that a connection is still intact. If the server process 922 detects that a predetermined number of expected confirmation signals from a mobile computing device 102 have not arrived, the server process 922 determines that the mobile computing device 102 has disconnected. If the server process 922 detects that a user has disconnected from a session 918, either intentionally, or unintentionally, the entry in the data store 926 related to the disconnected session 918 is modified to reflect the disconnection.
- a graphical representation of the at least one disconnected session associated with the user may be displayed to the user.
- the server process 922 transmits, to the client 102, a graphical representation of the at least one disconnected session associated with the user.
- the graphical representation displays a representation of output data generated by an executing, disconnected session.
- the graphical representation displays a representation of output data generated by an application executing in a disconnected session.
- the server process 922 transmits, to a mobile computing device 102, output data generated by an executing, disconnected session.
- the mobile computing device 102 displays, to the user, a graphical representation of the output data.
- the server process 922 transmits, to the mobile computing device 102, a graphical representation comprising a user interface element.
- an output transmitter 924 transmits, to the mobile computing device 102, the output data generated by an executing, disconnected session.
- the server process 922 transmits, to the mobile computing device 102, a representation of a state of the at least one disconnected session associated with the user. In another embodiment, the server process 922 transmits, to the mobile computing device 102, a description of a state of the at least one disconnected session associated with the user. In still another embodiment, the server process 922 transmits, to the mobile computing device 102, a text-based description of a state of the at least one disconnected session associated with the user.
- the server process 922 transmits, to the mobile computing device 102 output data generated in the at least one disconnected session.
- the mobile computing device 102 generates the representation, graphical or text-based, of the output data received from the at least one disconnected session.
- the mobile computing device 102 displays the output data as described above.
- the mobile computing device 102 displays the representation of the output data in a window generated by an internet browser application.
- the mobile computing device 102 replaces an existing representation of output data with an updated version of the representation.
- the representation 950 (not shown) of the output data is graphical.
- the representation of the output data is text-based.
- a graphical representation 950 may depict the output of multiple applications executing within a disconnected session 918. In one of these embodiments, the graphical representation 950 depicts the output of multiple applications executing from a desktop application, which is provided by a session 918. In other embodiments, a plurality of graphical representations 950 is displayed to the user of the mobile computing device 102. In one of these embodiments, a first graphical representation 950 in the plurality of graphical representations depicts a first type of session 918 and a second graphical representation 950' (not shown) in the plurality of graphical representations depicts a second type of session 918.
- the first graphical representation 950 may depict the application-output data generated by a single application executing in a session 918, while the second graphical representation 950' may depict the output data generated by a plurality of applications executing within a hosted desktop session 918'.
- a single graphical representation 950 is depicted.
- the server process 922 identifies a change in the at least one disconnected session associated with the user and transmits a modified version of the at least one disconnected graphical representation of the at least one session displayed to the user, responsive to the identified change. In one of these embodiments, the server process 922 requests an identification of a change in the at least one disconnected session. In another of these embodiments, the server process 922 requests the identification of the change from a server executing the session 918. In still another of these embodiments, the server process 922 requests, after a time interval, an identification of a second change in the at least one session associated with the user.
- the server process 922 polls a server executing the session 918 for changes in the at least one application session. In yet another of these embodiments, the server process 922 updates, after a time interval, the graphical representation of the at least one application session displayed to the user, responsive to an identification of a second change in the at least one application session associated with the user.
- the server process 922 queries the data store 926 to determine that a previously connected session has become disconnected from a mobile computing device 102. In one of these embodiments, the server process 922 instructs the output transmitter 924 to transmit, to the mobile computing device 102, the output data generated by the session 918 prior to the disconnection.
- the server process 922 receives a request for the identification of the change. In one of these embodiments, the server process 922 receives the request from the mobile computing device 102. In another of these embodiments, the server process 922 transmits the identification of the change to the mobile computing device 102. In other embodiments, the server process 922 transmits, to the mobile computing device 102, an updated graphical representation of the at least one session, responsive to an identification of a change in the at least one session associated. In still other embodiments, the server process 922 transmits, to the mobile computing device 102, output data generated in the session 918. In one of these embodiments, the mobile computing device 102 generates the graphical representation of the output data and displays the graphical representation to the user.
- the mobile computing device 102 requests, from the server 106, the identification of a change in the at least one previously disconnected session associated with the user. In other embodiments, the mobile computing device 102 requests, from a server 106, which executes the at least one previously disconnected session, the identification of a change in the at least one previously disconnected session associated with the user. In still other embodiments, the mobile computing device 102 requests, after a time interval, an identification of a change in the at least one previously disconnected session associated with the user. In still even other embodiments, the mobile computing device 102 polls a server for an identification of a change in the at least one previously disconnected session associated with the user.
- the mobile computing device 102 receives an identification of a change in the at least one previously disconnected session. In other embodiments, the mobile computing device 102 receives output data generated by a session during a time interval. In still other embodiments, the mobile computing device 102 receives a modified version of the graphical representation. In yet other embodiments, the mobile computing device 102 displays a modified version of the graphical representation.
- FIGS. 9B and 9C a flow diagram depicts one embodiment of the steps taken in a method for displaying, activating, and providing remote access to disconnected sessions.
- the steps in the lefthand column may be taken, for example, by a client, such as a mobile computing device.
- the steps in the righthand column may be taken, for example, by a server.
- the method includes the step of transmitting information associated with a user, (step 940)
- the method includes the step of receiving information associated with a user, (step 942) At least one disconnected session already associated with the user is identified in response to the information, (step 944)
- a graphical representation of at least one disconnected session associated with the user is transmitted to the user, (step 946)
- a graphical representation representing at least one disconnected session associated with the user is received, (step 948)
- the graphical representation is displayed to the user, (step 950).
- a request to connect to the at least one previously disconnected session is transmitted, (step 952) A request from a user to connect to the at least one disconnected session is received, (step 954) A connection is established between a client computer operated by the user and the at least one disconnected session in the request, (step 956) Output data generated by the at least one disconnected session associated with the user is received in response to the transmitted request, (step 958)
- information associated with a user is transmitted, (step 940)
- the information is a log-on request.
- the information is a log-on request for access to a resource.
- the information is a general log-on request for a server.
- information may include authentication information.
- the authentication information may be any type of authentication information, include, for example, any username-password/PIN combinations, voice samples, one-time passcodes, biometric data, digital certificates, or smart card data.
- the information is accepted from a user by an input module, such as input module 208, and provided to a client process, such as client process 910.
- the information is forwarded to server by a client process, a network module, or a client process and network module in communication with one another.
- the information may be encapsulated in a predetermined protocol prior to transmission.
- the information may be encrypted prior to transmission.
- the information is transmitted over a data network or link.
- information associated with a user is received, (step 942)
- the information is received by a server.
- the information may be decrypted.
- the user is authenticated according to the received information.
- the information may be analyzed to authenticate the user according to any known method. For example, a hash function may be performed upon the information and the results compared with entries stored in a look-up table to determine if the user should be authenticated.
- the information associated with a user includes information that identifies the user.
- sessions may be stored in a database, such as a data store.
- Each session may include information about the session's status, e.g. active, disconnected, terminated, or any other known status.
- Each session may include information about a user corresponding to the session.
- the database may be searched for sessions that have a disconnected status and that correspond to the user.
- Information about the sessions, such as the sessions' server addresses and process numbers, may be stored in a memory or other buffer.
- a graphical representation of at least one disconnected session associated with the user is transmitted to the user, (step 946)
- the graphical representation is generated by at least one disconnected session.
- the graphical representation is generated by at least one application executing within the at least one disconnected session.
- the graphical representation may be a graphical-based or text-based representation of the at least one disconnected session.
- the graphical representation includes a description of each of the disconnected sessions.
- the graphical representation displays output data or a representation of output data generated by at least one disconnected session.
- the graphical representation displays output data or a representation of output data generated by an application executing in at least one disconnected session.
- the graphical representation includes a user interface element.
- the graphical representation of at least one disconnected session associated with the user is transmitted using an output transmitter.
- the output transmitter intercepts output data generated by at least one executing, disconnected session and transmits the output data.
- a graphical representation representing at least one disconnected session associated with the user is received, (step 948)
- the graphical representation is received by a client process, a network module, or a client process and network module in communication with one another.
- the graphical representation is manipulated into a format suitable for interpretation by a standard display adapter.
- the graphical representation is stored in a frame buffer.
- the graphical representation is displayed to the user, (step 950).
- the graphical representation is displayed on a display module 924.
- the graphical representation is displayed on a native display 201 of a mobile computing device 102.
- the graphical representation is displayed on an external display device 202.
- a client process transmits the graphical representation to a display module 924 for display on the device.
- a request to connect to the at least one previously disconnected session is transmitted, (step 952)
- the request may include the server address, process number, or any other information about the disconnected sessions the user wishes to connect to.
- the user may make a request by selecting the graphical representation, in part or in whole, of a disconnected session. For example, the user may select an icon corresponding to the disconnected session to request connection. In another example, the user may select a disconnected session from a list of enumerated disconnected sessions. In yet another example, the user may select a disconnected session from a menu of disconnected sessions.
- a request from a user to connect to at least one disconnected session is received (step 954) and a connection is established between a client computer operated by the user and at least one disconnected session in the request, (step 956)
- a server process may establish the connection.
- the connection may be established by forming a connection using the server address, the process number, and the client address.
- the connection may be established if the requested connection is consistent with a policy. For example, the connection may be established if the policy permits automatic re-connection of disconnected sessions. In another example, the connection may be established if the policy requires a client to reside on a secure network before a connection may be made, and the client does reside on a secure network.
- applications that may have stalled on the previously disconnected session may be re-executed.
- output data generated by the at least one disconnected session associated with the user is received in response to the transmitted request, (step 958)
- the output data may be received from storage, such as a buffer, in an output transmitter.
- the output data may be received from an output transmitter that intercepts output data generated by applications executing in the sessions and transmits the output data to a client.
- FIG. 1OA a block diagram depicts one embodiment of a system for generating a computing environment on a remote machine for display on a local machine.
- the system includes a first client 102a, a first agent 1002, a second client 102b, and a second agent 1010.
- the first agent 1002 executing on the first client 102a, generates a computing environment 1004, which includes a plurality of windows 1006a-1006n and provides access to i) a resource 1015 available to a user of the second client 102b and provided by the first client 102a, and ii) a resource 1020 provided by the second client 102b that is available to the user of the second client 102b.
- the first agent 1002 generates a proxy window 1008 for display in the computing environment 1004.
- the proxy window 1008 represents a local window 1014 on the second client 102b that displays output data generated by the resource 1020 provided by the second client 102b.
- the proxy window 1008 has a z-order entry in a z-order list associated with the plurality of windows 1006a-n in the computing environment 1004.
- the second agent 1010 on the second client 102b receives, from the first agent 1002, at least one entry in the z-order list associated with the computing environment 1004 and transmits, to the first agent 1002, an identification of a change to window attribute data associated with the window 1014 on the second client 102b for modification, by the first agent 1002, of the computing environment 1004.
- the first client 102a is a mobile computing device.
- the second client 102b is a client computing device.
- one of the first client 102a and the second client 102b may be a server 106.
- a resource executes on an mobile computing device 102a.
- the output data generated by an execution of a resource on the mobile computing device 102a is displayed on the second client which may be a client computing device.
- the output data generated by an execution on a client 102b of a resource 204 is integrated into a display on the mobile computing device.
- the first agent 1002 executing on the first client 102a, generates a computing environment 1004, which includes a plurality of windows 1006a-1006n and provides access to i) a resource 1015 available to a user of the second client 102b and provided by the first client 102a, and ii) a resource 1020 provided by the second client 102b that is available to the user of the second client 102b.
- the first client 102a is a server 106.
- the second client 102b is a client device 102, connecting to the server 106 to access one or more resource available to a user of the second client 102b.
- a resource 1015, 1020 comprises a program, an application, a document, a file, a plurality of applications, a plurality of files, an executable program file, a desktop environment, a computing environment, or other resource made available to a user of the second client 102b.
- the resource 1020 may be delivered to the second client 102b via a plurality of access methods including, but not limited to, conventional installation directly on the second client 102b, delivery to the second client 102b via a method for application streaming, delivery to the second client 102b of output data generated by an execution of the resource 1020 on a third machine 106' and communicated to the second client 102b via a presentation layer protocol, delivery to the second client 102b of output data generated by an execution of the resource 1020 via a virtual machine executing on a first client 102a, or execution from a removable storage device connected to the second client 102b, such as a USB device.
- the second client 102b transmits output data generated by the execution of the resource 1020 to another client machine 102'.
- a user of a second client 102b connects to a first client
- a user may invoke a local application from the remote desktop interface generated by the first client 102a and have the local application appear in the same desktop environment as the remote applications provided by the first client 102a.
- the user of the second client 102b would prefer an integrated desktop environment providing access to all of the resources available to the user, instead of separate desktop environments for resources provided by separate machines. For example, a user may find navigating between multiple graphical displays confusing and difficult to use productively. Or, a user may wish to use the data generated by one application provided by one machine in conjunction with another resource provided by a different machine.
- requests for execution of a resource, windowing moves, application minimize/maximize, and termination of executing resources may be controlled by interacting with a computing environment that integrates the display of the remote resources and of the local resources.
- an application or other resource accessible via an integrated desktop environment 1004 - both those generated on the second client 102b and those generated on the first client 102a - is shown on the computing environment 1004 as if it were executing on, or executable from, the computing environment.
- a resource may also appear in a listing of available resources provided in a Start Menu, a shortcut may be provided on the desktop or the Quick Launch menu, and the resources can be launched, selected and interacted with in the same way as an application provided by the first client 102a.
- a single computing environment 1004 is displayed. In one of these embodiments, the computing environment 1004 is displayed as a full-screen desktop.
- a plurality of computing environments 1004 is displayed. In one of these embodiments, one or more of the computing environments are displayed in non- full-screen mode on one or more display devices 124. In another of these embodiments, the computing environments are displayed in full-screen mode on individual display devices. In still another of these embodiments, one or more of the computing environments are displayed in full-screen mode on one or more display devices 124. In yet another of the embodiments, a resource provided by the second client 102b may be integrated with one of the plurality of computing environments 1004.
- the first agent 1002 generates a computing environment 1004 including a plurality of windows 1006a-1006n.
- a window 1006a in the plurality of windows 1006 displays the output data generated by an execution of a resource provided by the first client 102a.
- a window 1006b in the plurality of windows 1006a-1006n displays the output data generated by an execution of a resource provided by a third machine 106', as discussed in further detail below, in connection with FIGs. 12A and 12B.
- a window 1006c in the plurality of windows 1006a-1006n depicts a taskbar from a desktop environment.
- a window 1006d represents a menu, such as a Start menu or a context-specific menu associated with an application.
- a window 1006e in the plurality of windows 1006a-n has a z-order entry such that it is displayed beneath the other windows in the plurality of windows 1006a-n and depicts a desktop.
- the first agent 1002 transmits, to the second agent 1010, window attribute data associated with each of the plurality of windows 1006a-n.
- the second agent 1010 displays a local version of the plurality of windows 1006a-n having window attribute data corresponding to the window attribute data received from the first agent 1002.
- the graphical display and the window attribute data for the plurality of windows 1006a-n and for the computing environment 1004 are transmitted to the second agent 1010 for generation of a local version of the computing environment 1004.
- the second agent 1010 displays, to a user of the second client 102b, a local version 1012 of the plurality of windows 1006 and the computing environment 1004.
- FIG. 1OB a block diagram depicts one embodiment of a system in which the first agent 1002 integrates resources from various sources into a computing environment 1004.
- the first agent 1002 may receive the identification of the resources available to a user of the second client 102b from a plurality of servers 106.
- the first agent 1002 receives an identification of the available resources from machines 106', 106", 106'", and 106"".
- the first client 102a receives an identification of the available resources from server 106', receives the output data generated by an execution of the resource on server 106' and transmits the output data so generated to second client 102b.
- the first agent 1002 receives only the identification of the available resources from the machine 106", and the machine 106" transmits the output data generated by an execution of the resource to the second client 102b.
- the first agent 1002 receives, from a machine
- the first agent 1002 receives, from the second agent 1010, an identification of available resources from the machines 106'" and 106".
- the second client 102b receives, via a method for resource streaming, the available resource from the server 106'".
- the second client 102b executes the resource locally and the graphical output data is integrated with the computing environment 1004.
- the first client 102a receives an identification of the available resources from server 106"", and receives via a method for resource streaming, the available resource from machine 106"” and transmits the output data generated by an execution of the resource to the second client 102b.
- the first agent 1002 receives, from the second agent 1010, an identification of a resource 1020 available to the user of the second client 102b for integration into the computing environment 1004.
- executing a resource provided by the second client 102b may allow a user to leverage a characteristic of the second client 102b that is not provided by the first client 102a; for example, the second client 102b may have more appropriate processing ability, graphics functionality, bandwidth, or license rights for accessing the resource than the first client 102a.
- the first client 102a may lack the ability to execute the resource; for example, because only the user of the second client 102b owns an application or owns a resource, such as an operating system, required to execute the application.
- the resource 1020 is a resource such as an application or desktop environment installed on the second client 102b via a method for streaming the application to the second client 102b from a third machine 106'.
- the resource 1020 is a resource such as an application or desktop environment whose output data the second client 102b receives via a presentation layer protocol communication with a third machine 106'.
- the first agent 1002 receives an identification of a conferencing or Internet-communication application; a user peripheral, such as a media player, a digital camera or a web camera; or a processor-intense, data-intense, or graphics-intense application, such as a media-editing application or a computer-aided design application.
- the first agent 1002 incorporates the identification of the resource 1020 into the computing environment 1004.
- the first agent 1002 adds a program execution shortcut to a menu for requesting execution of resources, the program execution shortcut corresponding to the identified resource 1020.
- the first agent 1002 generates a graphical representation, such as an icon, associated with the resource 1020 for display in the computing environment 1004.
- the first agent 1002 may alternatively receive the graphical representation from the second agent 1010 for display in the computing environment 1004.
- a user requests execution of a resource 1020 by interacting with a program execution shortcut or a graphical representation displayed in the local version of the computing environment 1004.
- the first agent 1002 receives the request for execution of a resource 1020, responsive to the user interaction with a program execution shortcut or a graphical representation, and sends the request for execution to the second agent 1010.
- a user requests execution of a resource by interacting with the local display 1012 of the computing environment 1004; for example, by selecting a graphical representation of a file, document, uniform resource locator, or other resource, displayed by the desktop environment.
- the first agent 1002 generates a proxy window 1008 for integration into the computing environment 1004.
- the first agent 1002 is referred to as a proxy window management component 1002.
- the proxy window management component 1002 is responsible for making the proxy window 1008 conform to client-initiated changes to window attribute data. In still another embodiment, the proxy window management component 1002 is responsible for monitoring any desktop-initiated changes to window attribute data and communicating them to the second agent 1010, which applies them to the corresponding local application window 1014.
- the proxy window management component 1002 provides additional functionality.
- such functionality may include a MICROSOFT Active Accessibility (MSAA) hook and other monitoring mechanisms to detect new application windows and window attribute data changes initiated on the desktop.
- the proxy window management component 1002 may further include support for launching published applications.
- the proxy window management component 1002 may send updates initiated from the remote machine 106 [desktop] regarding changes to window attribute data to the second agent 1010 where the updates will be applied to the local window 1014, and to the local displays of the plurality of windows 1006a-n.
- the proxy management component 1002 may be enabled to apply client-initiated updates to window attribute data associated with proxy window 1008 and to the plurality of data objects 1006a-n. In yet another of these embodiments, the proxy management component 1002 may remove proxy windows when the client connection disappears.
- the first agent 1002 and the second agent 1010 include a hooking component for intercepting window-related messages.
- an agent executing on a machine 102 or 106 on which the MICROSOFT WINDOWS operating system executes may use a version of the MICROSOFT Active Accessibility hook to monitor relevant window events.
- an agent may include a MICROSOFT Computer-based Training (CBT) window hook, or other hooking mechanisms, to intercept and monitor window events.
- CBT Computer-based Training
- the proxy window 1008 represents a window 1014 on the second client 102b that displays output data generated by the resource 1020 provided by the second machine.
- the proxy window 1008 has a z-order entry in a z-order list associated with the plurality of windows 1006a-n in the computing environment 1004.
- the proxy window 1008 has a traditional window visual state - position, size, Z-order, focus state, minimized/normal/maximized state - and a taskbar button, and appears in the remote desktop AIt-TAB selection dialog.
- the proxy window 1008 responds to requests, initiated by the remote desktop, to change visual state or to invoke its taskbar menu.
- the visual state of the local application window 1014 is synchronized with the visual state of the corresponding proxy window 1008.
- the proxy window 1008 can pass visual state change requests to the second agent 1010, via the first agent 1002, without needing to apply them to itself.
- the first agent 1002 does not transmit the proxy window
- the proxy window 1008 does not need to be painted on the computing environment 1004, or can be painted very efficiently e.g. in a uniform color.
- the first agent 1002 is aware of the clipping regions associated with the proxy window 1008 and does not send window attribute data for those regions.
- the second agent 1010 on the second client 102b receives, from the first agent
- the second agent 1010 transmits, to the first agent 1002, an identification of a change to window attribute data associated with the window 1014 on the second client 102b for modification, by the first agent 1002, of the computing environment 1004.
- the second agent 1010 is an integrated window management component 1010 executing on the second client 102b that enumerates, monitors, and manages local windows 1014 that are integrated into the remote desktop 1004.
- the integrated window management component 1010 manages all local windows 1014 that are integrated into the remote desktop 1004.
- the integrated window management component 1010 communicates window existence and client-initiated visual state changes over a virtual channel to the first agent on the first client 102a.
- the integrated window management component includes
- the proxy window management component 1002 may send updates initiated from the client 102a regarding changes to window attribute data to the first agent 1002 where the updates will be applied to the proxy window 1008, and to the local displays of the plurality of windows 1006a-n.
- the proxy management component 1002 may be enabled to apply desktop-initiated updates to window attribute data associated with local window 1014 and to the local displays of the plurality of windows 1006a-n.
- a first client 102a and a second client 102b communicate using a presentation layer protocol, for example, by communicating via the ICA protocol, or the RDP protocol.
- the first agent 1002 and the second agent 1010 exchange graphical data, i.e., the data actually displayed in each window on the desktop environment, via a first virtual channel.
- the first virtual channel is an ICA virtual channel.
- information about window positioning, window size, z-ordering of window and other such information is communicated between the first client 102a and the second client 102b via a second virtual channel.
- the second virtual channel is an ICA virtual channel.
- the first agent 1002 and the second agent 1010 exchange window attribute data. In one of these embodiments, the first agent 1002 and the second agent 1010 also exchange graphical data. In another of these embodiments, a third agent executing on the first client 102a and the second agent 1010 exchange graphical data. In still another of these embodiments, a third agent executing on the first client 102a and a fourth agent executing on the second client 102b exchange graphical data.
- a flow diagram depicts one embodiment of the steps taken in a method for generating a computing environment on a remote machine for display on a local machine.
- a method for generating a computing environment on the remote machine for display on the local machine includes the step of monitoring, by a first agent executing on a first machine, a computing environment for display on a second machine, the computing environment comprising a plurality of windows and providing access to i) a resource provided by the first machine that is available to a user of the second machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1102).
- the first agent generates a proxy window representing a window on the second machine that displays output data generated by the resources provided by the second machine, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1104).
- the first agent transmits, to a second agent on the second machine, at least one entry in the z-order list associated with a window in the plurality of windows in the computing environment (step 1106).
- the first agent receives an identification of a change to window attribute data associated with the window on the second machine (step 1108).
- the first agent modifies the computing environment, responsive to the identification of the change (step 1110).
- a first agent executing on a first machine monitors a computing environment for display on a second machine, the computing environment comprising a plurality of windows and providing access to i) a resource provided by the first machine that is available to a user of the second machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1102).
- the first agent 1002 monitors the generation and modification of the computing environment 1004 and of a plurality of windows 1006a-n providing access to a plurality of resources 1015 executing on remote servers, such as the first client 102a, and to at least one resource 1020 provided by the second client 102b.
- the first agent generates a proxy window representing a window on the second machine that displays output data generated by the resources provided by the second machine, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1104).
- the local window 1014 is reflected into the remote desktop window list using the proxy window 1008, which has dimensions, position, Z-order value and focus state in the computing environment 1004. These window attributes are kept in synch between the second client 102b and the first client 102a.
- the second agent 1010 receives an identification of a region in the computing environment corresponding to a region on the local display 1012 associated with a local resource 1020. In one of these embodiments, the second agent 1010 receives the identification from the first agent 1002. In another of these embodiments, described in additional detail below, the second agent 1010 identifies the region responsive to information received from the first agent 1002. In still another of these embodiments, as described in additional detail below, the second agent 1010 identifies the region responsive to identifying a color key in a communication from the first agent 1002.
- the local display 1012 is a single window that displays the graphical output data of the computing environment 1004, including the window output for the plurality of windows 1006a-n.
- an individual window 1006a does not need to have a corresponding seamless window on the client.
- the local display 1012 is maintained as the topmost window, thus keeping local applications underneath the local display 1012 in the client Z-order.
- a region of local display 1012 is shown as transparent to allow the correct parts of a local window 1014 to show through the local display 1012, as if the local window 1014 were on the computing environment 1004.
- the proxy window 1008 is displayed on a region of the computing environment 1004 corresponding to the region of local display 1012 which is to be transparently displayed.
- the transparent region is referred to as a clipping region.
- the first agent 1002 uses the proxy window 1008 to identify a clipping region. In one of these embodiments, the first agent 1002 intercepts a paint request on the computing environment 1004. In another of these embodiments, the first agent 1002 transmits, to the second agent 1010, an identification of a clipping region.
- the second agent 1010 identifies the areas of local display 1012 to be displayed by the local resource 1020. In one of these embodiments, the second agent 1010 identifies the areas responsive to information received from the first agent 1002. In another of these embodiments, the second agent 1010 identifies a key received from the first agent 1002, such as a color key or an identifiable pattern or tag identifying a clipping region.
- the second agent 1010 ensures that the resource
- the second agent 1010 paints output data to the appropriate local window 1014, which is located in a region corresponding to the clipping region on the computing environment.
- the second agent 1010 in communication with a window management component, ensures that the local display 1012 is kept topmost and displays the graphical output data associated with the computing environment 1004, other than the output data that would be displayed in a clipping region.
- the second agent 1010 instructs a window management component to regard an instruction to paint a particular color or pattern to a region (for example, to a region of the local display 1012) as an instruction to keep the region transparent.
- This color key may include an alpha value (e.g., a 32-bit ⁇ R,G,B,alpha ⁇ rather than a 104-bit ⁇ R,G,B ⁇ value) which can be used to distinguish it from all non-transparent colors (for resources that don't use partial transparency).
- an alpha value e.g., a 32-bit ⁇ R,G,B,alpha ⁇ rather than a 104-bit ⁇ R,G,B ⁇ value
- the agent 1002 paints a clipping region for a proxy window 1008 by responding to a paint request from a window management component to the proxy window by painting the window in a color key.
- the agent 1010 associates a paint request in a color key with the appropriate local application window.
- the agent 1010 also needs to use paint requests in a different color for a (part of) an existing clipping region to remove the painted region from the associated clipping region.
- the first agent 1002 may paint a clipping region in the color key rather than send a z-order entry to the second agent 1010.
- the first agent 1002 sends at least one z-order entry for a proxy window 1008 associated with a local window 1014.
- the second agent 1010 ensures that a local window 1014 has the same relative Z-order relationship on the client as the corresponding proxy window 1008 does on the first client 102a, i.e.
- proxy window 1008 is below proxy window 1008' on the first client 102a (regardless of how many server windows there are, and whether they are above or below either A or B), then the second agent 1010 ensures that a local window 1014 is below a local window 1014'.
- the windows underneath the local display 1012 show through wherever the local display 1012 is painted in the color key.
- the second agent 1010 maintains the local windows in the appropriate relative Z-order so that they will paint correctly and a window that's above another will correctly occlude the other even though both occupy the entire clipping region.
- the second agent 1010 also ensures that user input will be directed to the correct window - for example a mouse click on a transparent region will be sent to the underlying window, not the local display 1012.
- the first agent 1002 transmits to the second agent 1010 an identification of a clipping region for each proxy window 1008 in a computing environment 1004.
- the second agent 1010 directs the local resource 1020 to paint output data to the region of local display 1012 corresponding to the clipping region. In another of these embodiments, directing the local resource 1020 to paint output data to a region of the local display 1012 avoids the need for transparency.
- the first agent 1002 identifies a clipping region on the computing environment 1004 that corresponds to a region displaying local window 1014. In still even another of these embodiments, the first agent 1002 uses a color key or identifiable pattern or tag as described above to tag a clipping region. In one of these embodiments, the agent 1002 or the agent 1010 associates a paint request in a color key with the appropriate local application window.
- the agent 1002 or the agent 1010 responds to paint requests in a different color for a (part of) an existing clipping region for an application window by removing the newly painted region from the associated clipping region.
- the agent 1002 or agent 1010 associates a different color key for each local window.
- the second agent 1010 identifies a clipping region responsive to information received from the first agent 1002.
- the first agent 1002 may paint a clipping region in a color key rather than send a z-order entry to the second agent 1010.
- the first agent 1002 may send an identification of a clipping region to the second agent 1010 rather than send a z-order entry to the second agent 1010.
- the first agent 1002 may send information other than a z-order entry to agent 1010 that allows agent 1010 to identify a clipping region rather than send a z-order entry to the second agent 1010.
- the first agent 1002 does not send a z-order entry to the second agent 1010 at all.
- the second agent 1010 ensures that the local display
- the local display 1012 is used to show all of the remote desktop graphics output including areas belonging to local window 1014. In another of these embodiments, local windows never show through. In still another of these embodiments, when the second agent 1010 receives output data not containing the color key for a portion of a clipping region associated with a local window, the second agent 1010 removes the output region from the clipping region for that window.
- the second agent 1010 directs the local resource
- the second agent 1010 to paint the output data it generates to a region in the local display 1012 corresponding to the region identified by the first agent 1002 as a clipping region.
- the second agent 1010 in communication with a window management component, sends the local resource 1020 a WM PAINT message for a display context that references a clipping region on the local display 1012.
- the second agent 1010 sends the messages for any updated portions of a local window's associated clipping region whenever portions of the clipping region change.
- the second agent 1010 sends the paint messages in a periodic refresh loop as local windows are always covered by the local display 1012 and can no longer asynchronously update their regions that are visible on the client display.
- the second agent 1010 ensures that user input in clipping regions (including mouse clicks and, where appropriate, keyboard events) are redirected by the local display 1012 to the corresponding local application window. This means detecting mouse events in clipping regions, determining which local window is associated with the mouse event coordinates and sending the event to that window at those coordinates.
- a clipping region is identified.
- an agent 1002 or 1010 calculates a clipping region responsive to information associated with the computing environment, including a list of windows, their Z- order, size & position (and, for non-rectangular windows, their shape).
- the agent 1002 or 1010 acquires the information from one or more sources including, but not limited to, the seamless virtual channel, data used by the Headless Client, subclassing windows, Microsoft Active Accessibility (e.g. using WinEvents and MSAA hooking) and by hooking various Windows APIs.
- the agent 1002 or 1010 identifies a clipping region, or portion of a clipping region, by hooking or responding to window messages (including WM_PAINT, WM_NCPAINT, WM ERASEBKGND) which allow the agent 1002 or 1010 to see requests by Windows for a window to perform a paint operation within a region. Each time the agent 1002 or 1010 identifies one of these messages, the corresponding region is added to the clipping region for the associated window. The agent 1002 or 1010 uses knowledge of which windows receive which paint requests to maintain a record of which regions of the screen "belong" to each application window.
- window messages including WM_PAINT, WM_NCPAINT, WM ERASEBKGND
- the agent 1002 or 1010 Each time the agent 1002 or 1010 intercepts a paint request for a different window covering (all or part of) an existing clipping region for another window, that part is removed from that window's clipping region.
- the agent 1002 or 1010 can also directly retrieve the clipping region using Windows API functions such as GetRandomRgn() which allows the agent 1002 or 1010 to directly interrogate a window for clipping regions.
- the agents 1002 and 1010 communicate as described above to exchange information required by the second agent 1010 to modify the local display 1012 or local windows 1014 or local windows corresponding to the plurality of windows 1006a-n, as described below in connection with FIG. 1 IB.
- the first agent transmits, to a second agent on the second machine, at least one entry in the z-order list associated with a window in the plurality of windows in the computing environment (step 1106).
- the first agent 1002 transmits to the second agent 1010 the entire z-order list.
- the first agent 1002 transmits to the second agent 1010 a partial z-order list.
- the first agent 1002 transmits the at least one entry in the z-order list to the second agent 1010 via a virtual channel coupled to the computing environment 1004.
- the first agent 1002 transmits, to the second agent 1010, via a virtual channel conveying window attribute data associated with the proxy window 1008, messages directing the modification of a corresponding local window 1014, which displays, in accordance with the window attribute data conveyed by the virtual channel, graphical output data generated by a resource 1020 executing on the second client 102b.
- the second client 102b displays a local version of each of the plurality of windows 1006a-n.
- the second agent 1010 directs a window management component on the second client 102b to assign each of the locally- displayed windows 1006a-n a z-order entry in a z-order list of windows displayed on the second client 102b.
- Each of the assigned z-order entries for each of the locally-displayed windows 1006a-n result in the display of the locally-displays windows 1006a-n above a local window associated with a local application on the second client 102b other than the local display of a local application displayed in local window 1014.
- the local window is kept below the local display 1012 of the computing environment 1004 in the client- side Z-order - that is, it is kept below each of the windows in the plurality of windows 1006a-n and below the local window 1014, which is integrated with the plurality of windows 1006a-n.
- the plurality of windows 1006a-n and the local window 1014 are displayed on the second client 102b according to the z-order list maintained on the first client 102a.
- the first agent receives an identification of a change to window attribute data associated with the window on the second machine (step 1108).
- the first agent 1002 receives the identification of the change via a virtual channel.
- the first agent 1002 receives a message that a user minimized the local window 1014.
- the first agent 1002 receives a message that a user restored a minimized local window 1014.
- the first agent 1002 receives a message that a user changed a size or position of the local window 1014.
- the first agent 1002 receives a message that a user maximized a local window 1014.
- the first agent modifies the computing environment, responsive to the identification of the change (step 1110).
- the first agent 1002 directs a window management component of an operating system executing on the first client 102a to modify a window displayed in the computing environment.
- the first agent 1002 sends a message to the second agent 1010 to restore the local window 1014.
- the second agent 1010 sends a message to the resource 1020 to restore the minimized local window 1014.
- the first agent 1002 sends a message to a window management component of an operating system executing on the first client 102a to restore the proxy window 1008.
- the first agent 1002 sends a message to the second agent 1010 to terminate the execution of a local resource 1020 when the first agent 1002 terminates the generation and display of the computing environment 1004. In some embodiments, the first agent 1002 sends a message to the second agent 1010 to terminate an operating system executing on the second client 102b. In one of these embodiments, the first agent 1002 sends a message to the second agent 1010 to shut down the second client 102b.
- the second agent 1010 transmits a message to the first agent 1002 instructing the first agent 1002 to direct a window management component to remove focus from the proxy window 1008 and to focus on a window 1006a in the plurality of windows 1006a-n.
- the second agent 1010 allows the resource 1020 to paint inside its window, resulting in a completely transparent view of local and remote resources.
- FIG. 1 IB a flow diagram depicts one embodiment of the steps taken in a method for displaying, on the local machine, a desktop environment that is generated by the remote machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by the remote machine.
- the method includes the step of displaying, by an agent executing on a local machine, a computing environment generated by a remote machine and providing access to i) a remote resource available to a user of the local machine, and to ii) a local resource available to the user of the local machine (step 1120).
- the agent receives at least one entry in a z-order list including a z-order entry of at least one window in the computing environment and including the z-order entry of a proxy window corresponding to a local window on the local machine displaying output data generated by the local resource (step 1122).
- the agent modifies window attribute data for the displayed computing environment, responsive to the received z-order entry (step 1124).
- the agent executing on the local machine transmits, to an agent on the remote machine, an identification of a change to a z-order entry of a local window on the local machine displaying output data generated by the local resource (step 1126).
- the agent on the remote machine modifies a z-order entry for a proxy window representing the local window in a z-order list on the remote machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1128).
- An agent executing on a local machine displays a computing environment generated by a remote machine and providing access to i) a remote resource available to a user of the local machine, and to ii) a local resource available to the user of the local machine (step 1120).
- the agent 1010 receives graphical data and window attribute data from the agent 1002 and displays the graphical data in a window formed according to the window attribute data.
- the agent 1010 receives graphical data and corresponding window attribute data for a plurality of windows forming a computing environment and displays the graphical data in a plurality of windows formed according to the window attribute data.
- the first agent 1002 and the second agent 1010 exchange window attribute data.
- the first agent 1002 and the second agent 1010 also exchange graphical data.
- a third agent executing on the first client 102a and the second agent 1010 exchange graphical data.
- a third agent executing on the first client 102a and a fourth agent executing on the second client 102b exchange graphical data.
- the first agent 1002 on the first client 102a and a fourth agent executing on the second client 102b exchange graphical data.
- the agent receives at least one entry in a z-order list including a z-order entry of at least one window in the computing environment and including the z-order entry of a proxy window corresponding to a local window on the local machine displaying output data generated by the local resource (step 1122).
- the agent 1010 displays a local window 1014 displaying output data, generated by an execution of the resource 1020, and a plurality of windows 1006a-n formed in accordance with received window attribute data, in accordance with the at least one entry in the z-order.
- the agent 1010 monitors the local display of the plurality of windows 1006a-n and of the local window 1014.
- the agent 1010 in communication with a window management component of an operating system executing on the second client 102b, manages the local display of the plurality of windows 1006a-n and of the local window.
- the agent modifies window attribute data for the displayed computing environment, responsive to the received z-order entry (step 1124).
- the agent modifies a z-order entry for a local version of a window 1006a in the plurality of windows 1006a-n.
- the agent modifies window attribute data for a local version of a window 1006a in the plurality of windows 1006a-n.
- the agent resizes the local version of the window 1006a in the plurality of windows 1006a-n.
- the agent repositions the local version of the window 1006a in the plurality of windows 1006a-n.
- the agent may modify window attribute data of the local window 1014 responsive to the received z-order entry sent as a result of a window management component implementing a window cascade on the server.
- the second agent 1010 receives an identification of a clipping region in the computing environment corresponding to a region on the local display 1012 associated with a local resource 1020. In one of these embodiments, the second agent 1010 receives the identification from the first agent 1002. In another of these embodiments, described in additional detail above, the second agent 1010 identifies the region responsive to information received from the first agent 1002. In still another of these embodiments, as described in additional detail above, the second agent 1010 identifies the region responsive to identifying a color key in a communication from the first agent 1002.
- the agent executing on the local machine transmits, to an agent on the remote machine, an identification of a change to a z-order entry of a local window on the local machine displaying output data generated by the local resource (step 1126).
- the agent 1010 transmits a message to the agent 1002 to make a corresponding change to the proxy window 1008 and to its z-order entry in the z-order list ordering the plurality of windows 1006a-n and the proxy window 1008.
- the agent on the remote machine modifies a z-order entry for a proxy window representing the local window in a z-order list on the remote machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1128).
- the agent 1010 assigns the local window 1014 a z-order entry synchronized with the z-order entry of the proxy window 1008, which has a z-order entry in a z-order list including z-order entries for a plurality of windows 1006a-n.
- generating a z-order list including entries both for windows generated by a resource executing on a server 106 and for windows generated by a resource executing on a client 102 allows for integrated and overlapping displays of local and remote windows.
- FIG. 12A a block diagram depicts one embodiment of a system for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by a second remote machine.
- the system includes a first client 102a, a first agent 1002, a second client 102b, a second agent 1010, and a remote machine 106'.
- the first agent 1002 receives an identification of a remote resource 1210 available to a user of the second client 102b and provided by the remote machine 106'.
- the first agent 1002 executing on the first client 102a, generates a computing environment 1004, which includes a plurality of windows 1006a-206n and provides access to i) a resource 1210 available to a user of the second client 102b and provided by the remote machine 106', and ii) a resource 1020 provided by the second client 102b that is available to the user of the second client 102b.
- the first agent 1002 receives a request for execution of the remote resource 1210 provided by the remote machine 106'.
- the first agent 1002 directs the second agent 1010 on the second client 102b to request execution of the remote resource 1210 provided by the remote machine 106'.
- the second agent 1010 transmits to the first agent 1002 an identification of a change to a z-order entry of a local window 1014 on the second client 102b, the local window 1014 displaying output data generated by the remote resource 1210 provided by the remote machine 106'.
- the first agent 1002 modifies a z-order entry for a proxy window 1008 representing the local window 1014 in a z-order list on the first client 102a, responsive to the identification of the change, the proxy window 1008 having a z-order entry in a z-order list associated with the plurality of windows 1006 in the computing environment 1004.
- the second agent 1010 provides access to the computing environment 1004.
- a second client agent 1010' is executed on the second client 102b to instead establish the connection between the second client 102b and the remote machine 106'.
- the second client agent 1010' looks like a client-side application that is accordingly integrated into the remote desktop by the second agent 1010 and the first agent 1002 using the methods described above in connection with FIGs. 1OA, 1OB, 1 IA, and 1 IB.
- the second client agent 1010' is an RDP client.
- the second client agent 1010' is an ICA client.
- a screen shot depicts one embodiment of a system for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by a second remote machine.
- two presentation layer protocol sessions are executed on the second client 102b.
- Session 1220 is an ICA session providing access to a desktop environment 1004.
- Session 1230 is an ICA session providing access to a resource (in FIG. 12B, the resource is a word processing program, MICROSOFT WORD).
- the WORD application window which is local window 1014, has been merged with the presentation of the desktop session, which is represented by the computing environment 1004. If the first client 102a had established the session, the first client 102a would have received the output data and would have then forwarded the output data to the second client 102b for display. Instead, in this embodiment, the second client 102b established the presentation layer protocol session required to access WORD, instead of the first client 102a establishing a presentation layer protocol session, and the second client 102b receives the output data directly from the remote machine 106.
- a flow diagram depicts one embodiment of the steps taken in a method for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by a second remote machine, includes the step of receiving, by a first agent executing on a first machine, an identification of a remote resource available to a user of a second machine, the remote resource provided by a third machine (step 1302).
- the first agent monitors a computing environment on the first machine, the computing environment comprising a plurality of windows and providing access to i) the remote resource provided by the third machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1304).
- the first agent receives a request for execution of the remote resource provided by the third machine (step 1306).
- the first agent directs a second agent on the second machine to request execution of the remote resource provided by the third machine (step 1308).
- the second agent transmits, to the first agent, an identification of a change to a z- order entry of a local window on the second machine displaying output data generated by the remote resource provided by the third machine (step 1310).
- the first agent modifies a z- order entry for a proxy window representing the local window in a z-order list on the first machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1312).
- a first agent executing on a first machine receives an identification of a remote resource available to a user of a second machine, the remote resource provided by a third machine (step 1302).
- the first agent 1002 receives the identification from the remote machine 106'.
- the first agent 1002 receives the identification from the second client 102b.
- the first agent 1002 receives the identification from an agent on one of the remote machine 106' and the second client 102b.
- the first agent 1002 receives the identification from a fourth machine 106'".
- a resource 1210 comprises a program, an application, a document, a file, a plurality of applications, a plurality of files, an executable program file, a desktop environment, a computing environment, or other resource made available to a user of the second client 102b.
- the resource 1210 may be delivered to the remote machine 106 via a plurality of access methods including, but not limited to, conventional installation directly on the remote machine 106, delivery to the remote machine 106 via a method for application streaming, or execution from a removable storage device connected to the remote machine 106, such as a USB device.
- the first agent monitors a computing environment on the first machine, the computing environment comprising a plurality of windows and providing access to i) the remote resource provided by the third machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1304).
- the first agent monitors the computing environment 1004 as described above in connection with FIGs. 1OA, 1OB, HA, and HB.
- the first agent receives a request for execution of the remote resource provided by the third machine (step 1306).
- the first agent 1002 receives the request via a communications channel, such as an ICA channel, between the first agent 1002 and the second agent 1010.
- the first agent 1002 receives the request when a user of the second client 102b selects a graphical representation of the remote resource 1210 depicted in the local display 1012 of the computing environment 1004.
- the first agent 1002 may generate a graphical representation of the remote resource 1210, such as an icon, and incorporate the icon into the computing environment 1004 upon receiving the identification of the remote resource 1210 from the remote machine 106'.
- the first agent 1002 receives a request for execution of the remote resource 1210 when a resource executing on one of the first client 102a and the second client 102b attempts to access a resource, such as a file, that requires the execution of the remote resource 1210.
- the first agent directs a second agent on the second machine to request execution of the remote resource provided by the third machine (step 1308).
- the first agent 1002 directs the second agent 1010 to execute another agent 1010' on the second client 102b and to instruct the second agent 1010' to request execution of the remote resource 1210 provided by the remote machine 106'.
- the second agent 1010' is a presentation layer protocol client, such as an ICA or RDP client.
- the second agent 1010' establishes a connection to the remote machine 106' and requests execution of the remote resource 1210. In yet another embodiment, the second agent 1010' receives output data generated by the execution of the remote resource 1210 and displays the output data in the local window 1014. In some embodiments, the second agent 1010 directs the first agent 1002 to request execution of the remote resource 1210 on server 106 upon making a determination the requested resource is unavailable or not executable from the second client 102b.
- the second agent transmits, to the first agent, an identification of a change to a z-order entry of a local window on the second machine displaying output data generated by the remote resource provided by the third machine (step 1310).
- the second agent 1010 transmits the identification of the change to the z-order entry of the local window 1014 to the first agent 1002.
- the second agent 1010 transmits the identification of the change to the z-order entry of the local window 1014 to the first agent 1002 as described above in connection with FIG. 1OA, 1OB, 1 IA, and 1 IB.
- the first agent modifies a z-order entry for a proxy window representing the local window in a z-order list on the first machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1312).
- the first agent 1002 modifies a z-order entry for a proxy window 1008 as described above in connection with FIG. 1OA, 1OB, HA, and HB.
- methods and systems described herein provide functionality for addressing limitations on displaying output data on mobile devices, including limitations resulting from the relatively small display form factor and the limited computing capacity.
- methods and systems are provided for turning the mobile device into an application delivery channel.
- no changes are required of existing desktop applications to implement these methods and systems, and a wide variety of applications can be accessed on such an application-delivery enabled mobile device.
- FIG. 14A a block diagram depicts one embodiment of a system for executing on a mobile computing device a virtual input device for interacting with a resource, wherein output data for the resource is displayed on an external display device.
- the system includes a mobile computing device 102, a native display 201 on the mobile computing device 102, an external display device 202, and a virtual input device 1402.
- the mobile computing device 102 executes a first resource 1404.
- the external display device 202 displays output data generated by the execution of the first resource 1404.
- the mobile computing device 102 executes a second resource 1406.
- the second resource 1406 generates a virtual input device 1402.
- the virtual input device 1402 leverages one or more capabilities of the mobile computing device 102.
- the virtual input device 1402 allows interaction by a user of the mobile computing device 102 with the display of output data generated by the execution of the first resource 1404.
- the system includes a mobile computing device 102.
- the system is a mobile device 102 as described above in connection with FIGs. 1-13.
- the system includes a native display 201 on the mobile computing device 102, an external display device 202, and a virtual input device 1402.
- the mobile computing device 102 executes a first resource 1404.
- the external display device 202 displays output data generated by the execution of the first resource 1404.
- the external display device 202 is a monitor.
- the external display device 202 is a projector.
- the mobile computing device 102 is connected or attached to an external display device 202, such as a projector that transmits output data to an external display surface.
- the external display device 202 may be integrated into the mobile computing device 102.
- the mobile computing device 102 may include a pico projector, which directs a display of output data generated by a resource executing on the mobile computing device 102 to an external display surface such as a projection screen, wall, or other display surface.
- the mobile computing device 102 may include a projection module based on a microelectromechanical system (MEMS), such as, for example and without limitation, a PicoP Display Engine manufactured by Microvision, Inc., of Redmond, WA.
- MEMS microelectromechanical system
- the mobile computing device 102 executes a second resource 1406.
- the second resource 1406 generates a virtual input device 1402 for interaction by a user of the mobile computing device with the display of output data generated by the execution of the first resource.
- the mobile computing device 102 may execute the second resource 1406 according to the input needs of the first resource 1404, the identification of the mobile computing device 102, or both.
- a virtual input device 1402 includes a graphical representation of a keyboard.
- a virtual input device 1402 includes a virtual track pad.
- a virtual input device 1402 includes a virtual pointing device, such as a cursor which may be manipulated by interacting with the virtual input device 1402.
- the virtual input device 1402 may be displayed on the native display 201 on the mobile computing device 102.
- the virtual input device 1402 allows interaction by a user of the mobile computing device 102 with the display of output data generated by the execution of the first resource 1404.
- the native display 201 on the mobile computing device 102 includes a touch-screen.
- a touchscreen sensor includes a touch-responsive surface that detects touch input from a user of the mobile computing device 102.
- the virtual input device 140 is displayed on the touch-responsive surface.
- the touch-screen sensor redirects an identification of a location of the touch input on the touch- responsive surface to an operating system on the mobile computing device 102.
- the operating system redirects the identification of the location of the touch input to the second resource 1406 for processing.
- the second resource 1406 correlates the location of the touch input with a displayed user interaction element.
- the second resource 1406 transmits an identification of a user interaction with the displayed user interaction element to the first resource 1404.
- the virtual input device 1402 includes both a virtual keyboard and a virtual pointing device and interactions with these virtual components are redirected to the mobile device capabilities or to secondary external devices.
- mobile devices that have full QWERTY keyboards and cursor positioning buttons or joystick keys can have input received by the virtual keyboard and mouse redirected to those capabilities.
- mobile devices with touch screens can be repurposed as a "soft keyboard” and "soft mouse” and the input received by the virtual keyboard and mouse can be redirected directly to an executing resource.
- input data received by a mobile device with an externally connected keyboard or mouse can be redirected to the virtual keyboard and mouse.
- a first resource 1404 may also accept input from other devices, such as camera, microphone, point-of-sale (POS) device, etc.
- a virtual input device 1402 may provide a virtualized version of any such devices and can redirect data communication for these devices. For instance, and in another of these embodiments, if a mobile computing device 102 has a built-in camera, an application displayed on an external display device 202 and accepting input from a camera could be directed to use the mobile device camera. In still another of these embodiments, and by way of example, an application that has audio output could be directed, for example, to the mobile device's speaker or headphone jack.
- a screen shot depicts an embodiment of a mobile computing device 102 displaying a virtual input device 1402 on the native display 201, wherein the virtual input device 1402 permits the user to interact with the output data associated with a resource 1406, such as a word processing application.
- the resource 1406 may be executing on the mobile computing device 102.
- the window management service 206 may detect the connection and display output data associated with the resource 1406 on the external display device 202 instead of the native display 201 of the mobile computing device 102.
- the external display device 202 is a projector that displays the output data on a surface, such as a wall.
- the mobile computing device 102 may execute a resource that generates a virtual input device 1402.
- the mobile computing device 102 may evaluate the resource 1406 to determine its input needs.
- the mobile computing device 102 may determine that a resource accepts alphanumeric input, audio input, video input, or any other type of input or combinations thereof.
- the resource 1402 is a word processing application that needs at least alphanumeric input.
- the mobile computing device 102 executes a resource to generate a virtual or "soft" keyboard for a virtual input device 1402.
- the virtual keyboard may be displayed on the native display 201, such as a touch- screen.
- the touch-screen may detect user interactions and redirect information associated with the user interactions to the virtual input device 1402.
- the virtual input device 1402 may communicate the resources to the resource 1402.
- a screen shot depicts an embodiment of a mobile computing device displaying a virtual input device 1402 for interaction by a user with a display of output data associated with a resource.
- the mobile computing device 102 executes a resource to generate at least a virtual trackpad.
- the virtual trackpad may be displayed on the native display 201, such as a touch-screen, of the mobile computing device 102.
- the touch-screen may redirect information associated with the user interaction to the resource generating the virtual input device 1402.
- the virtual input device 1402 may process the information and determine a user command.
- the virtual input device 1402 may communicate the user command to the resource being displayed on an external display device (not shown), and the resource being displayed may output data corresponding to the user command.
- FIG. 14C also depicts a virtual input device 1402 that repurposes a keyboard on the mobile computing device 102 for the resource being displayed on an external display device.
- the mobile computing device 102 displays a keyboard generated by an application installed on the device.
- the touch-screen may detect a user interaction and transmit information associated with the user interaction to the keyboard application.
- the keyboard application may interpret the user interaction to determine a user command, such as entering or deleting alphanumeric characters.
- the keyboard application may communicate the user command to the virtual input device 1402, which may transmit information about the user command to the resource whose output data is being displayed on an external display device.
- one resource may generate a virtual trackpad and another resource may repurpose the keyboard on the mobile computing device 102.
- the same resource may generate a virtual input device 1402 that generates the virtual trackpad and repurposes the keyboard.
- a flow diagram depicts one embodiment of a method for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device.
- the method includes executing, by a mobile computing device, a first resource (1502).
- the method includes directing a display, on an external display device, of output data generated by the executing first resource (1504).
- the method includes executing, on the mobile computing device, a second resource generating a virtual input device (1506).
- the method includes displaying, on a screen of the mobile computing device, the virtual input device (1508).
- the method includes transmitting, by the second resource to the first resource, an indication of an interaction by a user of the mobile computing device with the virtual input device (1510).
- the mobile device 102 is repurposed as a virtual input device that transmits user input to either a locally- executing application or, via presentation-layer protocols, to a remotely-executing application.
- the mobile computing device 102 executes a first resource (1502).
- the mobile computing device 102 executes an application generating output data for display to a user of the mobile computing device.
- the first resource requests, from a remote computing device, execution of a remote Iy- located resource.
- a first resource communicates with the remote computing device according to a presentation layer protocol, for example, by communicating via the ICA protocol, or the RDP protocol.
- the first resource directs the display of remotely-generated output data on a display device associated with the mobile computing device.
- the first resource is stored on the mobile computing device, executes locally, and directs the display of locally-generated output data on a display device associated with the mobile computing device.
- a display, by an external display device, of output data generated by the executing first resource is directed (1504).
- a window management service directs the display of output data to the external display device 124b.
- an operating system executing on the mobile computing device 102 and in communication with the external display device 124b directs the display of output data to the external display device 124b.
- the external display device projects the output data to an external display surface, such as a wall, screen, or other surface.
- the display of the output data occurs without modification to the output data generated by the execution of the first resource.
- the output data is rendered on a secondary physical display, such as an embedded or external micro projector, heads-up display, laptop, HDTV, or other display.
- a secondary physical display such as an embedded or external micro projector, heads-up display, laptop, HDTV, or other display.
- displayed output from the first resource is shown via the secondary display and is, therefore, not limited to only the display capabilities of the mobile computing device display.
- the mobile computing device executes a second resource generating a virtual input device (1506).
- the second resource communicates with the first resource.
- the second resource identifies a type of virtual input device to display; for example, the second resource may select a type of virtual input device to display based on a type of interaction supported by the first resource.
- the virtual input device is displayed on a screen of the mobile computing device (1508).
- the virtual input device is displayed in conjunction with an existing virtual input device; for example, the mobile computing device may display a virtual (or "soft") keyboard and the second resource may generate a second virtual input device, such as a virtual trackpad displayed in conjunction with the virtual keyboard.
- the second resource may generate a customized virtual input device displayed on the mobile computing device in place of the display of output data generated by the first resource.
- the display of the output data generated by the first resource is separated from the display of the virtual input device, instead of combined into a singled display surface, such as either an internal or an external display device.
- the second resource transmits, to the first resource, an indication of an interaction by a user of the mobile computing device with the virtual input device (1510).
- the second resource communicates with the first resource, allowing a user to interact with the first resource via the virtual input device generated by the second resource and displayed on a separate display device than the first resource.
- the first resource is a client application executing on the mobile computing device 102 and communicating with a remote computing device 106 according to a presentation layer protocol.
- the first resource requests execution of a remote resource by the remote computing device.
- the first resource receives, from the remote computing device, output data generated by an execution of the remote resource on the remote computing device.
- the first resource transmits, to the remote computing device, execution of a remote resource, and receives, from a second remote computing device, output data generated by an execution of the remote resource on the second remote computing device.
- the mobile computing device directs a display of the remotely- generated output data on the external display device.
- the methods and systems described herein provide functionality for smart translation of user interaction with endpoint devices.
- methods and systems are provided for translating a user interaction with an endpoint device into a command for processing by a remotely-executing resource.
- a resource may execute on a first computing device and generate output data that is displayed on a second computing device, which may be referred to as an endpoint device; a user interaction with the endpoint device may be translated into a command that is transmitted to the resource on the first computing device for processing.
- a resource executing on a first computing device provides a user interface allowing a user to interact with output data generated by a resource.
- output data may include one or more graphical user interface elements, such as a cursor or a scroll bar, displayed with data, such as the text in a word processing application or an image in an image viewing or editing software, and the graphical user interface elements provide a mechanism by which a user of the resource may interact with data generated by the resource.
- the type of user interface varies based upon the computing environment from which the user views the output data.
- a first type of user interface may be provided when a user accesses output data from a mobile computing device and a second type of user interface may be provided when a user accesses output data from a different type of computing device, such as a laptop, or when the user has access to a different type of display device.
- methods for interacting with a user interface are different across different user interfaces - for example, users may access a pointing device (such as a mouse) to manipulate an image of a cursor on a screen in order to interact with a graphical user interface element, or users may interact with a touch screen on a display device and use their fingers to tap on or drag graphical user interface elements from one portion of the display to another.
- a pointing device such as a mouse
- a method for interacting with a first type of user interface does not work when applied to a second type of user interface (for example, if a second type of user interface accepts as input a user touch to a screen, the second type of user interface may not provide functionality for displaying a cursor or for accepting input from a pointing device); even if the method does work, it may be a suboptimal method for interacting with the second type of user interface or it may only provide limited functionality for interacting with the second type of user interface.
- a first type of user interface may be preferable for use on a first type of computing device - for example, a user interface that supports a user entering commands by touching a screen may be better suited for use on a computing device that includes a touch screen than on one that does not.
- a remotely-executing application generates and displays a first type of user interface intended for access via a first type of computing device but which is displayed to a user accessing the user interface from a second type of computing device. For example, a user may access a first type of user interface that was designed for access from a computing device including a pointing device from a computing device that includes a touch screen but not a pointing device.
- the methods and systems described herein provide functionality for translating input of a first type into a second type.
- input provided by a user via a first type of interface is translated into a command that a second type of interface can process.
- a user may view data including a user interface displaying interface elements such as a scroll bar and accepting input commands from the user via a pointing device; however, the user may enter a command via a touch screen and the command is translated into a format that the user interface can process.
- a user viewing a first section of data may have requested a display of a second section of data, for example by using a touch screen to scroll through the data; the command to scroll from the first section of data to the second may be translated into a command to select a graphical user interface element for scrolling down (such as an arrow in a scroll bar) and transmitted to the resource generating the data.
- implementation of this functionality allows a user of a first type of computing device to interact with a display of a first type of user interface using a second type of user interface element optimized for use on the first type of computing device.
- a flow diagram depicts one embodiment of a method for translating a user interaction with an input/output device of a first type into a command for processing by a user interface supporting interaction with an input/output device of a second type.
- the method includes providing an input/output device of a first type for interaction with a user interface of a first type (1602).
- the method includes identifying a user interaction with a user interface of a second type via the input/output device of the first type (1604).
- the method includes generating, responsive to the identified user interaction, a command formatted for processing by the user interface of the second type according to a protocol for interacting with an input/output device of a second type (1606).
- the method includes processing the command by the user interface of the second type (1608).
- an input/output device of a first type is provided for interaction with a user interface of a first type (1602).
- the input/output device of the first type is a touch screen.
- the user interface of the first type includes functionality for processing touch screen-based interactions with user interface elements displayed by the user interface of the first type.
- the input/output device of the first type is an input/output device such as a keyboard or pointing device.
- the user interface of the first type includes a graphical element (such as a cursor) displayed to a user and with which the user interacts via the input/output device of the first type.
- the user interface of the first type provides functionality for processing interactions with the graphical element received via input/output devices such as keyboards or pointing devices.
- a user interface of a second type is displayed to a user.
- a user of a mobile computing device 102 views a user interface of the second type generated by a resource executing on the mobile computing device 102. In another of these embodiments, a user of a mobile computing device 102 views a user interface of the second type generated by a resource executing on a remote computing device 106. In still another of these embodiments, the user interface of the second type does not include functionality for processing user input generated by user interactions with the input/output device of the first type.
- a user interface of the second type may be a windowing environment that displays widgets such as pull-down menus, scroll bars, and task bars and that includes functionality for processing user input generated by a user interaction with a pointing device or keyboard (e.g., a user clicking on a scroll bar with a mouse), but may not include functionality for processing user input generated by a user interaction with a touch screen.
- a user interaction with a user interface of a second type via the input/output device of the first type is identified (1604).
- an interaction management component identifies the user interaction.
- the user interaction is transmitted to an interaction management component for processing.
- the input/output device of the first type is a touch screen
- a user may touch the screen of a mobile computing device in an attempt to manipulate data displayed by the user interface of the second type, such as a windowing environment displaying a pull-down menu.
- a command is generated responsive to the identified user interaction, the command formatted for processing by the user interface of the second type according to a protocol for interacting with an input/output device of a second type (1606).
- an interaction management component accesses a mapping, table, or other data structure storing configuration data to generate the command.
- the interaction management component analyzes the user interaction with input/output device of the first type to identify a request or command intended by the user interaction; for example, the interaction management component may determine that a user touching a section of a touch screen intending to request access to data or to generate a command to modify an existing display of data.
- the interaction management component generates a type of command that is typically a result of a user interaction with the user interface of the second type.
- the interaction management component determines that a user has interacted with a touch screen in order to request a modification of an existing display of data and the interaction management component generates a command equivalent to a command that would have been generated if the user had accessed an input/output device such as a mouse to select a graphical user interface element.
- the command is processed by the user interface of the second type (1608).
- the interaction management component transmits the command to a resource generating the user interface of the second type.
- the interaction management component transmits the command to a resource executing on the mobile computing device 102.
- the interaction management component transmits the command to a resource executing on a second computing device 106.
- the systems and methods described above may be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture.
- the article of manufacture may be a floppy disk, a hard disk, a CD-ROM, a flash memory card, a PROM, a RAM, a ROM, or a magnetic tape.
- the computer-readable programs may be implemented in any programming language, LISP, PERL, C, C++, PROLOG, or any byte code language such as JAVA.
- the software programs may be stored on or in one or more articles of manufacture as object code.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Information Transfer Between Computers (AREA)
- User Interface Of Digital Computer (AREA)
- Digital Computer Display Output (AREA)
- Telephone Function (AREA)
Abstract
The present disclosure relates to methods and systems for using external display devices with a mobile computing device. When an external display device is connected or attached to a mobile computing device, output data associated with a resource executing on the mobile computing device may be displayed on the external display device. The output data may include a user interface appropriate for the display of the external display device. Execution of the resource may be transferred from the mobile computing device to the external display device. The mobile computing device may be repurposed to display a virtual input device for interacting with a resource with output data being displayed on an external display device.
Description
PATENT APPLICATION
METHODS AND SYSTEMS FOR USING EXTERNAL DISPLAY DEVICES WITH A MOBILE COMPUTING DEVICE
RELATED APPLICATIONS
[0001] The present application claims priority to United States Provisional Patent
Application No. 61/054,551, entitled "METHODS AND SYSTEMS FOR MANAGING DISPLAY AND EXECUTION OF RESOURCES ACROSS MULTIPLE COMPUTING ENVIRONMENTS" and filed on May 20, 2008, which is incorporated herein by reference in its entirety.
FIELD OF THE INVENTION
[0002] The present disclosure relates to methods and systems for using external display devices with a mobile computing device. When an external display device is connected or attached to a mobile computing device, output data associated with a resource executing on the mobile computing device may be displayed on the external display device. The output data may include a user interface appropriate for the display of the external display device. Execution of the resource may be transferred from the mobile computing device to the external display device. The mobile computing device may be repurposed to display a virtual input device for interacting with a resource with output data being displayed on an external display device.
BACKGROUND OF THE INVENTION
[0003] With the pervasiveness of computers, network access, and Internet usage in daily personal and business-related activities, many users desire mobility and flexibility in their computing activities. As users physically move from location to location, and shift from activity to activity, they typically seek convenience and familiarity in performing computing activities. Users may carry one or more mobile computing devices - such as a smartphone, personal digital assistant, laptop or notebook computer - to provide a personalized computing experience as they move from location to location. Many users also use a portable storage device, such as a flash drive, to carry files, documents or applications in a convenient and
portable manner. With such a storage device, a user accesses desired files or documents from publicly available computers to which the portable storage device can connect.
[0004] Many widely available modern computer desktop applications include rich graphical user interfaces (GUI) that have been developed for use on conventional personal computers, workstations, laptops, or other computing devices, typically have large displays, keyboard and mouse, etc. In contrast, mobile devices are often designed to be small enough to fit in a pocket, have relatively small displays, and some (often limited) means for data input such as telephone keypad, full keyboard, touch screen, stylus, etc. When it comes to supporting desktop or enterprise applications on mobile devices, limiting factors include the small display size and meager computational capacity. Various methods have been developed to run some desktop applications on mobile devices, but the results typically range from cumbersome and clumsy to not feasible or cost-prohibitive. The most common examples include rewriting applications, in whole or in part, to accommodate mobile device limitations, providing limited or "lite" versions of applications, scaling the application's graphics to fit on the mobile device display, and using zooming and scrolling to turn the mobile device display into a "virtual lens" to view sub-regions of the application at a time.
BRIEF SUMMARY OF THE INVENTION
[0005] By enabling mobile computing devices to leverage external display devices, users may continue to benefit from a small form factor device's portability and convenience without sacrificing quality of display or resource capabilities. Transferring a resource's display from a mobile computing device to an external display device and selecting a user interface optimized for the external display can greatly improve a user's comfort and ease while using the resource. Repurposing the mobile computing device as an input device for the external display minimally disrupts a user's work since the repurposing eliminates the need to obtain and configure additional devices that interact with the external display device. Further, by accessing the external display device's superior computational capacity, a user may run resources with complex and desired features without being constrained by the mobile computing device's own limitations.
[0006] In one aspect, a method for displaying, on an external display device connected to a mobile computing device, output data associated with at least one resource. The method includes the step of receiving, by a virtual graphics driver on the mobile
computing device, an identification of an external display device connected to the mobile computing device. The method includes the step of selecting, by a window management service on the mobile computing device according to the identification, at least one resource to display on the external display device. The method includes the step of positioning, by the window management service, output data associated with the at least one resource in a virtual screen space. The method includes the step of transmitting, by the virtual graphics driver, the output data to the external display device according to the virtual screen space.
[0007] In some embodiments, the method includes the step of allocating, by the window management service, a screen space corresponding to the external display device in the virtual screen space according to the identification. In some embodiments, the method includes the step of allocating, by the window management service, the screen space corresponding to the external display device according to a resolution of the external display device. In some embodiments, the method includes the step of removing, by the window management service, output data associated with the at least one resource from a position in the virtual screen space corresponding to a native display of the mobile computing device. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a type of the resource. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a user association between the resource and a cell in a grid display layout. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a policy associated with cells in a grid display layout. In some embodiments, the method includes the step of re-positioning, by the window management service, the output data in the virtual screen space according to an alteration of a grid display layout. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a user command to display the output data in an empty cell. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a policy associated with a dynamic display layout. In some embodiments, the method includes the step of positioning, by the window management service, the output data in the virtual screen space according to a default position associated with the resource. In some embodiments, the method includes the step of receiving, by the window management service, information associated with a user interaction with the output data. In some
embodiments, the method includes the step of re-positioning, by the window management service, the output data according to the information associated with the user interaction. In some embodiments, the method includes the step of re-positioning, by the window management service, the output data to a position in the virtual screen space corresponding to a native display of the mobile computing device or a different external display device. In some embodiments, the method includes the step of obscuring, by the window management service, output data associated with a different resource. In some embodiments, the method includes the step of re-sizing, by the window management service, the output data associated with the resource.
[0008] In another aspect, a mobile computing device for displaying output data associated with at least one resource on an external display device. The mobile computing device also includes a window management service that selects, according to an identification of the external display device, the at least one resource to display on the external display device and positions output data associated with the at least one resource in a virtual screen space. The mobile computing device also includes a virtual graphics driver that transmits the output data to the external display device according to the virtual screen space. In some embodiments, the window management service allocates a screen space corresponding to the external display device in the virtual screen space according to the identification. In further embodiments, the window management service allocates a screen space corresponding to the external display device in the virtual screen space according to a resolution of the external display device. In additional embodiments, the window management service removes output data associated with the at least one resource from a position in the virtual screen space corresponding to a native display of the mobile computing device.
[0009] In yet another aspect, a method for displaying, on an external display device connected to a mobile computing device, a user interface for a resource. The method includes the step of receiving from the resource, by a window management service on the mobile computing device, a first user interface for display on a native display of the mobile computing device. The method also includes the step of transmitting, by a virtual graphics driver on the mobile computing device, the first user interface to the native display. The method also includes the step of transmitting, by the window management service, an identification of the external display device to the resource. The method also includes the step of receiving from the resource, by the window management service, a second user
interface according to the identification, the second user interface being distinct from the first user interface. The method also includes the step of transmitting, by the virtual graphics driver, the second user interface to the external display device.
[0010] In some embodiments, the method includes the step of receiving from the resource, by the window management service, the second user interface optimized for the external display device. In more embodiments, the method includes the step of transmitting, by the window management service, at least one property of the external display device to the resource. In additional embodiments, the method includes the step of transmitting, by the window management service, an indication that a display on the external display device has a larger resolution or size than the native display. In many embodiments, the method includes the step of receiving from the resource, by the window management service, a default user interface in the absence of a user interface optimized for the external display device. In various embodiments, the method includes the step of receiving from the resource, by the window management service, a communication that a user interface optimized for the external display device is unavailable. In some embodiments, the method includes the step of transmitting, by the window management system, an identification of the second user interface and an instruction to display the second user interface to the external display device.
[0011] In another aspect, a method for transferring execution of a virtualized resource from a mobile computing device to an external display device. The method also includes the step of receiving, by a server from the mobile computing device, an identification of an external display device connected to the mobile computing device. The method also includes the step of identifying, by the server, a virtual machine being delivered to the external display device. The method also includes the step of disconnecting, by the server, the virtualized resource from a virtual machine being delivered to the mobile computing device. The method also includes the step of connecting, by the server, the virtualized resource to the virtual machine being delivered to the external display device.
[0012] In some embodiments, the virtualized resource is a virtualized application. In some embodiments, the method includes the step of transferring, by the server, information related to the state of execution of the virtualized resource from the virtual machine being to delivered to the mobile computing device to the virtual machine being delivered to the external display device.
[0013] In another aspect, a method of repurposing a mobile computing device as a virtual input device for a first resource being displayed on an external display device. The method also includes the step of evaluating, by the server, input capabilities of the mobile computing device and input needs of the first resource. The method also includes the step of delivering according to the evaluation, by the server to the mobile computing device, a second resource that generates a virtual input device.
[0014] In some embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that generates a virtual keyboard, virtual track pad, virtual pointing device, virtual camera, virtual microphone, virtual point-of- sale device, virtual speaker, virtual headphone jack, or any combination thereof. In many embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that allows user interaction with output data of the first resource. In various embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that transmits an identification of a user interaction with the virtual input device to the first resource. In additional embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that generates at least one user interaction element. In some embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that redirects user interactions with the virtual input device to the input capabilities of the mobile computing device. In many embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that correlates a location of a touch input with a displayed user interaction element. In various embodiments, the method includes the step of delivering, by the server to the mobile computing device, a third resource that generates another virtual input device in conjunction with the virtual input device of the second resource. In some embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource for display in conjunction with the first resource on a single display surface. In various embodiments, the method includes the step of delivering, by the server to the mobile computing device, the second resource that generates a virtual input device for a second external device connected to the mobile computing device.
[0015] In another aspect, a system for repurposing a mobile computing device as a virtual input device for a first resource being displayed on an external display device. The
system also includes a server that evaluates input capabilities of the mobile computing device and input needs of the first resource and delivers to the mobile computing device, according to the evaluation, a second resource that generates a virtual input device.
[0016] In many embodiments, the second resource generates a virtual keyboard, virtual track pad, virtual pointing device, virtual camera, virtual microphone, virtual point-of- sale device, virtual speaker, virtual headphone jack, or any combination thereof. In some embodiments, the second resource allows user interaction with output data of the first resource. In additional embodiments, the second resource transmits an identification of a user interaction with the virtual input device to the first resource. In various embodiments, the second resource generates at least one user interaction element. In some embodiments, the second resource redirects user interactions with the virtual input device to the input capabilities of the mobile computing device. In many embodiments, the second resource correlates a location of a touch input with a displayed user interaction element. In many embodiments, the server delivers a third resource that generates another virtual input device in conjunction with the virtual input device of the second resource. In various embodiments, the second resource is displayed in conjunction with the first resource on a single display surface. In several embodiments, the second resource generates a virtual input device for a second external device connected to the mobile computing device.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The foregoing and other objects, aspects, features, and advantages of the disclosure will become more apparent and better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:
[0018] FIG. IA is a block diagram depicting an embodiment of a network environment comprising client machines in communication with remote machines;
[0019] FIGs. IB and 1C are block diagrams depicting embodiments of computers useful in connection with the methods and systems described herein;
[0020] FIG. 2A is a block diagram depicting one embodiment of a system for displaying a plurality of resources in a user-configurable display layout on an external display device;
[0021] FIG. 2B is a block diagram depicting a system for mapping a display of one or more resources to one or more display devices;
[0022] FIG. 2C is a screen shot depicting one embodiment of a system for displaying a plurality of resources in a user-configurable display layout on an external display device, the user-configurable display layout divided into a grid;
[0023] FIG. 2D is a screen shot depicting one embodiment of a system for displaying a plurality of resources in a user-configurable, dynamic display layout on an external display device;
[0024] FIG. 3 A is a block diagram depicting one embodiment of a plurality of screen spaces provided by a mobile computing device attached to one or more external display devices;
[0025] FIG. 3B is a block diagram depicting one embodiment of a mobile computing device providing a plurality of screen spaces;
[0026] FIG. 3 C is a block diagram depicting one embodiment of a logical representation of a plurality of screen spaces managed by a virtual graphics driver;
[0027] FIG. 4A is a block diagram depicting one embodiment of a graphical user interface for the addition of and configuration of additional screen spaces;
[0028] FIG. 4B is a block diagram depicting one embodiment of a graphical user interface for customizing a grid display layout;
[0029] FIG. 4C is a screen shot depicting one embodiment of a customizable grid display layout;
[0030] FIG. 4D is a screen shot depicting one embodiment of unstacked displays on a mobile computing device;
[0031] FIG. 4E is a flow diagram depicting one embodiment of the steps taken in a method for displaying, by an external display device, output data generated by a plurality of resources executing on a mobile computing device;
[0032] FIG. 5 A is a block diagram depicting one embodiment of a system for displaying, by an external display device, a first user interface to a resource distinct from a second user interface to the resource displayed by a mobile computing device executing the resource;
[0033] FIG. 5B is a block diagram depicting a system for mapping a display of one or more user interfaces to one or more display devices;
[0034] FIG. 5C is a screen shot depicting one embodiment of a system for displaying, by an external display device, a first user interface to an application distinct from a second user interface to the application displayed by a mobile computing device executing the application;
[0035] FIG. 6 is a flow diagram depicting an embodiment of the steps taken in a method for displaying, by an external display device, a first user interface to a resource distinct from a second user interface to the resource displayed by a mobile computing device executing the resource.
[0036] FIG. 7A is a block diagram depicting one embodiment of a system for executing and displaying a plurality of resources on a client computing device;
[0037] FIG. 7B is a block diagram depicting one embodiment of a system for executing a plurality of resources on a server and displaying them on a client computing device;
[0038] FIG. 8A is a block diagram of one embodiment of a system for executing a plurality of resources from a mobile computing device on a remote server and displaying results on a client computing device;
[0039] FIG. 8B is a flow diagram depicting one embodiment of the steps taken in a method for executing a plurality of resources from a mobile computing device on a remote server and displaying results on a client computing device;
[0040] FIG. 9A is a block diagram depicting one embodiment of a server for displaying and activating disconnected sessions with a mobile computing device and/or a client computing device;
[0041] FIG. 9B is a flow diagram depicting an embodiments of the steps taken in a method for receiving output data generated by the at least one previously disconnected session;
[0042] FIG. 9C is a flow diagram depicting one embodiment of the steps taken in a method for displaying and activating disconnected sessions with a mobile computing device and/or a client computing device;
[0043] FIG. 1OA is a block diagram depicting an embodiment of a system for generating a computing environment on a remote machine for display on a local machine;
[0044] FIG. 1OB is a block diagram depicting one embodiment of a system in which an agent integrates applications from various sources into a computing environment;
[0045] FIG. 1 IA is a flow diagram depicting one embodiment of the steps taken in a method for generating a computing environment on a remote machine for display on a local machine;
[0046] FIG. 1 IB is a flow diagram depicting one embodiment of the steps taken in a method for displaying, on the local machine, a desktop environment that is generated by the remote machine;
[0047] FIG. 12A is a block diagram depicting an embodiment of a system for generating a computing environment incorporating output data from multiple remote machines for display on a local machine;
[0048] FIG. 12B is a screen shot depicts one embodiment of a system for generating a computing environment incorporating output data from multiple remote machines for display on a local machine;
[0049] FIG. 13 is a flow diagram depicting one embodiment of a method for generating a computing environment incorporating output data from multiple remote machines for display on a local machine;
[0050] FIG. 14A is a block diagram depicting one embodiment of a system for displaying, on a screen of a mobile computing device, a virtual input device for interacting
with a resource generating output data displayed on a display surface external to the mobile computing device;
[0051] FIGS. 14B and 14C are screen shots depicting embodiments of a mobile computing device displaying a virtual input device for interaction by a user of the mobile computing device with a display of output data generated by an execution of a resource;
[0052] FIG. 15 is a flow diagram depicting one embodiment of a method for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device; and
[0053] FIG. 16 is a flow diagram depicting one embodiment of a method for translating a user interaction with an input/output device of a first type into a command for processing by a user interface supporting interaction with an input/output device of a second type.
TABLE OF CONTENTS
[0054] FIGs. IA- 1C relate to general computing devices and networks;
[0055] FIGs. 2A-4E relate to methods and systems for displaying resources executing on a mobile computing device in a user configurable layout on an external display;
[0056] FIGs. 5A-6 relate to dynamic switching between user interfaces;
[0057] FIGs. 7A-7B relate to methods and systems for displaying on a client computing device, output data generated by resources executed on the client computing device or a server;
[0058] FIGs. 8A-8B relate to methods and systems for displaying on a client computing device, output data generated by resources executed on a mobile computing device;
[0059] FIGs. 9A-9C relate to methods and systems for handling disconnected sessions between one or more of computing devices (included to support workspace control/smooth roaming aspects of a mobile computing device communicating with a server); and
[0060] FIGs. 10A- 13 relate to methods and systems for generating a computing environment on one or more remote machines and displaying them on a local machine (included to provide support for combining Nirvana techniques with reverse seamless subject matter).
[0061] FIGs. 14A-15 relate to methods and systems for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device.
[0062] FIG. 16 relates to methods and systems for translating a user interaction with an input/output device of a first type into a command for processing by a user interface supporting interaction with an input/output device of a second type.
DETAILED DESCRIPTION OF THE INVENTION
[0063] Referring now to Figure IA, an embodiment of a network environment is depicted. In brief overview, the network environment comprises one or more clients 102a- 102n (also generally referred to as local machine(s) 102, client(s) 102, client node(s) 102, client machine(s) 102, client computer(s) 102, client device(s) 102, endpoint(s) 102, or endpoint node(s) 102) in communication with one or more servers 106a-106n (also generally referred to as server(s) 106 or remote machine(s) 106) via one or more networks 104. In some embodiments, a client 102 has the capacity to function as both a client node seeking access to resources provided by a server and as a server providing access to hosted resources for other clients 102a-102n.
[0064] Although FIG. IA shows a network 104 between the clients 102 and the servers 106, the clients 102 and the servers 106 may be on the same network 104. The network 104 can be a local-area network (LAN), such as a company Intranet, a metropolitan area network (MAN), or a wide area network (WAN), such as the Internet or the World Wide Web. In some embodiments, there are multiple networks 104 between the clients 102 and the servers 106. In one of these embodiments, a network 104' (not shown) may be a private network and a network 104 may be a public network. In another of these embodiments, a network 104 may be a private network and a network 104' a public network. In still another embodiment, networks 104 and 104' may both be private networks.
[0065] The network 104 may be any type and/or form of network and may include any of the following: a point to point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network and a wireline network. In some embodiments, the network 104 may comprise a wireless link, such as an infrared channel or satellite band. The topology of the network 104 may be a bus, star, or ring network topology. The network 104 may be of any such network topology as known to those ordinarily skilled in the art capable of supporting the operations described herein. The network may comprise mobile telephone networks utilizing any protocol or protocols used to communicate among mobile devices, including AMPS, TDMA, CDMA, GSM, GPRS or UMTS. In some embodiments, different types of
data may be transmitted via different protocols. In other embodiments, the same types of data may be transmitted via different protocols.
[0066] In some embodiments, the system may include multiple, logically-grouped servers 106. In one of these embodiments, the logical group of servers may be referred to as a server farm 38. In another of these embodiments, the servers 106 may be geographically dispersed. In other embodiments, a server farm 38 may be administered as a single entity. In still other embodiments, the server farm 38 comprises a plurality of server farms 38. The servers 106 within each server farm 38 can be heterogeneous - one or more of the servers 106 can operate according to one type of operating system platform (e.g., WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Washington), while one or more of the other servers 106 can operate on according to another type of operating system platform (e.g., Unix or Linux).
[0067] The servers 106 of each server farm 38 do not need to be physically proximate to another server 106 in the same server farm 38. Thus, the group of servers 106 logically grouped as a server farm 38 may be interconnected using a wide-area network (WAN) connection or a metropolitan-area network (MAN) connection. For example, a server farm 38 may include servers 106 physically located in different continents or different regions of a continent, country, state, city, campus, or room. Data transmission speeds between servers 106 in the server farm 38 can be increased if the servers 106 are connected using a local-area network (LAN) connection or some form of direct connection.
[0068] Server 106 may be a file server, application server, web server, proxy server, appliance, network appliance, gateway, application gateway, gateway server, virtualization server, deployment server, SSL VPN server, or firewall. In some embodiments, a server 106 provides a remote authentication dial-in user service, and is referred to as a RADIUS server. In other embodiments, a server 106 may have the capacity to function as either an application server or as a master application server. In still other embodiments, a server 106 is a blade server. In yet other embodiments, a server 106 executes a virtual machine providing, to a user or client computer 102, access to a computing environment.
[0069] In one embodiment, a server 106 may include an Active Directory. The server
106 may be an application acceleration appliance. For embodiments in which the server 106 is an application acceleration appliance, the server 106 may provide functionality including
firewall functionality, application firewall functionality, or load balancing functionality. In some embodiments, the server 106 comprises an appliance such as one of the line of appliances manufactured by the Citrix Application Networking Group, of San Jose, CA, or Silver Peak Systems, Inc., of Mountain View, CA, or of Riverbed Technology, Inc., of San Francisco, CA, or of F5 Networks, Inc., of Seattle, WA, or of Juniper Networks, Inc., of Sunnyvale, CA.
[0070] In some embodiments, a server 106 executes an application on behalf of a user of a client 102. In other embodiments, a server 106 executes a virtual machine, which provides an execution session within which applications execute on behalf of a user or a client 102. In one of these embodiments, the execution session is a hosted desktop session. In another of these embodiments, the execution session provides access to a computing environment, which may comprise one or more of: an application, a plurality of applications, a desktop application, and a desktop session in which one or more applications may execute.
[0071] In some embodiments, a client 102 communicates with a server 106. In one embodiment, the client 102 communicates directly with one of the servers 106 in a server farm 38. In another embodiment, the client 102 executes a program neighborhood application to communicate with a server 106 in a server farm 38. In still another embodiment, the server 106 provides the functionality of a master node. In some embodiments, the client 102 communicates with the server 106 in the server farm 38 through a network 104. Over the network 104, the client 102 can, for example, request execution of various applications hosted by the servers 106a-106n in the server farm 38 and receive output of the results of the application execution for display. In some embodiments, only the master node provides the functionality required to identify and provide address information associated with a server 106b hosting a requested application.
[0072] In one embodiment, the server 106 provides the functionality of a web server.
In another embodiment, the server 106a receives requests from the client 102, forwards the requests to a second server 106b and responds to the request by the client 102 with a response to the request from the server 106b. In still another embodiment, the server 106 acquires an enumeration of applications available to the client 102 and address information associated with a server 106' hosting an application identified by the enumeration of applications. In yet another embodiment, the server 106 presents the response to the request to the client 102 using a web interface. In one embodiment, the client 102 communicates directly with the
server 106 to access the identified application. In another embodiment, the client 102 receives output data, such as display data, generated by an execution of the identified application on the server 106.
[0073] In some embodiments, the server 106 or a server farm 38 may be running one or more applications, such as an application providing a thin-client computing or remote display presentation application. In one embodiment, the server 106 or server farm 38 executes as an application any portion of the CITRIX ACCESS SUITE by Citrix Systems, Inc., such as the METAFRAME or CITRIX PRESENTATION SERVER and/or any of the MICROSOFT WINDOWS Terminal Services manufactured by the Microsoft Corporation. In another embodiment, the application is an ICA client, developed by Citrix Systems, Inc. of Fort Lauderdale, Florida. In still another embodiment, the server 106 may run an application, which, for example, may be an application server providing email services such as MICROSOFT EXCHANGE manufactured by the Microsoft Corporation of Redmond, Washington, a web or Internet server, or a desktop sharing server, or a collaboration server. In yet another embodiment, any of the applications may comprise any type of hosted service or products, such as GOTOMEETING provided by Citrix Online Division, Inc. of Santa Barbara, California, WEBEX provided by WebEx, Inc. of Santa Clara, California, or Microsoft Office LIVE MEETING provided by Microsoft Corporation of Redmond, Washington.
[0074] A client 102 may execute, operate or otherwise provide an application, which can be any type and/or form of software, program, or executable instructions such as any type and/or form of web browser, web-based client, client-server application, a thin-client computing client, an ActiveX control, or a Java applet, or any other type and/or form of executable instructions capable of executing on client 102. In some embodiments, the application may be a server-based or a remote -based application executed on behalf of the client 102 on a server 106. In one embodiments the server 106 may display output to the client 102 using any thin-client or remote-display protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Washington. The application can use any type of protocol and it can be, for example, an HTTP client, an FTP client, an Oscar client, or a Telnet client. In other embodiments, the application comprises any type of software related to voice over
internet protocol (VoIP) communications, such as a soft IP telephone. In further embodiments, the application comprises any application related to real-time data communications, such as applications for streaming video and/or audio.
[0075] The client 102 and server 106 may be deployed as and/or executed on any type and form of computing device, such as a computer, network device or appliance capable of communicating on any type and form of network and performing the operations described herein. FIGs. IB and 1C depict block diagrams of a computing device 100 useful for practicing an embodiment of the client 102 or a server 106. As shown in FIGs. IB and 1C, each computing device 100 includes a central processing unit 121, and a main memory unit 122. As shown in FIG. IB, a computing device 100 may include a storage device 128, an installation device 116, a network interface 118, an I/O controller 123, display devices 124a- n, a keyboard 126 and a pointing device 127, such as a mouse. The storage device 128 may include, without limitation, an operating system, software, and a client agent 120. As shown in FIG. 1C, each computing device 100 may also include additional optional elements, such as a memory port 103, a bridge 170, one or more input/output devices 130a-130n (generally referred to using reference numeral 130), and a cache memory 140 in communication with the central processing unit 121.
[0076] The central processing unit 121 is any logic circuitry that responds to and processes instructions fetched from the main memory unit 122. In many embodiments, the central processing unit 121 is provided by a microprocessor unit, such as: those manufactured by Intel Corporation of Mountain View, California; those manufactured by Motorola Corporation of Schaumburg, Illinois; those manufactured by Transmeta Corporation of Santa Clara, California; the RS/6000 processor, those manufactured by International Business Machines of White Plains, New York; or those manufactured by Advanced Micro Devices of Sunnyvale, California. The computing device 100 may be based on any of these processors, or any other processor capable of operating as described herein.
[0077] Main memory unit 122 may be one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the microprocessor 121, such as Static random access memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM), Dynamic random access memory (DRAM), Fast Page Mode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM), Extended Data Output DRAM (EDO DRAM), Burst Extended Data Output DRAM (BEDO DRAM),
Enhanced DRAM (EDRAM), synchronous DRAM (SDRAM), JEDEC SRAM, PClOO SDRAM, Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), SyncLink DRAM (SLDRAM), Direct Rambus DRAM (DRDRAM), or Ferroelectric RAM (FRAM). The main memory 122 may be based on any of the above described memory chips, or any other available memory chips capable of operating as described herein. In the embodiment shown in FIG. IB, the processor 121 communicates with main memory 122 via a system bus 150 (described in more detail below). FIG. 1C depicts an embodiment of a computing device 100 in which the processor communicates directly with main memory 122 via a memory port 103. For example, in FIG. 1C the main memory 122 may be DRDRAM.
[0078] FIG. 1C depicts an embodiment in which the main processor 121 communicates directly with cache memory 140 via a secondary bus, sometimes referred to as a backside bus. In other embodiments, the main processor 121 communicates with cache memory 140 using the system bus 150. Cache memory 140 typically has a faster response time than main memory 122 and is typically provided by SRAM, BSRAM, or EDRAM. In the embodiment shown in FIG. 1C, the processor 121 communicates with various I/O devices 130 via a local system bus 150. Various buses may be used to connect the central processing unit 121 to any of the I/O devices 130, including a VESA VL bus, an ISA bus, an EISA bus, a MicroChannel Architecture (MCA) bus, a PCI bus, a PCI-X bus, a PCI-Express bus, or a NuBus. For embodiments in which the I/O device is a video display 124, the processor 121 may use an Advanced Graphics Port (AGP) to communicate with the display 124. FIG. 1C depicts an embodiment of a computer 100 in which the main processor 121 communicates directly with I/O device 130b via HYPERTRANSPORT, RAPIDIO, or INFINIBAND communications technology. FIG. 1C also depicts an embodiment in which local busses and direct communication are mixed: the processor 121 communicates with I/O device 130a using a local interconnect bus while communicating with I/O device 130b directly.
[0079] A wide variety of I/O devices 13 Oa- 13 On may be present in the computing device 100. Input devices include keyboards, mice, trackpads, trackballs, microphones, and drawing tablets. Output devices include video displays, speakers, inkjet printers, laser printers, and dye-sublimation printers. The I/O devices may be controlled by an I/O controller 123 as shown in FIG. IB. The I/O controller may control one or more I/O devices such as a keyboard 126 and a pointing device 127, e.g., a mouse or optical pen. Furthermore, an I/O device may also provide storage and/or an installation medium 116 for the computing
device 100. In still other embodiments, the computing device 100 may provide USB connections (not shown) to receive handheld USB storage devices such as the USB Flash Drive line of devices manufactured by Twintech Industry, Inc. of Los Alamitos, California.
[0080] Referring again to FIG. IB, the computing device 100 may support any suitable installation device 116, such as a floppy disk drive for receiving floppy disks such as 3.5-inch, 5.25-inch disks or ZIP disks, a CD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, tape drives of various formats, USB device, hard-drive or any other device suitable for installing software and programs. The computing device 100 may further comprise a storage device, such as one or more hard disk drives or redundant arrays of independent disks, for storing an operating system and other related software, and for storing application software programs such as any program related to the client agent 120. Optionally, any of the installation devices 116 could also be used as the storage device. Additionally, the operating system and the software can be run from a bootable medium, for example, a bootable CD, such as KNOPPIX, a bootable CD for GNU/Linux that is available as a GNU/Linux distribution from knoppix.net.
[0081] Furthermore, the computing device 100 may include a network interface 118 to interface to the network 104 through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (e.g., 802.11, Tl, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or some combination of any or all of the above. Connections can be established using a variety of communication protocols (e.g., TCP/IP, IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), RS232, IEEE 802.11, IEEE 802.11a, IEEE 802.11b, IEEE 802.1 Ig, CDMA, GSM, WiM ax and direct asynchronous connections). In one embodiment, the computing device 100 communicates with other computing devices 100' via any type and/or form of gateway or tunneling protocol such as Secure Socket Layer (SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida. The network interface 118 may comprise a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device 100 to any type of network capable of communication and performing the operations described herein.
[0082] In some embodiments, the computing device 100 may comprise or be connected to multiple display devices 124a-124n, which each may be of the same or different type and/or form. As such, any of the I/O devices 13 Oa- 13 On and/or the I/O controller 123 may comprise any type and/or form of suitable hardware, software, or combination of hardware and software to support, enable or provide for the connection and use of multiple display devices 124a-124n by the computing device 100. For example, the computing device 100 may include any type and/or form of video adapter, video card, driver, and/or library to interface, communicate, connect or otherwise use the display devices 124a-124n. In one embodiment, a video adapter may comprise multiple connectors to interface to multiple display devices 124a-124n. In other embodiments, the computing device 100 may include multiple video adapters, with each video adapter connected to one or more of the display devices 124a-124n. In some embodiments, any portion of the operating system of the computing device 100 may be configured for using multiple displays 124a-124n. In other embodiments, one or more of the display devices 124a-124n may be provided by one or more other computing devices, such as computing devices 100a and 100b connected to the computing device 100, for example, via a network. These embodiments may include any type of software designed and constructed to use another computer's display device as a second display device 124a for the computing device 100. One ordinarily skilled in the art will recognize and appreciate the various ways and embodiments that a computing device 100 may be configured to have multiple display devices 124a-124n.
[0083] In further embodiments, an I/O device 130 may be a bridge between the system bus 150 and an external communication bus, such as a USB bus, an Apple Desktop Bus, an RS-232 serial connection, a SCSI bus, a Fire Wire bus, a Fire Wire 800 bus, an Ethernet bus, an AppleTalk bus, a Gigabit Ethernet bus, an Asynchronous Transfer Mode bus, a HIPPI bus, a Super HIPPI bus, a SerialPlus bus, a SCI/LAMP bus, a FibreChannel bus, or a Serial Attached small computer system interface bus.
[0084] A computing device 100 of the sort depicted in FIGs. IB and 1C typically operates under the control of operating systems, which control scheduling of tasks and access to system resources. The computing device 100 can be running any operating system such as any of the versions of the MICROSOFT WINDOWS operating systems, the different releases of the Unix and Linux operating systems, any version of the MAC OS for Macintosh computers, any embedded operating system, any real-time operating system, any open source
operating system, any proprietary operating system, any operating systems for mobile computing devices, or any other operating system capable of running on the computing device and performing the operations described herein. Typical operating systems include, but are not limited to: WINDOWS 3.x, WINDOWS 95, WINDOWS 98, WINDOWS 2000, WINDOWS NT 3.51, WINDOWS NT 4.0, WINDOWS CE, WINDOWS XP, and WINDOWS VISTA, all of which are manufactured by Microsoft Corporation of Redmond, Washington; MAC OS, manufactured by Apple Computer of Cupertino, California; OS/2, manufactured by International Business Machines of Armonk, New York; and Linux, a freely-available operating system distributed by Caldera Corp. of Salt Lake City, Utah, or any type and/or form of a Unix operating system, among others.
[0085] The computer system 100 can be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile telephone or other portable telecommunication device, media playing device, a gaming system, mobile computing device, or any other type and/or form of computing, telecommunications or media device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein. For example, the computer system 100 may comprise a device of the IPOD family of devices manufactured by Apple Computer of Cupertino, California, a PLAYSTATION 2 , PLAYSTATION 3, or PERSONAL PLAYSTATION PORTABLE (PSP) device manufactured by the Sony Corporation of Tokyo, Japan, a NINTENDO DS, NINTENDO GAMEBOY, NINTENDO GAMEBOY ADVANCED or NINTENDO REVOLUTION device manufactured by Nintendo Co., Ltd., of Kyoto, Japan, or an XBOX or XBOX 360 device manufactured by the Microsoft Corporation of Redmond, Washington.
[0086] In some embodiments, the computing device 100 may have different processors, operating systems, and input devices consistent with the device. For example, in one embodiment, the computing device 100 is a TREO 180, 270, 600, 650, 680, 70Op, 70Ow, or 750 smart phone manufactured by Palm, Inc. In some of these embodiments, the TREO smart phone is operated under the control of the PalmOS operating system and includes a stylus input device as well as a five-way navigator device.
[0087] In other embodiments the computing device 100 is a mobile device, such as a
JAVA-enabled cellular telephone or personal digital assistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the iml 100, all of which are manufactured by Motorola Corp. of
Schaumburg, Illinois, the 6035 or the 7135, manufactured by Kyocera of Kyoto, Japan, or the i300 or i330, manufactured by Samsung Electronics Co., Ltd., of Seoul, Korea. In some embodiments, the computing device 100 is a mobile device manufactured by Nokia of Finland, or by Sony Ericsson Mobile Communications AB of Lund, Sweden.
[0088] In still other embodiments, the computing device 100 is a Blackberry handheld or smart phone, such as the devices manufactured by Research In Motion Limited, including the Blackberry 7100 series, 8700 series, 7700 series, 7200 series, the Blackberry 7520, or the Blackberry Pearl 8100. In yet other embodiments, the computing device 100 is a smart phone, Pocket PC, Pocket PC Phone, or other handheld mobile device supporting Microsoft Windows Mobile Software. Moreover, the computing device 100 can be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile telephone, any other computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.
[0089] In some embodiments, the computing device 100 is a digital audio player. In one of these embodiments, the computing device 100 is a digital audio player such as the Apple IPOD, IPOD Touch, IPOD NANO, and IPOD SHUFFLE lines of devices, manufactured by Apple Computer of Cupertino, California. In another of these embodiments, the digital audio player may function as both a portable media player and as a mass storage device. In other embodiments, the computing device 100 is a digital audio player such as the DigitalAudioPlayer Select MP3 players, manufactured by Samsung Electronics America, of Ridgefield Park, NJ, or the Motorola m500 or m25 Digital Audio Players, manufactured by Motorola Inc. of Schaumburg, IL. In still other embodiments, the computing device 100 is a portable media player, such as the Zen Vision W, the Zen Vision series, the Zen Portable Media Center devices, or the Digital MP3 line of MP3 players, manufactured by Creative Technologies Ltd. In yet other embodiments, the computing device 100 is a portable media player or digital audio player supporting file formats including, but not limited to, MP3, WAV, M4A/AAC, WMA Protected AAC, AIFF, Audible audiobook, Apple Lossless audio file formats and .mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats.
[0090] In some embodiments, the computing device 100 comprises a combination of devices, such as a mobile phone combined with a digital audio player or portable media
player. In one of these embodiments, the computing device 100 is a Motorola RAZR or Motorola ROKR line of combination digital audio players and mobile phones. In another of these embodiments, the computing device 100 is an iPhone smartphone, manufactured by Apple Computer of Cupertino, California.
[0091] Referring now to FIG. 2A, a block diagram depicts one embodiment of a system for displaying a plurality of resources on one or more display devices. In brief overview, the system includes a mobile computing device 102 that may communicate with one or more external display devices 202a-n. The embodiment in FIG. 2A shows a mobile computing device 102 with a native display 201, although devices without native displays may be used in other embodiments. The mobile computing device 102 executes a plurality of resources 204a-n (collectively, 204). The window management system 206 and virtual graphics driver 208 manage the locations and sizes of the display of output data associated with each of the plurality of resources in a user-configurable display layout. In many embodiments, the mobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to an external display device 202. In some of these embodiments, the mobile computing device 102 transmits the output data upon establishing a connection with the external display device 202. In various embodiments, the mobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to the device's native display 201. In many embodiments, the mobile computing device 102 transmits the output data associated with certain of the plurality of resources 204 to the native display 201 and transmits the output data associated with other of the plurality of resources 204 to the external display devices 202a-n.
[0092] Referring now to FIG. 2 A, and in greater detail, a mobile computing device
102 executes a plurality of resources 204. In one embodiment, the mobile computing device 102 is a client 102 as described above in connection with FIGs. 1A-1C. In another embodiment, the mobile computing device 102 displays the output data associated with a resource 204a in a plurality of resources 204a-n executed by the mobile computing device 102. In some embodiments, the mobile computing device 102 displays the output data associated with each of the plurality of resources 204.
[0093] In one embodiment, a resource in the plurality of resources 204 may include, without limitation, a data file, an executable file, configuration files, an application, a desktop environment (which may itself include a plurality of applications for execution by the user), a
computing environment image (such as a virtual machine image), and/or operating system software or other applications needed to execute a computing environment image.
[0094] In one embodiment, the mobile computing device 102 includes a window management service 206 allowing an external display device 202 to display the output data associated with each of a plurality of resources 204 executed on the mobile computing device 102. In still another embodiment, the window management service 206 allows multiple resources running on the mobile computing device 102 to be viewed on an external display device 202 or the native display 201 at substantially the same time, as opposed to allowing the output of one resource 204 to be viewed exclusively on the native display 201 or external display device 202. In yet another embodiment, the window management service 206, in conjunction with a virtual graphics driver 208, manages the display layout of the windows displayed on the external display device 202 and the native display 201. In some embodiments, the virtual graphics driver 208 is a driver-level component that manages a virtual screen frame buffer storing output data that will be displayed by the native display 201 on the mobile computing device 102 or an external display device 202. In one of these embodiments, the window management service 206, in conjunction with the virtual graphics driver 208, manages the boundaries and size of a screen space used to display output data and on which display device the output data is displayed.
[0095] In some embodiments, an external display device 202 receives output data associated with each of the plurality of resources 204 and displays the output data in a user- configurable display layout. In one embodiment, the external display device 202 includes a dock to which the mobile computing device 102 connects. In another embodiment, the external display device 202 includes a receiver for communicating with the mobile computing device 102 wirelessly, for example, via BLUETOOTH, Wi-Fi or other networking protocols, as described above in connection with FIGs. IA- 1C. In still another embodiment, the external display device 202 is a display device 124 as described above in connection with FIG. lB-lC.
[0096] Referring now to FIG. 2B, a block diagram depicts one embodiment of a system for mapping the display of one or more resources 204 of the mobile computing device 102 on one or more display devices 201 and/or 202. In various embodiments, the window management service 206 manages a virtual screen space 210. The virtual screen space 210 may map to the native display 201 and one or more external display devices 202. The
window management service 206 may position output data associated with the user interfaces of one or more resources 204 on the virtual screen space 210 to specify where each user interface will be displayed. In some embodiments, the window management service 206 positions the output data according to a user preference. In additional embodiments, the window management service 206 positions the output data according to a policy. In various embodiments, the window management service 206 positions the output data based on the resource 204 associated with the output data.
[0097] The window management service 206 communicates with the virtual graphics driver 208 to transmit output data associated with user interfaces of resources 204 to the native display 201 and one or more external display devices 202. In some embodiments, the window management service 206 may transmit output data and associated coordinates from the virtual screen space 210 to the virtual graphics driver 208. In various embodiments, the virtual graphics driver 208 stores the output data in a virtual screen frame buffer. In many embodiments, the virtual graphics driver 208 transmits the entries in the virtual screen frame buffer to the native display 201 and external display devices 202. In many embodiments, the virtual graphics driver 208 transmits an entry in the virtual screen frame buffer to a native display 201 or an external display device 202 based on the position of the entry in the frame buffer.
[0098] Referring now to FIG. 2C, a screen shot depicts one embodiment of a user- configurable display layout in which the external display device 202 displays the output data of the resources 204. In this embodiment, the output data of the resources 204 is displayed in a grid display layout. The grid display layout may include one or more cells in an arrangement. A cell may display output data associated with a resource. In some embodiments, more than one cell displays output data associated with the same resource.
[0099] In some embodiments, the cells are uniformly sized, whereas in other embodiments, the cells have different sizes. The cells may be arranged in any configuration. In some embodiments, the cells may be arranged in rows, columns, or both. A cell may have a descriptor associated with the cell's position in the grid. The descriptor may indicate the position of a cell within a row. In the embodiment depicted in FIG. 2C, the cell for resource 204a may have the descriptor "1-1," the cell for resource 204b may have the descriptor "1-2," the cell for resource 204c may have the descriptor "1-3," the cell for resource 204d may have the descriptor "2-1," and the cell for resource 204d may have the descriptor "2-2." In other
embodiments, the cells may be numbered, e.g. "Cell 1," "Cell 2," etc. However, any system of choosing descriptors known to those of ordinary skill in the art may be used.
[0100] In various embodiments, the window management service 206 configures a grid display layout according to the resources 204 being displayed on the native display 201 or the external display device 202. In some embodiments, the service 206 configures a grid display layout according to the number of resources 204 being displayed. In other embodiments, the service 206 configures a grid display layout according to the size or amount of content in the user interfaces of the resources 204. For example, if an external display device 202 will display four resources with comparable amounts of content, the window management service 206 may configure a grid display layout with four uniform cells. In another example, if an external display device 202 will display four resources and one resource includes three times as much content as the others, the window management service 206 may configure a grid display layout with three uniform cells in a first row and a single cell in a second row. The single cell in the second row may be three times as wide as the cells in the first row. In various embodiments, the window management service 206 may configure a grid display layout to reserve a cell for displaying information about the resources being displayed, such as a menu of the resources. In many embodiments, the window management service 206 may configure a grid display layout to reserve a cell for allowing a user to configure the grid display layout.
[0101] Referring now to FIG. 2D, a screen shot depicts one embodiment of a user- configurable display layout providing a dynamic display layout in which the external display device 202 displays the output data associated with the plurality of resources 204. In this embodiment, windows on the external display device 202 that display output data for resources 204 may be dynamically positioned and sized. The window management service 206 may position a user interface for a resource at a default position and with a default size chosen according to a policy, the resource 204, or any other method. The window management service 206 may order overlapping user interfaces such that higher-order user interfaces obscure lower-order user interfaces. The window management service 206 may transmit output data to the virtual graphics driver 208 reflecting the obfuscation. The user may re-position or re-size a window by, for example, clicking and dragging the window or a window edge. In these embodiments, the virtual graphics driver 208 may detect the user's change to the window, and transmit information about the user's change to the window
management service 206. The window management service 206 may process the change and transmit updated output data to the virtual graphics driver 208. In some embodiments, the user moves the user interface for a resource 204 to any location on a native display 201 or external display device 202. In some embodiments, the user moves the user interface for a resource 204 to a different display device. In some embodiments, the updated output data indicates that one user interface's size has been increased or location has been adjusted to obscure another user interface. In other embodiments, the updated output data indicates that one user interface's size has been decreased or location has been adjusted such that more of another user interface shall be visible.
[0102] Referring now to FIG. 3A, a block diagram depicts one embodiment of virtual screen spaces 210 provided by a mobile computing device attached to one or more external display devices. As shown in FIG. 3 A, the mobile computing device 102 includes a virtual graphics driver 208 and a virtual screen 210. The virtual screen 210 includes a plurality of virtual screen spaces 310 and 312a-n. Virtual screen space 310 may be a native display screen space for the native display 201 on the mobile computing device 102. The other virtual screen spaces 312a-n may be extended screen spaces that correspond to the displays of external display devices 202. The window management service 206 and virtual graphics driver 208 manage the virtual screen 210. In one embodiment, the virtual graphics driver 208 uses a virtual screen frame buffer to manage the mobile computing device's native display 201 and change the native display's 201 screen resolution. In another embodiment, the virtual graphics driver 208 uses a virtual screen frame buffer to manage an extended screen space 312 and to change a resolution of the extended screen 312.
[0103] In some embodiments, the virtual graphics driver 208 allocates and manages a plurality of virtual screen spaces 310, 312a-n and virtual screen frame buffers. In some of these embodiments, each virtual screen space and virtual screen frame buffer has a resolution independent of the other screen spaces and frame buffers. In one of these embodiments, output data associated with each of the plurality of resources 204 can reside within any of the virtual screen spaces 310, 312a-n. In another of these embodiments, each of the extended screen spaces 312a-n is associated with at least one external display device 202, depending on the capabilities of the device.
[0104] In various embodiments, the window management service 206 and the virtual graphics driver 208 allocate and manage the display, on a plurality of external display devices
202, of output data associated with a plurality of resources. In one of these embodiments, for example, output data associated with a resource 204a displays on a mobile computing device 102, output data associated with a resource 204b displays on one external display device 202a, and output data associated with a resource 204c display on another external display device 202b. In another of these embodiments, the window management device 206 identifies one of the external display devices 202 for displaying output data generated by a resource 204a based upon a type of the resource 204a. For example, the window management service 206 may determine that a type of resource rendering a video may display on a television screen, while a type of resource rendering a word processing application may render on a display of a laptop computer.
[0105] Referring now to FIG. 3B, a block diagram depicts one embodiment of a mobile computing device 102 providing a virtual screen 210 with virtual screen spaces 310, 312a-n of varying resolutions. In this embodiment, the virtual screen 210 includes a native display screen space 310 corresponding to the native display 201 of the mobile computing device 102 with a resolution of 320 pixels x 240 pixels. The virtual screen 210 also includes an extended screen 312a corresponding to the display of an external display device 202 with a resolution of 1024 pixels x 768 pixels, an extended screen 312n-l corresponding to the display of an external display device 202 with a resolution of 800 pixels x 600 pixels, and an extended screen 312n corresponding to the display of an external display device 202 with a resolution of 640 pixels x 480 pixels. In many embodiments, the virtual screen 210 may include a native display screen space 310 and any number of extended screens 312 of any resolution. The entire virtual screen space 210 may be mapped into a single virtual screen frame buffer, although embodiments that map into multiple buffers may be used.
[0106] Referring now to FIG. 3C, a block diagram depicts one embodiment of a logical representation of a plurality of virtual screen spaces managed by a virtual graphics driver. In this embodiment, the virtual graphics driver 208 manages multiple virtual screen spaces with different resolutions in a virtual screen frame buffer. In this embodiment, the native display 201 of the mobile computing device is the primary display and the external display device 202, corresponding to the extended screen 312a, is a secondary display. In various embodiments, output data associated with resources 204 on the native display screen space 310 will be displayed on the native display 201 and output data associated with
resources 204 on the extended screen space 312a will be displayed on the external display device 202 associated with the extended screen spaces 312a.
[0107] Referring now to FIG. 4A, a block diagram depicts one embodiment of a graphical user interface for customizing a dynamic display layout. In general overview, the user selects a screen space and sets parameters to configure the space. In this embodiment, the user selects a screen space from interface element 401. In various embodiments, the user may select a screen space by selecting a tab, icon, button, or radio button associated with a screen space, selecting a screen space from a drop-down menu, typing the name of a screen space into a text box, or any other methods known to persons of ordinary skill in the art. In further embodiments, the user may create a new screen space. In this embodiment, the user may create a new screen space by selecting the "Add New Display" button 403, and an interface element 401 corresponding to the new screen space may be created and added to the graphical user interface. The user may name the new screen space, assign an external display device 202 corresponding to the space, and configure the screen space according to the methods described below.
[0108] In various embodiments, the graphical user interface may include interface elements the user manipulates to set parameters for the space. In this embodiment, the graphical user interface includes interface elements 402 and 404 for setting the resolution and mode of the screen space, respectively, and in many embodiments, the graphical user interface may include interface elements for setting any other parameters such as z-order or default position. In this embodiment, the user sets the resolution by selecting a radio button from an interface element 402 corresponding to a predetermined resolution or selecting a radio button to customize the width and height of the screen's resolution. In other embodiments, the user may set the resolution by adjusting a slider to a desired resolution, selecting a thumbnail from a group of thumbnails representing displays of the screen space at different resolutions, or clicking and dragging a boundary of an image representing the screen space to change the resolution. In this embodiment, the user sets the mode by selecting a radio button from an interface element 404 indicating whether the user wishes the screen space to be a primary display, an extended display, a display that mirrors the primary display, or a disabled display, although any other modes may be used. In other embodiments, the user may set the mode by selecting an icon representing the mode, selecting the mode from a drop-down menu, or any other method.
[0109] Referring now to FIG. 4B, a block diagram depicts one embodiment of a graphical user interface for customizing a grid display layout. The user may select the screen space to customize according to any of the methods described in FIG. 4A. In many embodiments, the graphical user interface allows users to configure the number, size, and position of cells within a grid in which the output data for each of the plurality of resources 204 is displayed. In one embodiment, the user may add cells to the grid by selecting the user interface element 406, an "Add Cell" button. Each time the user selects the "Add Cell" button, a new numbered cell appears in the grid, and the user may delete a cell by selecting the cell and the user interface element 407, a "Delete Cell" button. In many embodiments, the user may configure the position, resolution, or both of each cell. In some embodiments, the user may configure the position by dragging and dropping the cell to the desired location. In other embodiments, the user may configure the position by entering a numerical value corresponding to the desired position for the cell. For example, the user may select a cell and input the position "2-1" to indicate the cell should appear in the second row and first column of the grid. In all of these embodiments, the remaining cells may automatically readjust their positions in response to the user's selection of a position for the new cell.
[0110] In various embodiments, the user may configure the resolution of each cell. In some embodiments, the user configures the resolution by dragging and dropping a boundary of a cell to the desired size. In other embodiments, the user configures the resolution by selecting from a group of resolutions. The embodiment shown in FIG. 4B depicts a user interface element 408 in the form of radio buttons corresponding to a group of resolutions, including a radio button that permits the user to customize the resolution of the cell. In other embodiments, a user may select a resolution by selecting a check box, clicking a button, choosing a resolution from a drop-down menu, or any other known method of inputting information.
[0111] In many embodiments, a user may configure the grid display layout prior to the display of output data by the external display device 202. For example, the user may choose a setting associated with a preferred cell position and resolution for each resource in the plurality of resources 204 in a user profile. Upon execution of the resource, the window management service 206 may consult the user profile to determine how to display the user interface of the resource 204. In other embodiments, the user may configure the grid display layout during the display of output data by the external display device 202. For example,
upon execution of a resource 204 by the mobile computing device 102, the user may choose or modify a setting associated with a preferred cell position and resolution for the resource 204.
[0112] In various embodiments, a user may associate output data for a resource 204 with a cell in the grid display layout. In some embodiments, the user may select a cell on the grid display layout and associate a resource with the cell such that output data of the resource appears in the selected cell. In some embodiments, the user may associate a plurality of resources with the same cell and prioritize the resources to determine which output data will be displayed in the cell if more than one of the associated resources are being executed. In other embodiments, the user may associate a resource with a cell by dragging the output data generated by the resource to the cell in the grid display layout. The window management service 206 may store the association between the cell and the resource 204.
[0113] Referring now to FIG. 4C, a screen shot depicts one embodiment of a customized grid display layout. In this embodiment, the grid display layout includes output data for a calendar 420, file explorer 422, and e-mail 424 corresponding to cells 1, 2, and 3 in the grid layout of FIG. 4B. In this embodiment, no resources have been associated with the interface element 430 corresponding to cell 4, an empty cell. In various embodiments, empty cells include a user interface element 430 for selecting a resource whose output data should display in the cell. In some embodiments, the user interface element 430 enumerates at least one resource 204 whose output data is available for display on the external display device 202. In still another embodiment, upon selection of a resource 204 by a user via the user interface element 430, the window management system 206 receives an identification of the selected resource 204. The window management system 206 may associate the selected resource with the cell corresponding to the user interface element 430. In still even another embodiment, the window management system 206, in communication with the virtual graphics driver 208, manages an association between the selected resource and the cell corresponding to the user interface element 430 to output data associated with the resource to the cell in the customizable grid display layout.
[0114] Referring now to FIG. 4D, a screen shot depicts one embodiment of a dynamic display layout provided by a mobile computing device 102 displaying the output data generated by each of a plurality of resources 204 on the native display 201 of the mobile computing device 102. In this embodiment, the output data for all the resources may be
displayed on the native display 201 because the mobile computing device 102 is not attached to any external display devices. In this embodiment, the dynamic layout includes four windows 440, 442, 444, and 446 that contain output data generated by four resources executing on the mobile computing device 102. In various embodiments, a window 440 for a resource 204 may be dynamically displayed on the native display 201 of the mobile computing device 102 when the resource 204 is first executed. In many embodiments, the user may dynamically resize the window 440 by, for example, dragging and dropping a boundary of a window 440 to the desired location. In further embodiments, the user may move the window 440 to any position on the native display 201. In some embodiments, the user may move the window 440 to an external display device 202 connected to the mobile computing device 102. In some embodiments, the user may minimize the window 440 on the native display 201 of the mobile computing device 102. In many embodiments, the user may change focus from a window displaying output data associated with one resource 204a to a window displaying output data associated with another resource 204b. The user may change focus by selecting the window containing output data for the desired resource 204b. When the user changes the focus to resource 204b, the output data generated by resource 204b may obscure output data generated by other resources 204 executing on the mobile computing device 102. In still another embodiment, the windows 440-448 can be adjusted in any manner and/or form on any display, either the native display 201 or external display devices 202, as apparent to one of ordinary skill in the art.
[0115] Referring now to FIG. 4E, a flow diagram depicts one embodiment of the steps taken in a method for displaying, by an external display device 202, output data generated by a resource 204 executing on a mobile computing device 102. In brief overview, the method includes the step of executing, by a mobile computing device 102, a plurality of resources 204 (step 402). The method includes the step of displaying, by the external display device 202, in a user-configurable display layout, the received output data associated with each of the plurality of resources 204 (step 404).
[0116] Referring still to FIG. 4E, and in greater detail, a mobile computing device
102 executes a plurality of resources 204 (step 402). In one embodiment, the mobile computing device 102 substantially simultaneously executes the plurality of resources 204. In another embodiment, the mobile computing device 102 substantially simultaneously executes a subset of the plurality of resources 204.
[0117] The external display device displays, in a user-configurable grid display layout, the received output data generated by each of the plurality of resources 204 (step 404). As described above in connection with FIGS. 2A-2D, in various embodiments, the mobile computing device 102 determines for each of the plurality of resources 204 whether to transfer output data associated with the resource 204 to the external display device 202.
[0118] In some embodiments, a window management service 206 manages the size, position, resolution, or any other characteristic of a resource 204 in the virtual screen space 210. In one embodiment, a window management service 206 executing on the mobile computing device 102, and in communication with a virtual graphics driver 208, identifies the resources 204 whose output data the external display device 202 should display. In another embodiment, the window management service 206 receives, from the virtual graphics driver 208, an identification of an external display device 202 attached to the mobile computing device 102. In still another embodiment, the window management service 206 evaluates each of the plurality of resources 204 to determine whether the external display device 202 should display the output data generated by each of the plurality of resources 204. In still even another embodiment, the window management service 206 evaluates a setting associated with each of the plurality of resources 204 to determine whether the external display device 202 should display the output data generated by each of the plurality of resources 204. In yet another embodiment, the window management service 206 evaluates a user- specified preference setting associated with each of the plurality of resources 204 to determine whether the external display device 202 should display the output data generated by each of the plurality of resources 204.
[0119] In one embodiment, the window management service 206 instructs the virtual graphics driver 208 to forward output data generated by a resource 204 executing on the mobile computing device to the external display device 202 for display. In another embodiment, the window management service 206 instructs the virtual graphics driver 208 to forward the output data generated by the resource 204 to the external display device responsive to determining that a setting associated with the resource 204 indicates that the output data generated by the resource 204 should be displayed on the external display device 202. In still another embodiment, a plurality of settings are associated with the resource 204, each of the plurality of settings indicating whether the output data generated by the resource 204 should be displayed on each of a plurality of types of external display device 202; for
example, a first setting associated with the resource 204 may indicate that the generated output data should be displayed on the external display device 202 if the external display device 202 is a television screen and a second setting associated with the resource 204a may indicate that the generated output data should not be displayed on the external display device 202 if the external display device 202 has a display below a certain size and/or resolution.
[0120] In one embodiment, the window management service 206 instructs the virtual graphics driver 208 to have the output data for a resource 204 displayed on the external display device 202 in a grid display layout. In another embodiment, the window management service 206 instructs the virtual graphics driver 208 to have the output data for a resource 204 displayed on the external display device 202 in a dynamic display layout. In still another embodiment, the window management service 206 evaluates a setting associated with each of the plurality of resources 204 to identify a display layout according to which the external display device 202 should display the output data generated by each of the plurality of resources 204. In yet another embodiment, the window management service 206 evaluates a user- specified preference setting associated with each of the plurality of resources 204 to identify a display layout according to which the external display device 202 should display the output data generated by each of the plurality of resources 204; for example, and in some embodiments, a user specifies a subset of the plurality of resources 204 for which the external display device 202 should display output data.
[0121] In one embodiment, the virtual graphics driver 208 forwards output data generated by a resource 204 executing on the mobile computing device to the external display device 202 for display. In another embodiment, the virtual graphics driver 208 instructs an operating system of the mobile computing device 102 to forward output data generated by a resource 204 executing on the mobile computing device to the external display device 202 for display. In still another embodiment, the virtual graphics driver 208 forwards an identification of a display layout to the external display device 202. In yet another embodiment, the virtual graphics driver 208 instructs an operating system of the mobile computing device 102 to forward an identification of a display layout to the external display device 202.
[0122] Referring now to FIG. 5 A, a block diagram depicts one embodiment of a system for displaying, by an external display device 202, a second user interface for a resource 204 distinct from a first user interface for the resource 204 displayed by the mobile
computing device 102 that is executing the resource 204. The system includes a window management service 206, a virtual graphics driver 208, and resources 204 that may provide more than one user interfaces 502a-n. In some embodiments, the resource 204 may provide a user interface 502a optimized for the native display 201 on the mobile computing device 102. In additional embodiments, the resource 204 may provide a user interface 504b optimized for display on the external display device 202. In further embodiments, the resource 204 may provide user interfaces 502a-n optimized for display on different external display devices 202. In various embodiments, the window management service 206 provides functionality to allow resources to dynamically change user interfaces 502 to adapt to the external display device 202 that will display output data generated by the resource. In some embodiments, the window management service 206 indicates to a resource 204 the properties of the external display device 202 that will display the resource's output data. In response, the resource 204 provides a user interface 502b-n appropriate for the properties of the external display device 202, which may differ from the user interface 502a provided to the native display 201 of the mobile computing device. In various embodiments, the mobile computing device 101 may execute a plurality of resources 204, and each resource may provide a plurality of user interfaces 502 optimized for display on the native display 201 of a mobile computing device 102 or on an external display device 202.
[0123] Referring now to FIG. 5B, a block diagram depicts one embodiment of a system for mapping a first user interface 502a on the native display 201 of the mobile computing device 102 to a second user interface 502b on one or more external display devices 202. In one embodiment, the virtual graphics driver 208 maps all the screen spaces 210a-n from the display of the mobile computing device 102 and/or the external display devices 202 into one virtual screen space 210. In another embodiment, the virtual graphics driver 208 communicates the coordinates related to the virtual screen space 210 to the window management service 206. In still another embodiment, the window management service 206 positions user interfaces of one or more resources 204 onto the virtual screen space 210. In yet another embodiment, the window management services 206 uses information provided by the virtual graphics driver 208 to position the user interfaces on an external display device 202. In one embodiment, the user interface 502b for the resource 204 displayed by an external display device 202 is substantially same as the user interface 502a displayed on the native display 201. In another embodiment, the user interface 502b displayed by an external display device 202 is distinct from the user interface 502a displayed
on the native display 201. In various embodiments, the user interface for any resource 204 displayed on an external display device 202 is distinct from the user interface 510 displayed on a different external display device 202.
[0124] Referring now to FIG. 5C, a screen shot depicts one embodiment of a system for displaying, by an external display device 202, a second user interface 502b for a resource 204 distinct from a first user interface 502a for the resource 204 displayed by a mobile computing device 102 that is executing the resource 204. As depicted in FIG. 5C, and in many embodiments, a resource 204 may generate a plurality of user interfaces 502a and 502b, each of which may be optimized for use with a different type of display device. In one embodiment, a first user interface 502a of the plurality of user interfaces may be optimized for use with the native display 201 of a mobile computing device 102. In another embodiment, the second user interface 502b of the plurality of user interfaces may be optimized for use with an external display device 202. In various embodiments, the first user interface 502a may be displayed on the native display 201 of the mobile computing device 102 until the mobile computing device 102 is attached to an external display device 202. When the mobile computing device 102 is attached, the second user interface 502b may be displayed on the external display device 102. In some embodiments, the mobile computing device 102 may cease to display the first user interface 502a upon attachment, and in other embodiments, the mobile computing device 102 may continue to display the first user interface 502a as the external display device 102 displays the second user interface 502b.
[0125] Referring now to FIG. 6, a flow diagram depicts an embodiment of the steps taken in a method for displaying, by an external display device, a first user interface to an application distinct from a second user interface to the application displayed by a mobile computing device executing the application. In brief overview, the method includes the step of generating, by a resource executing on a mobile computing device, a first user interface for display on the mobile computing device (step 610). The method includes the step of receiving, by the resource, from a window management service, an identification of an external display device (step 612). The method includes the step of generating, by the resource, a second user interface for display on the external display device, responsive to receiving the identification of the external display device (step 614).
[0126] Referring now to FIG. 6, and in greater detail, a resource executing on a mobile computing device generates a first user interface for display on the mobile computing
device (step 610). In one embodiment, the resource 204 generates a user interface 502a optimized for use with the mobile computing device 102. The resource receives, from a window management service, an identification of an external display device (step 612). In one embodiment, the resource receives the identification of the external display device 202 via a virtual graphics driver 208.
[0127] In response to receiving the identification of the external display device 202, the resource generates a second user interface 502b for display on the external display device (step 614). In one embodiment, the window management service 206 receives an identification of an external display device 202 to which the mobile computing device 102 has attached. In some embodiments, the window management service 206 transmits, to the resource, an application programming interface (API) call indicating to the resource that the external display device 202 is available. The window management service 206 may inform the resource 204 executing on the mobile computing device of the identification of the attached external display device 202. In some embodiments, the window management service 206 indicates to the resource 204 that the external display device 202 has a display with a larger resolution or size than the native display 201 on the mobile computing device 102. In further embodiments, the window management service 206 allows the resource to change its user interface to an interface more optimized for use with the external display device 202.
[0128] In one embodiment, the window management service receives, from the resource, an identification of a user interface associated with the attached external display device. In some embodiments, the window management service 206 receives the identification via a virtual graphics driver 208. In other embodiments, the window management service 206 forwards the identification to the virtual graphics driver 208.
[0129] In one of these embodiments, the window management service 206 receives, from the resource, a user interface associated with the attached external display device 202. In another embodiment, when a user interface associated with the attached external display device is not available, the window management service 206 receives, from the resource, a default user interface. In still another embodiment, the window management service 206 receives from the resource a communication that a user interface associated with the attached external display device 202 is not available.
[0130] In one embodiment, the resource 204 generates a user interface 502b optimized for use with an external display device 202 to which the mobile computing device 102 is attached. In another embodiment, the resource 204 instructs an operating system on the mobile computing device 102 to display one of a plurality of user interfaces selected in response to receiving an identification of the external display device 202 upon which the user interface will be displayed. In still another embodiment, the resource 204 instructs the window management system 206 on the mobile computing device 102 to display one of the plurality of user interfaces selected responsive to receiving an identification of the external display device 202 upon which the user interface will be displayed. In some embodiments, the window management system 206 transmits, to the external display device 202, an identification of a user interface 504 with an instruction to display the user interface 504. In other embodiments, the window management system 206 transmits, to a virtual graphics driver 208, an identification of a user interface 504 with an instruction to display the user interface 502b on the external display device 202.
[0131] Referring now to FIG. 7 A, a block diagram depicts one embodiment of a system for executing and displaying a plurality of resources 204a-204n (204 in general) on a client 102b. In one embodiment, the system includes a client 102a, which is a mobile computing device, in connection with another client 102b. In one embodiment, the client 102b includes a dock to which the mobile computing device 102a attaches. In another embodiment, the client 102b includes a receiver for communicating with the mobile computing device 102a wirelessly, for example, via BLUETOOTH, Wi-Fi or other networking protocols, as described above in connection with FIGs. IA- 1C. In still another embodiment, the client 102b includes an external display 124 as described above in connection with FIG. 1B-1C.
[0132] In one embodiment, one or more resources 204 execute on the mobile computing device 102a. In another embodiment, the mobile computing device 102a accesses hardware and/or software components of the client 102b to execute the resources 204. In still another embodiment, a version of a resource 204 is executed on the client 102b. In one embodiment, the version of the resource 204 on the client 102b can be a local copy of the resource 204 installed on the client 102b. In another embodiment, the version on the client 102b can offer more functionalities of the resource 204 than available on the mobile computing device 102a. In still another embodiment, the version on the client 102b can be
another resource 204b producing the same output data as the resource 204 executing on the mobile computing device 102a. In yet another embodiment, the mobile computing device 102a and the client 102b share hardware and software resources to execute one or more resources 204.
[0133] Execution of the resource 204 may be transferred to the client 102b in many ways. In one embodiment, the execution of the resource 204 is terminated on the mobile computing device 102a and the version of the resource 204 on the client 102b is started. In another embodiment, information and/or data values related to the state of execution of the resource 204 on the mobile computing device 102a is transferred to the client 102b. In still another embodiment, no information and/or data values related to the state of execution of the resource 204 on the mobile computing device 102a is transferred to the client 102b. In such an embodiment, recent updates are not transferred to the client 102b when the execution is transferred from the mobile computing device 102a to the client 102b. In yet another embodiment, the resource 204 is transferred to the client 102b when the execution is transferred from the mobile computing device 102a to the client 102b. In one embodiment, information and/or data values related to the state of execution is also transferred when the resource 204 is transferred to the client 102b.
[0134] In some embodiments, the client 102b includes an operational or performance characteristic not provided by the mobile computing device 102a. In one of these embodiments, the client 102b has a more powerful processor and/or larger memory than the processor and memory of the mobile computing device 102a. In another of these embodiments, the client 102b provides an I/O device, display device, installation device, or other peripherals, such as a keyboard or printer not available to the mobile computing device 102a. In still another of these embodiments, the client 102b may provide a feature, a resource, or peripheral desired to be used by the user of the mobile computing device 102a For example, the user may want to access a file or an application provided on a remote machine available via a connection across a network. In yet another of these embodiments, the client 102b provides access to machines on a network 104, such as those in machine farm 38, not available to the mobile computing device 102a, or to a user of the mobile computing device 102a.
[0135] In one embodiment, one or more resources 204 may execute in a virtual machine on the mobile computing device 102a. In another embodiment, a virtual machine
executing on the mobile computing device 102a provides access to a computing environment based at a remote location. In still another embodiment, an application program stored in the mobile computing device 102a executes to access data associated with the computing environment provided on the mobile computing device 102a. In another embodiment, the mobile computing device 102a executes virtualization software, at least a portion of which is stored on the mobile computing device 102a. In still another embodiment, the mobile computing device 102a provides access to a computing environment by executing an operating system with access to one or more applications stored on the mobile computing device 102a, the operating system and the one or more applications having access to user data stored in the mobile computing device 102a.
[0136] In one embodiment, the mobile computing device 102a executes a virtual machine, responsive to data stored in the mobile computing device 102a. In another embodiment, the mobile computing device executes a virtual machine responsive to a policy stored in the mobile computing device 102a. In still another embodiment, the mobile computing device 102a executes a virtual machine that provides access to a requested resource or computing environment, the virtual machine executed responsive to a virtual machine image stored in the mobile computing device 102a. In yet another embodiment, the mobile computing device 102a transfers execution of the virtual machine to the client 102b.
[0137] In another embodiment, the client 102b connects to the mobile computing device 102a, executes a virtual machine, and provides access to a computing environment responsive to data stored in the mobile computing device 102a. In one embodiment, the client 102b may mount the storage device of the mobile computing device 102a as a removable hard drive or storage element 128 of the client 102b. In some embodiments, the mobile computing device 102a may be a plug and play device (PnP) of the client 102b, such that a PnP protocol such as that manufactured by Microsoft Corporation of Redmond, Washington, is used between the mobile computing device 102a and client 102b, such as via I/O devices 130a- 130n or a network interface 118.
[0138] In one embodiment, the client 102b provides access to a computing environment based on a portable computing environment provided in the mobile computing device 102a. The client 102b executes a virtual machine and a virtualization layer to execute the computing environment based on a virtualization software, virtual machine image, or user
data. In some embodiments, the client 102b includes a transceiver for accessing data stored in the mobile computing device 102a.
[0139] In some embodiments, a loading mechanism on the mobile computing device
102a actuates the establishment of the computing environment on the client 102b based on the portable computing environment stored in the mobile computing device 102a. In other embodiments, the loading mechanism of the client 102b actuates the establishment of the computing environment. In yet another embodiment, a loading mechanism on the mobile computing device 102a works in conjunction with the loading mechanism of the client 102b to establish the computing environment.
[0140] Referring now to FIG. 7B, a block diagram depicts one embodiment of a system for executing resources 204 on a remote server 106 and displaying output data on a client computing device 102b. In one embodiment, the system includes the mobile computing device 102a, a client 102b and a server 106. The mobile computing device 102a may be attached or connected to the client 102b in a plurality of possible ways as described with reference to FIG. 7A.
[0141] In one embodiment, the client 102b connects to a remote server 106 to request execution of a resource 204 by the server 106. In some embodiments, the server 106 may have an operational or performance characteristic not present in the client 102b. In another embodiment, the server 106 has more powerful hardware and/or software resources not available to the client 102b. In still another embodiment, the server 106 provides access to machines on a network 104, such as those in machine farm 38, not directly available to the client 102b.
[0142] In one embodiment, one or more resources 204 execute on the client 102b. In another embodiment, the one or more resources 204 may include an application program to establish a remote connection with the server 106. The client 102b may execute, operate or otherwise provide the application, which can be any type and/or form of software, program, or executable instructions such as any type and/or form of web browser, web-based client, client-server application, a thin-client computing client, an ActiveX control, or a Java applet, or any other type and/or form of executable instructions capable of executing on the client 102. In some embodiments, the application may be a server-based application executed on behalf of the client 102 on a server 106. In one embodiment, the server 106 may
communicate with the client 102 using a presentation layer protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Washington.
[0143] In one embodiment, one or more resources 204 may execute in a virtual machine on the client 102b as described above in connection to FIG. 7A. In another embodiment, the server 106 connects to the client 102b, executes a virtual machine, and provides access to a computing environment. In one embodiment, the server 106 may transmit the virtual machine image to the client 102b. In another embodiment, the server 106 transmits output data generated by the virtual machine image executed on the server 106 to the client 102b. In still another embodiment, the virtual machine image includes a version of each of the plurality of resources 204 of the mobile computing device 102a.
[0144] Output data from the execution of the resources 204 are displayed in one or more ways. In one embodiment, the output data generated by an execution of a resource 204a on the server 106 is displayed on the mobile computing device 102a. In another embodiment, the output data generated by an execution of a resource 204 on the server 106 is displayed on the client 102b. In still another embodiment, the output data from the server 106 is displayed on the client 102b as a part of a computing environment. In yet another embodiment, a resource 204 is executed locally on the mobile computing device 102a and displayed on the client 102b. In some embodiments, output data is simultaneously displayed in the mobile computing device 102a and the client 102b.
[0145] Referring now to Fig. 8A, a block diagram depicts one embodiment of a system for executing resources for a mobile computing device remotely on a server and displaying output data generated by the resources on the mobile computing device, the client, or both. In one embodiment, the system includes a mobile computing device 102a in communication with a client 102b over a connection 104. The system also includes a server 106 communicating with the mobile computing device 102a over a connection 104' and the client 102b over a connection 104".
[0146] In one embodiment, one or more resources 204 executes on the mobile computing device 102a. In another embodiment, the one or more resources 204 may include an application program to establish a connection 104 with the client 102b and/or a connection
104' with the server 106. In one embodiment, the mobile computing device 102a or the client 102b may execute, operate or otherwise provide the application, which can be any type and/or form of software, program, or executable instructions such as any type and/or form of web browser, web-based client, client-server application, a thin-client computing client, an ActiveX control, or a Java applet, or any other type and/or form of executable instructions capable of executing on the mobile computing device 102. In some embodiments, the application may be a server-based or a remote -based application executed on behalf of the mobile computing device 102a or the client 102b on a server 106. In one embodiments the server 106 may communicate with the mobile computing device 102a or the client 102b using any presentation layer protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Florida or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Washington.
[0147] In one embodiment, one or more resources 204 may execute in a virtual machine on the mobile computing device 102a as described with respect to FIG. 7A. In one embodiment, the mobile computing device 102a transfers execution of the virtual machine to the client 102b and/or the server 106. In another embodiment, the mobile computing device 102a transfers execution of a resource 204 in a virtual machine to the client 102b and/or the server 106. In one embodiment, when the mobile computing device 102a attaches to the client 102b, the mobile computing device 102a may transmit to the server 106 an identification of the attached client 102b. The server 106 may identify a virtual machine on the client 102b. In some embodiments, the server 106 may disconnect a resource 204 executing in a virtual machine on the mobile computing device 102a and connect the resource 204 to the virtual machine on the client 102b. In these embodiments, the server 106 may transmit information regarding the state of execution of the resource to the virtual machine on the client 102b. In other embodiments, the server 106 may transfer execution of a resource executing in a virtual machine by disconnecting the virtual machine from the mobile computing device 102a and connecting the virtual machine to the client 102b.
[0148] In one embodiment, the server 106 provides access to a computing environment to the client 102b. An example of such a computing environment is the XenDesktop distributed by Citrix Inc. of Ft. Lauderdale, Florida. In another embodiment, the output of the resource 204 that is executed on the server 106 for the mobile computing device
102a is displayed as a part of the computing environment on the client 102b through the connection 104". In still another embodiment, the output of the resource 204 executing on the server 106 for the mobile computing device 102a is displayed on the client 102b through the connection 104" but not as a part of the computing environment.
[0149] Referring now to Fig. 8B, a flow diagram depicts one embodiment of the steps taken in a method for executing resources for a mobile computing device remotely on a server and displaying output from the resources on a client device. The method includes the step of connecting, by an agent on a mobile computing device 102a, to a server 106 via a first session to execute and display output data of a resource 204 executed on the server 106 (step 810). The method also includes the step of connecting by the agent on the mobile computing device 102a with a client 102b via a second session (step 812). The method further includes the step of establishing, by an agent executing on the client 102b a third session with the server 106 (step 814). In one embodiment, the server 106 displays a computing environment on the client 102b. In another embodiment, the computing environment includes a display of output generated by an execution, on a remote server 106, of the resource 204.
[0150] In one embodiment, an agent or application program on the mobile computing device 102a connects to a server 106 to establish the connection 104' (step 810). In another embodiment, a first session is established between the mobile computing device 102a and the server 106 for the mobile computing device 102a to access resources 204 on the server 106. In still another embodiment, the mobile computing device 102a executes a resource 204 on the server 106 via the first session. In yet another embodiment, the output of the execution is displayed on the mobile computing device 102a. In a further embodiment, the output of the execution is transmitted to a client 102b.
[0151] The agent on the mobile computing device 102a connects with a client 102b via a second session over connection 104 (step 812). In one embodiment, the second session is set up by the agent that sets up the first session. In another embodiment, the second session is set up by an agent separate from the agent that sets up the first session. In one embodiment, the first session and the second session function according to a substantially same protocol. In another embodiment, the first session and the second session function according to separate protocols. In still another embodiment, the connection between the mobile computing device 102a and the client 102b may be chosen from a plurality of possible connections as described with reference to FIG. 5A.
[0152] In one embodiment, an agent on the client 102b establishes a third session between the client 102b and the server 106 over the connection 104" (step 814). In another embodiment, the server 106 transmits to the client 102b a computing environment over the connection 104". In an example embodiment, the computing environment is a XenDesktop distributed by Citrix Systems Inc. of Ft. Lauderdale, Florida. In one embodiment, the output data generated by the execution of the resource 204 is incorporated into a display of the computing environment as described in greater detail in connection with FIGs. 10-11. In another embodiment, the output data generated by the execution of the resource 204 is transmitted from the server 106 to the client computing device 102b over the connection 104". In still another embodiment, the output data generated by the execution of the resource 204 is displayed on the client 102b separately from the computing environment.
[0153] In one embodiment, the resource 204 is executed on the mobile computing device 102a and output data generated by the execution of the resource 204 is transmitted to the server 106 over connection 104' to be forwarded to the client 102b over the connection 104" for display. In another embodiment, execution of the resource is transferred or relocated to the client 102b. In still another embodiment, output data generated by the resource 204 is displayed inside the computing environment. In yet another embodiment, the execution of the resource 204 is transferred to the server 106. In one embodiment the output data from the execution on the server 106 is transmitted to the mobile computing device 102a and/or the client 102b.
[0154] FIG. 9 A depicts an embodiment of a system for displaying and activating disconnected sessions between a mobile computing device and a server. In FIG. 9A, a block diagram depicts a system 900 including the mobile computing device 102 and the server 106, although any type of client may be used in place of the mobile computing device 102. In one embodiment, the mobile computing device 102 includes an input module 908, a client process 910, a network module 912, and a display module 924. The input module 908 provides an interface for a user of the mobile computing device 102 to interact with the mobile computing device 102, for example, to request the remote execution of an application 916 in a session 918 from the server 106.
[0155] Referring now to FIG. 9A, now in greater detail, a server 106 for displaying and activating disconnected sessions includes a network module 920, a data store 926, and a server process 922. The network module 920 receives authentication information associated
with a user of a mobile computing device 102. The data store 926 enumerates at least one disconnected session associated with the user. The server process 922 transmits, to the mobile computing device 102, a graphical representation of the at least one disconnected session associated with the user. The server process 922 receives a request to connect to the at least one disconnected session. The server process 922 connects the mobile computing device 102 to the requested at least one disconnected session enumerated in the data store 926.
[0156] In one embodiment, a session 918 is a process, operating on the server 106, that supports the execution of one or more applications 916. In some embodiments, execution of a session 918 includes execution of a desktop application 916 from which the execution of other applications 916 can be initiated. In other embodiments, a session 918 is a session between a mobile computing device 102 and a hosted desktop provided by a virtual machine executing on a server 106. In one of these embodiments, a desktop session 918' executes within the session 918. In another of these embodiments, one or more applications 916 execute within a desktop session 918' executing within the session 918. In still other embodiments, a session 918' may execute within a second session 918. In one of these embodiments, for example, a virtual machine on a server 106 executes a session 918 and a second session 918' - such as a desktop session within which a plurality of applications 916 executes - executes within the session 918. In yet another embodiment, the session 918 includes an instance of the execution of a single application 916.
[0157] In one embodiment, the input module 908 is, for example, a graphical user interface that provides one or more icons or menu selections for a user to select. In another embodiment, at least one icon or menu selection represents a specific application 916 available for remote execution. In still another embodiment, selecting an icon or menu selection initiates the transmittal of a log-on request to the server 106 for access to that application 916. In still even another embodiment, an icon or menu selection does not represent any specific application 916, but instead represents a general server 106 log-on procedure. In yet another embodiment, the input module 108 is non-graphical user interface. In this embodiment, the user can enter a command to send a log-on request to server 106. Entering a command can include typing a predefined set of characters or depressing a specified key sequence on an input device (e.g., a keyboard or keypad). In one embodiment, the log-on request includes user-provided authentication information. In another
embodiment, the input module 908 accepts the input of the user-provided authentication information, which can include any type of authentication information, including, without limitation, any of user name-password/PIN combinations, voice samples, one-time passcodes, biometric data, digital certificates, or smart card data. In some embodiments, the input module 908 is in communication with additional hardware peripherals to facilitate acceptance of user authentication information.
[0158] Information associated with a user may be transmitted from the mobile computing device and received by the server. In one embodiment, the input module 908 accepts authentication information and provides it to the client process 910. In another embodiment, the client process 910 manages the mobile computing device-side functionality of the remotely-executing session. In still another embodiment, the client process 910 forwards user input including the authentication information and requests for termination or disconnection of sessions 918 to the server 106. In still even another embodiment, the server 106 receives the information associated with a user of a mobile computing device 102 and authenticates the user responsive to the information. In yet another embodiment, the client process 910 processes data received from the server 106, for example, by forwarding graphical output data generated in a session 918, or a graphical representation of output data from a session 918, to the display module 924.
[0159] The network module 912 provides for communication between a mobile computing device 102 and the server 106. The network module 920 provides communication functionality for the server 106. In one embodiment, the network module 912 sends user input, such as authentication information and requests for access to, disconnection from, or termination of sessions 918 executing on the server 106. In another embodiment, the network module 912 also receives output data from the sessions 918 and forwards the output data to the client process 910. In still another embodiment, the network module 912 encapsulates user input into, and reconstitutes session output data from, a predetermined protocol for transmission to the server 106. In yet another embodiment, the network module 912 encrypts outgoing transmissions and decrypts incoming transmissions.
[0160] In some embodiments, the network module 920 receives authentication information associated with a user of a mobile computing device 102. In another embodiment, the network module 920 receives communications from the mobile computing device 102 over one or more data networks or links 921. In still another embodiment, the
network module 920 transmits output data to the mobile computing device 102. In still even another embodiment, the network module 920 encrypts outgoing communications and decrypts incoming communications. In one embodiment, the network module 920 of the server 106 communicates with the network module 912 of a mobile computing device 102 over a network 104. In another embodiment, incoming communications, once decrypted or retrieved from a protocol (if necessary), are forwarded to a session 918 or to the server process 922, as appropriate.
[0161] In some embodiments, the network module 920 encapsulates outgoing communications in a protocol for transmission and retrieves incoming data from transmissions received according to a communications protocol. In one of these embodiments, network module 920 uses at least one communication protocol to encapsulate data. In another of these embodiments, a first communication protocol, capable of encapsulating secondary protocols used in communications between the mobile computing device and the host service, ensures that data is maintained during a disrupted network connection. In still another of these embodiments, data communicated between the mobile computing device and the host service is buffered. When, for example, a mobile computing device roams between different access points in the same network, the buffered data is maintained during the temporarily disrupted network connection. Similarly, in another example, when a mobile computing device switches between networks (e.g., from a wired network to a wireless network) the buffered data is maintained during the temporarily disrupted connection to the host service. In still even another of these embodiments, buffered data can also be maintained, for example, when the network connection is disrupted due to a failure of a server side component (e.g., a failure of a server side proxy), due to a time-out in the system, or due to other reasons. In yet another of these embodiments, the network module 920 provides session persistence and reliability by encapsulating secondary protocols within the first communication protocol.
[0162] In one embodiment, at least one previously disconnected application session already associated with the user is identified responsive to the authentication information. In another embodiment, at least one previously disconnected desktop session already associated with the user is identified responsive to the authentication information. In some embodiments, the mobile computing device 102 receives output data generated by the at least one previously disconnected session. In one of these embodiments, an application executing
within the at least one previously disconnected session generates the output data. In another of these embodiments, the mobile computing device 102 generates a graphical representation of the at least one previously disconnected session using the received output data. In still another of these embodiments, the mobile computing device 102 receives a graphical representation of the output data.
[0163] The mobile computing device 102 may receive and display a graphical representation representing at least one previously disconnected session associated with the user. In one embodiment, the display module 924 displays the graphical representation of the at least one previously disconnected session to a user of the mobile computing device 102. In some embodiments, output data is displayed to the user. In other embodiments, a graphical user interface is displayed to the user with the received graphical representation. In still other embodiments, the received graphical representation incorporates a graphical user interface element. In yet other embodiments, a text-based representation of output data generated by a disconnected session is displayed to the user.
[0164] In one embodiment, the display module 924 displays the output data generated by an application 916 or a session 918 from a remotely-executing session 918. In another embodiment, the display module 924 forwards output data received from the client process 910 directly to a display device, such as the display device 124 described above in connection with FIGs. IB and 1C, or other suitable form of display device. In some embodiments, the received output data is encrypted, encapsulated in a protocol, or both. In one of these embodiments, the display module 924 first manipulates the output data so that the output data can be interpreted by a standard display adapter such as a computer video card.
[0165] A user may request to connect to at least one previously disconnected session, and the request may be transmitted to the server. In one embodiment, a user of the mobile computing device 102 selects a graphical representation to which to connect. In another embodiment, the mobile computing device 102 transmits the request to connect responsive to selection of a corresponding graphical representation selected by a user. Output data generated by the at least one previously disconnected session associated with the user is received in response to the transmitted request. The output data generated by the session, and by applications executing within the session, are received and displayed as described below.
[0166] The server process 922 manages the execution and termination of sessions 918 and the connections and disconnections of those sessions 918 to the mobile computing device 102. In one embodiment, the server process 922 can initiate new sessions 918, disconnect a mobile computing device 102 from a session 918, detect a mobile computing device 102 disconnection from a session 918, locate a session 918 from which a user has disconnected, locate a session 918 to which a user of a mobile computing device 102 is connected to a client computing device 102b, receive a request to connect to a disconnected session, and connect a user to a disconnected session 918. In another embodiment, the sessions 918 are configured with a user's personal preferences and authorization privileges.
[0167] The output transmitter 924 transmits output data from a session 918 to a mobile computing device 102 through the network module 920. In one embodiment, the output transmitter 924 intercepts the output data generated in a session 918 and determines which mobile computing device 102 is connected to the session 918. If the session 918 is connected to a mobile computing device 102, the output transmitter 924 transmits the output data to the connected device via the network module 920. In one embodiment, if the session 918 is not connected to a mobile computing device 102, the output transmitter 924 discards the output data and waits to receive future output data. In another embodiment, if the session 918 is not connected to a mobile computing device 102, the output transmitter 924 disregards all further output data until the output transmitter 924 receives notification that a session 918 has connected to a mobile computing device 102. In some embodiments, as described in further detail below, if the sessions 918 are not connected to a mobile computing device 102, the output transmitter 924 transmits, to a mobile computing device 102, a graphical representation of at least one disconnected session associated with the user.
[0168] In one embodiment, the output transmitter 924 stores the data until the output transmitter 924 receives notification that the session 918 has connected to a mobile computing device 102. In another embodiment, the output transmitter 924 attempts to send output data to a mobile computing device 102 until the server process 922 notifies the output transmitter 924 that the mobile computing device 102 is disconnected from the server 106. In still another embodiment, the output transmitter 924 determines which of the mobile computing device 102 or other client, if any, the session 918 is connected to by consulting the data store 926. In yet another embodiment, the server process 922 determines to which of the
mobile computing device 102 or other client, if any, the session 918 is connected by consulting the data store 926 and transmits the determination to the output transmitter 924.
[0169] At least one disconnected session already associated with the user is identified in response to the information. In one embodiment, the at least one disconnected session already associated with the user continues to execute at least one application. In another embodiment, the at least one disconnected session already associated with the user continues to execute a desktop session. In still another embodiment, a server 106 identifies the at least one disconnected session.
[0170] In one embodiment, a first session executing on a first server is identified. In some embodiments, a second session executing on a second server is identified. In one of these embodiments, one of the first session and the second session is identified as a disconnected session. In other embodiments, a second session executing on the first server is identified. In one of these embodiments, one of the first session and the second session is identified as a disconnected session.
[0171] The data store 926 includes information related to sessions 918 initiated by users. In one embodiment, the data store 926 is stored in volatile or non-volatile memory. In another embodiment, the data store 926 is distributed through multiple servers. Table 1 shows the data included in a portion of an illustrative data store 926.
Table 1
[0172] The illustrative data store 926 in Table 1 includes data associating each session 918 with the user that initiated the session 918, an identification of the mobile
computing device 102 or the client computing device 102b, if any, from which the user is currently connected to the server 106, and the internet protocol (IP) address of the mobile computing device 102 or the client computing device 102b. The illustrative data store 926 also includes the status of each session. A session 918 status can be, for example, "active" (meaning a user is connected to the session 918), or "disconnected" (meaning a user is not connected to the session 918). In another embodiment, a session status can also be set to "executing-disconnected" (meaning the user has disconnected from the session 918, but applications in the session 918 are still executing), or "stalled-disconnected" (meaning the user is disconnected and applications 916 in the session 918 are not executing, but their operational state immediately prior to the disconnection has been stored). The data store 926 further stores information indicating the applications 916 that are executing within each session 918 and data indicating a process associated with each application 916. In one embodiment, where the server 106 resides in a server farm, the data store 926 also includes the data in the last two rows of Table 1 that indicate on which server in the server farm each application 916 is or was executing, and the IP address of that server. In other embodiments, the data store 926 includes a status indicator for each application 916 in each session 918.
[0173] For example, and referring to Table 1, three sessions 918 exist, Session 1,
Session 9, and Session 3. Session 1 is associated with User 1, who is currently using terminal 1. Terminal one's IP address is 172.16.2.50. The status of Session 1 is active, and in Session 1 , a word processing program is being executed on Server A as process number 1. Server A's IP address is 172.16.2.55. Session 2 in Table 1 is an example of a disconnected session 918. Session 2 is associated with User 2, but Session 2 is not connected to a mobile computing device 102. Session 9 includes a database program that is executing on Server A, at IP address 172.16.2.55 as process number 3. Session 3 is an example of how a user can interact with sessions 918 operating on different servers 106. Session 3 is associated with User 1, as is Session 1. Session 3 includes a spreadsheet program that is executing on Server B at IP address 172.16.2.56 as process number 2, whereas the session 918 included in App Session 1 is executing on Server A.
[0174] In one embodiment, the server 106 also includes a rules source 928. In another embodiment, the rules source 928 stores rules specifying a policy applicable to a user requesting access to a session 918, or to an application 916 in a session 918. In still another embodiment, the rules stored in the rules source 928 are specified at least in part by the
system administrator. In still even another embodiment, a user specifies at least some of the rules stored in the rules source 928. In this embodiment, the user-specified rule(s) may be referred to as preferences. In yet another embodiment, the rules source 928 can be stored in volatile or non- volatile memory or distributed through multiple servers.
[0175] In one embodiment, a rule stored in the rule source 928, for example, might require or forbid automatic connection to disconnected sessions 918. In another embodiment, a rule might require or forbid automatic connection to active sessions 918 currently connected to a different mobile computing device 102. In still another embodiment, a rule might make a connection contingent on the mobile computing device 102 residing within a secure network. In still even another embodiment, a rule might only allow connection to sessions 918 after receiving user approval. In still another embodiment, a rule might only allow connection for a predetermined time after disconnection. In yet another embodiment, a rule may only allow connection to sessions 918 that provide access to specific applications 916.
[0176] In some embodiments, the authentication module 930 authenticates a user that attempts to log on to the server 106. In one embodiment, the authentication module 930 receives user-provided authentication information transmitted from the mobile computing device 102. In another embodiment, the authentication module 930 authenticates the user based on user-provided authentication information. In yet another embodiment, the authentication module 930 transmits, responsive to a successful authentication, the results of the authentication process (e.g., allow or deny access, the user's system ID, mobile computing device ID, user access permissions, etc.) to the server process 922.
[0177] Unintentional termination of sessions 918 resulting from imperfect network connections or users' failure to terminate their sessions 918 themselves can lead to user difficulties. In one embodiment, these difficulties are addressed by differentiating disconnection (which is treated as if the user is not done working with a session 918) from termination (which is assumed to be an intentional completion of the session) and by correlating sessions 918 with users as opposed to correlating sessions with client computers. In some embodiments, when a user is finished executing an application 916 operating in a session 918, the user can terminate a session 918. In one of these embodiments, termination generally involves the affirmative input of the user indicating that the server should no longer maintain the session 918. In another of these embodiments, affirmative user input can
include selecting an "Exit" option from a menu, clicking on an icon, or entering a termination command into a command- line interface. In still another of these embodiments, in response to receiving a termination request, the server process 922 terminates the execution of the session 918 and of any application 916 within that session 918 is halted. In another of these embodiments, data related to the session 918 is also removed from the data store 926.
[0178] In other embodiments, disconnection, either intentional or unintentional does not result in termination of sessions 918. In one of these embodiments, the application or applications operating in a session 918 are executing on the server 106 and a connection to the mobile computing device 102 is not usually necessary to continue execution of the applications 916. In another of these embodiments, the applications 916 continue to execute while waiting for a user to connect to the session 918. In still another of these embodiments, upon disconnection of a user, the server process 922 stalls the execution of the applications 916 operating in the session 918. In this embodiment, the server process 922 halts further execution of the applications 916, and the server process 922 stores the operational state of the application 916 and any data the application 916 is processing. In still even another of these embodiments, the server process 922 can selectively stall execution of specific applications 916 after a user disconnects. For example, and in one embodiment, the server continues execution of an application 916 for a fixed time period, and if a user fails to connect within that time period, the server process 922 stalls the application 916. In yet another of these embodiments, the server process 922 stalls specified sessions 918 that cannot continue executing without user input. In other embodiments, the server process 922 updates a data record associated with the application 916 or with the session 918 to include an identification of the status of the application or session.
[0179] In some embodiments, the server process 922 continues execution of the application 916 while the session 918 remains disconnected from the mobile computing device 102. In other embodiments, if the user of a mobile computing device 102 disconnects from the server 106 and then connects to the server 106 while operating the mobile computing device 102, the client computing device 102b, or a second client computing device 102n (not shown), the server process 922 can connect the user to one or more previously initiated, non-terminated session(s) 918 associated with the user, and reinitiate execution of any stalled applications 916.
[0180] In one embodiment, the server process 922 detects a disconnection. A user can intentionally and manually instruct the server to disconnect a session 918 from the mobile computing device 102. For example, in one embodiment, sessions 918 provide a menu option for disconnection (as distinguished from termination above) that a user can select. The server process 922 can also detect an unintentional disconnection. For example, in one embodiment, the network module 920 of the server 106 informs the server process 922 when a predetermined number of data packets transmitted by the network module 920 to a mobile computing device 102 have not been acknowledged by the mobile computing device 102. In another embodiment, the mobile computing device 102 periodically transmits a signal to the server 106 to confirm that a connection is still intact. If the server process 922 detects that a predetermined number of expected confirmation signals from a mobile computing device 102 have not arrived, the server process 922 determines that the mobile computing device 102 has disconnected. If the server process 922 detects that a user has disconnected from a session 918, either intentionally, or unintentionally, the entry in the data store 926 related to the disconnected session 918 is modified to reflect the disconnection.
[0181] A graphical representation of the at least one disconnected session associated with the user may be displayed to the user. The server process 922 transmits, to the client 102, a graphical representation of the at least one disconnected session associated with the user. In one embodiment, the graphical representation displays a representation of output data generated by an executing, disconnected session. In another embodiment, the graphical representation displays a representation of output data generated by an application executing in a disconnected session. In still another embodiment, the server process 922 transmits, to a mobile computing device 102, output data generated by an executing, disconnected session. In still even another embodiment, the mobile computing device 102 displays, to the user, a graphical representation of the output data. In yet another embodiment, the server process 922 transmits, to the mobile computing device 102, a graphical representation comprising a user interface element. In some embodiments, an output transmitter 924 transmits, to the mobile computing device 102, the output data generated by an executing, disconnected session.
[0182] In one embodiment, the server process 922 transmits, to the mobile computing device 102, a representation of a state of the at least one disconnected session associated with the user. In another embodiment, the server process 922 transmits, to the mobile computing
device 102, a description of a state of the at least one disconnected session associated with the user. In still another embodiment, the server process 922 transmits, to the mobile computing device 102, a text-based description of a state of the at least one disconnected session associated with the user.
[0183] In some embodiments, the server process 922 transmits, to the mobile computing device 102 output data generated in the at least one disconnected session. In one of these embodiments, the mobile computing device 102 generates the representation, graphical or text-based, of the output data received from the at least one disconnected session. In another of these embodiments, the mobile computing device 102 displays the output data as described above. In still another of these embodiments, the mobile computing device 102 displays the representation of the output data in a window generated by an internet browser application. In yet another of these embodiments, the mobile computing device 102 replaces an existing representation of output data with an updated version of the representation. In some embodiments, the representation 950 (not shown) of the output data is graphical. In other embodiments, the representation of the output data is text-based.
[0184] In some embodiments, a graphical representation 950 may depict the output of multiple applications executing within a disconnected session 918. In one of these embodiments, the graphical representation 950 depicts the output of multiple applications executing from a desktop application, which is provided by a session 918. In other embodiments, a plurality of graphical representations 950 is displayed to the user of the mobile computing device 102. In one of these embodiments, a first graphical representation 950 in the plurality of graphical representations depicts a first type of session 918 and a second graphical representation 950' (not shown) in the plurality of graphical representations depicts a second type of session 918. For example, the first graphical representation 950 may depict the application-output data generated by a single application executing in a session 918, while the second graphical representation 950' may depict the output data generated by a plurality of applications executing within a hosted desktop session 918'. In still other embodiments, a single graphical representation 950 is depicted.
[0185] In some embodiments, the server process 922 identifies a change in the at least one disconnected session associated with the user and transmits a modified version of the at least one disconnected graphical representation of the at least one session displayed to the user, responsive to the identified change. In one of these embodiments, the server process
922 requests an identification of a change in the at least one disconnected session. In another of these embodiments, the server process 922 requests the identification of the change from a server executing the session 918. In still another of these embodiments, the server process 922 requests, after a time interval, an identification of a second change in the at least one session associated with the user. In still even another of these embodiments, the server process 922 polls a server executing the session 918 for changes in the at least one application session. In yet another of these embodiments, the server process 922 updates, after a time interval, the graphical representation of the at least one application session displayed to the user, responsive to an identification of a second change in the at least one application session associated with the user.
[0186] In other embodiments, the server process 922 queries the data store 926 to determine that a previously connected session has become disconnected from a mobile computing device 102. In one of these embodiments, the server process 922 instructs the output transmitter 924 to transmit, to the mobile computing device 102, the output data generated by the session 918 prior to the disconnection.
[0187] In some embodiments, the server process 922 receives a request for the identification of the change. In one of these embodiments, the server process 922 receives the request from the mobile computing device 102. In another of these embodiments, the server process 922 transmits the identification of the change to the mobile computing device 102. In other embodiments, the server process 922 transmits, to the mobile computing device 102, an updated graphical representation of the at least one session, responsive to an identification of a change in the at least one session associated. In still other embodiments, the server process 922 transmits, to the mobile computing device 102, output data generated in the session 918. In one of these embodiments, the mobile computing device 102 generates the graphical representation of the output data and displays the graphical representation to the user.
[0188] In some embodiments, the mobile computing device 102 requests, from the server 106, the identification of a change in the at least one previously disconnected session associated with the user. In other embodiments, the mobile computing device 102 requests, from a server 106, which executes the at least one previously disconnected session, the identification of a change in the at least one previously disconnected session associated with the user. In still other embodiments, the mobile computing device 102 requests, after a time
interval, an identification of a change in the at least one previously disconnected session associated with the user. In still even other embodiments, the mobile computing device 102 polls a server for an identification of a change in the at least one previously disconnected session associated with the user.
[0189] In some embodiments, the mobile computing device 102 receives an identification of a change in the at least one previously disconnected session. In other embodiments, the mobile computing device 102 receives output data generated by a session during a time interval. In still other embodiments, the mobile computing device 102 receives a modified version of the graphical representation. In yet other embodiments, the mobile computing device 102 displays a modified version of the graphical representation.
[0190] Referring now to FIGS. 9B and 9C, a flow diagram depicts one embodiment of the steps taken in a method for displaying, activating, and providing remote access to disconnected sessions. The steps in the lefthand column may be taken, for example, by a client, such as a mobile computing device. The steps in the righthand column may be taken, for example, by a server. The method includes the step of transmitting information associated with a user, (step 940) The method includes the step of receiving information associated with a user, (step 942) At least one disconnected session already associated with the user is identified in response to the information, (step 944) A graphical representation of at least one disconnected session associated with the user is transmitted to the user, (step 946) A graphical representation representing at least one disconnected session associated with the user is received, (step 948) The graphical representation is displayed to the user, (step 950). A request to connect to the at least one previously disconnected session is transmitted, (step 952) A request from a user to connect to the at least one disconnected session is received, (step 954) A connection is established between a client computer operated by the user and the at least one disconnected session in the request, (step 956) Output data generated by the at least one disconnected session associated with the user is received in response to the transmitted request, (step 958)
[0191] Referring to FIG. 9B, and in greater detail, information associated with a user is transmitted, (step 940) In some embodiments, the information is a log-on request. In some embodiments, the information is a log-on request for access to a resource. In other embodiments, the information is a general log-on request for a server. In various embodiments, information may include authentication information. The authentication
information may be any type of authentication information, include, for example, any username-password/PIN combinations, voice samples, one-time passcodes, biometric data, digital certificates, or smart card data.
[0192] In some embodiments, the information is accepted from a user by an input module, such as input module 208, and provided to a client process, such as client process 910. In some embodiments, the information is forwarded to server by a client process, a network module, or a client process and network module in communication with one another. In any embodiments described herein where information is transmitted, the information may be encapsulated in a predetermined protocol prior to transmission. In any embodiments described herein where information is transmitted, the information may be encrypted prior to transmission. In various embodiments, the information is transmitted over a data network or link.
[0193] Referring to FIG. 9B, and in greater detail, information associated with a user is received, (step 942) In various embodiments, the information is received by a server. In any embodiments described herein where information is received, the information may be decrypted. In some embodiments, the user is authenticated according to the received information. The information may be analyzed to authenticate the user according to any known method. For example, a hash function may be performed upon the information and the results compared with entries stored in a look-up table to determine if the user should be authenticated.
[0194] Referring to FIG. 9B, and in greater detail, at least one disconnected session already associated with the user is identified in response to the information, (step 944) In various embodiments, the information associated with a user includes information that identifies the user. In many embodiments, sessions may be stored in a database, such as a data store. Each session may include information about the session's status, e.g. active, disconnected, terminated, or any other known status. Each session may include information about a user corresponding to the session. The database may be searched for sessions that have a disconnected status and that correspond to the user. Information about the sessions, such as the sessions' server addresses and process numbers, may be stored in a memory or other buffer.
[0195] Referring to FIG. 9B, and in greater detail, a graphical representation of at least one disconnected session associated with the user is transmitted to the user, (step 946) In various embodiments, the graphical representation is generated by at least one disconnected session. In some embodiments, the graphical representation is generated by at least one application executing within the at least one disconnected session. The graphical representation may be a graphical-based or text-based representation of the at least one disconnected session. In some embodiments, the graphical representation includes a description of each of the disconnected sessions. In one embodiment, the graphical representation displays output data or a representation of output data generated by at least one disconnected session. In another embodiment, the graphical representation displays output data or a representation of output data generated by an application executing in at least one disconnected session. In various embodiments, the graphical representation includes a user interface element. In many embodiments, the graphical representation of at least one disconnected session associated with the user is transmitted using an output transmitter. In some embodiments, the output transmitter intercepts output data generated by at least one executing, disconnected session and transmits the output data.
[0196] Referring to FIG. 9B, and in greater detail, a graphical representation representing at least one disconnected session associated with the user is received, (step 948) In various embodiments, the graphical representation is received by a client process, a network module, or a client process and network module in communication with one another. In some embodiments, the graphical representation is manipulated into a format suitable for interpretation by a standard display adapter. In many embodiments, the graphical representation is stored in a frame buffer. Further, the graphical representation is displayed to the user, (step 950). In many embodiments, the graphical representation is displayed on a display module 924. In some embodiments, the graphical representation is displayed on a native display 201 of a mobile computing device 102. In other embodiments, the graphical representation is displayed on an external display device 202. In some embodiments, a client process transmits the graphical representation to a display module 924 for display on the device.
[0197] Referring to FIG. 9C, and in greater detail, a request to connect to the at least one previously disconnected session is transmitted, (step 952) In various embodiments, the request may include the server address, process number, or any other information about the
disconnected sessions the user wishes to connect to. The user may make a request by selecting the graphical representation, in part or in whole, of a disconnected session. For example, the user may select an icon corresponding to the disconnected session to request connection. In another example, the user may select a disconnected session from a list of enumerated disconnected sessions. In yet another example, the user may select a disconnected session from a menu of disconnected sessions.
[0198] Referring to FIG. 9C, and in greater detail, a request from a user to connect to at least one disconnected session is received (step 954) and a connection is established between a client computer operated by the user and at least one disconnected session in the request, (step 956) In many embodiments, a server process may establish the connection. The connection may be established by forming a connection using the server address, the process number, and the client address. The connection may be established if the requested connection is consistent with a policy. For example, the connection may be established if the policy permits automatic re-connection of disconnected sessions. In another example, the connection may be established if the policy requires a client to reside on a secure network before a connection may be made, and the client does reside on a secure network. In some embodiments, when the connection is established, applications that may have stalled on the previously disconnected session may be re-executed.
[0199] Referring to FIG. 9C, and in greater detail, output data generated by the at least one disconnected session associated with the user is received in response to the transmitted request, (step 958) In various embodiments, the output data may be received from storage, such as a buffer, in an output transmitter. In other embodiments, the output data may be received from an output transmitter that intercepts output data generated by applications executing in the sessions and transmits the output data to a client.
[0200] Referring now to FIG. 1OA, a block diagram depicts one embodiment of a system for generating a computing environment on a remote machine for display on a local machine. In brief overview, the system includes a first client 102a, a first agent 1002, a second client 102b, and a second agent 1010. The first agent 1002, executing on the first client 102a, generates a computing environment 1004, which includes a plurality of windows 1006a-1006n and provides access to i) a resource 1015 available to a user of the second client 102b and provided by the first client 102a, and ii) a resource 1020 provided by the second client 102b that is available to the user of the second client 102b. The first agent 1002
generates a proxy window 1008 for display in the computing environment 1004. The proxy window 1008 represents a local window 1014 on the second client 102b that displays output data generated by the resource 1020 provided by the second client 102b. The proxy window 1008 has a z-order entry in a z-order list associated with the plurality of windows 1006a-n in the computing environment 1004. The second agent 1010 on the second client 102b receives, from the first agent 1002, at least one entry in the z-order list associated with the computing environment 1004 and transmits, to the first agent 1002, an identification of a change to window attribute data associated with the window 1014 on the second client 102b for modification, by the first agent 1002, of the computing environment 1004.
[0201] In one embodiment, the first client 102a is a mobile computing device. In another embodiment, the second client 102b is a client computing device. In still another embodiment, one of the first client 102a and the second client 102b may be a server 106. In still even another embodiment a resource executes on an mobile computing device 102a. In yet another embodiment, the output data generated by an execution of a resource on the mobile computing device 102a is displayed on the second client which may be a client computing device. In some embodiments, the output data generated by an execution on a client 102b of a resource 204 is integrated into a display on the mobile computing device.
[0202] Referring now to FIG. 1OA, and in greater detail, the first agent 1002, executing on the first client 102a, generates a computing environment 1004, which includes a plurality of windows 1006a-1006n and provides access to i) a resource 1015 available to a user of the second client 102b and provided by the first client 102a, and ii) a resource 1020 provided by the second client 102b that is available to the user of the second client 102b. In one embodiment, the first client 102a is a server 106. In another embodiment, the second client 102b is a client device 102, connecting to the server 106 to access one or more resource available to a user of the second client 102b.
[0203] In one embodiment, a resource 1015, 1020 comprises a program, an application, a document, a file, a plurality of applications, a plurality of files, an executable program file, a desktop environment, a computing environment, or other resource made available to a user of the second client 102b. The resource 1020 may be delivered to the second client 102b via a plurality of access methods including, but not limited to, conventional installation directly on the second client 102b, delivery to the second client 102b via a method for application streaming, delivery to the second client 102b of output data
generated by an execution of the resource 1020 on a third machine 106' and communicated to the second client 102b via a presentation layer protocol, delivery to the second client 102b of output data generated by an execution of the resource 1020 via a virtual machine executing on a first client 102a, or execution from a removable storage device connected to the second client 102b, such as a USB device. In some embodiments, the second client 102b transmits output data generated by the execution of the resource 1020 to another client machine 102'.
[0204] In some embodiments, a user of a second client 102b connects to a first client
102a and views a display on the second client 102b of a local version 1012 of a computing environment 1004, comprising a plurality of windows 1006a-n, generated on the first client 102a. In one of these embodiments, at least one resource is provided to the user by the first client 102a and displayed in the computing environment 1004. However, there may be resources that the user executes on the second client 102b, either by choice, or due to a policy or technological requirement. In another of these embodiments, a user may invoke a local application from the remote desktop interface generated by the first client 102a and have the local application appear in the same desktop environment as the remote applications provided by the first client 102a. In still another of these embodiments, the user of the second client 102b would prefer an integrated desktop environment providing access to all of the resources available to the user, instead of separate desktop environments for resources provided by separate machines. For example, a user may find navigating between multiple graphical displays confusing and difficult to use productively. Or, a user may wish to use the data generated by one application provided by one machine in conjunction with another resource provided by a different machine. In another of these embodiments, requests for execution of a resource, windowing moves, application minimize/maximize, and termination of executing resources may be controlled by interacting with a computing environment that integrates the display of the remote resources and of the local resources. In yet another of these embodiments, an application or other resource accessible via an integrated desktop environment 1004 - both those generated on the second client 102b and those generated on the first client 102a - is shown on the computing environment 1004 as if it were executing on, or executable from, the computing environment. For example, a resource may also appear in a listing of available resources provided in a Start Menu, a shortcut may be provided on the desktop or the Quick Launch menu, and the resources can be launched, selected and interacted with in the same way as an application provided by the first client 102a.
[0205] In some embodiments, a single computing environment 1004 is displayed. In one of these embodiments, the computing environment 1004 is displayed as a full-screen desktop. In other embodiments, a plurality of computing environments 1004 is displayed. In one of these embodiments, one or more of the computing environments are displayed in non- full-screen mode on one or more display devices 124. In another of these embodiments, the computing environments are displayed in full-screen mode on individual display devices. In still another of these embodiments, one or more of the computing environments are displayed in full-screen mode on one or more display devices 124. In yet another of the embodiments, a resource provided by the second client 102b may be integrated with one of the plurality of computing environments 1004.
[0206] In some embodiments, the first agent 1002 generates a computing environment 1004 including a plurality of windows 1006a-1006n. In one of these embodiments, a window 1006a in the plurality of windows 1006 displays the output data generated by an execution of a resource provided by the first client 102a. In another of these embodiments, a window 1006b in the plurality of windows 1006a-1006n displays the output data generated by an execution of a resource provided by a third machine 106', as discussed in further detail below, in connection with FIGs. 12A and 12B. In still another of these embodiments, a window 1006c in the plurality of windows 1006a-1006n depicts a taskbar from a desktop environment. In still even another of these embodiments, a window 1006d represents a menu, such as a Start menu or a context-specific menu associated with an application. In yet another of these embodiments, a window 1006e in the plurality of windows 1006a-n has a z-order entry such that it is displayed beneath the other windows in the plurality of windows 1006a-n and depicts a desktop. In other embodiments, the first agent 1002 transmits, to the second agent 1010, window attribute data associated with each of the plurality of windows 1006a-n. In one of these embodiments, the second agent 1010 displays a local version of the plurality of windows 1006a-n having window attribute data corresponding to the window attribute data received from the first agent 1002.
[0207] In one of these embodiments, the graphical display and the window attribute data for the plurality of windows 1006a-n and for the computing environment 1004 are transmitted to the second agent 1010 for generation of a local version of the computing environment 1004. In another of these embodiments, the second agent 1010 displays, to a
user of the second client 102b, a local version 1012 of the plurality of windows 1006 and the computing environment 1004.
[0208] Referring ahead to FIG. 1OB, a block diagram depicts one embodiment of a system in which the first agent 1002 integrates resources from various sources into a computing environment 1004. In some embodiments, and as shown in FIG. 1OB, the first agent 1002 may receive the identification of the resources available to a user of the second client 102b from a plurality of servers 106. In one of these embodiments, the first agent 1002 receives an identification of the available resources from machines 106', 106", 106'", and 106"".
[0209] In one of these embodiments, the first client 102a receives an identification of the available resources from server 106', receives the output data generated by an execution of the resource on server 106' and transmits the output data so generated to second client 102b. In another of these embodiments, the first agent 1002 receives only the identification of the available resources from the machine 106", and the machine 106" transmits the output data generated by an execution of the resource to the second client 102b.
[0210] In one of these embodiments, the first agent 1002 receives, from a machine
106'", an identification of resources available to the second client 102b. In another of these embodiments, the first agent 1002 receives, from the second agent 1010, an identification of available resources from the machines 106'" and 106". In still another of these embodiments, the second client 102b receives, via a method for resource streaming, the available resource from the server 106'". In this embodiment, the second client 102b executes the resource locally and the graphical output data is integrated with the computing environment 1004. In still even another of these embodiments, the first client 102a receives an identification of the available resources from server 106"", and receives via a method for resource streaming, the available resource from machine 106"" and transmits the output data generated by an execution of the resource to the second client 102b. In other embodiments, the first agent 1002 receives, from the second agent 1010, an identification of a resource 1020 available to the user of the second client 102b for integration into the computing environment 1004.
[0211] In one embodiment, executing a resource provided by the second client 102b may allow a user to leverage a characteristic of the second client 102b that is not provided by the first client 102a; for example, the second client 102b may have more appropriate
processing ability, graphics functionality, bandwidth, or license rights for accessing the resource than the first client 102a. In another embodiment, the first client 102a may lack the ability to execute the resource; for example, because only the user of the second client 102b owns an application or owns a resource, such as an operating system, required to execute the application. In still another embodiment, the resource 1020 is a resource such as an application or desktop environment installed on the second client 102b via a method for streaming the application to the second client 102b from a third machine 106'. In still even another embodiment, the resource 1020 is a resource such as an application or desktop environment whose output data the second client 102b receives via a presentation layer protocol communication with a third machine 106'. In yet another embodiment, the first agent 1002 receives an identification of a conferencing or Internet-communication application; a user peripheral, such as a media player, a digital camera or a web camera; or a processor-intense, data-intense, or graphics-intense application, such as a media-editing application or a computer-aided design application.
[0212] Referring back to FIG. 1OA, the first agent 1002 incorporates the identification of the resource 1020 into the computing environment 1004. In one of these embodiments, the first agent 1002 adds a program execution shortcut to a menu for requesting execution of resources, the program execution shortcut corresponding to the identified resource 1020. In another of these embodiments, the first agent 1002 generates a graphical representation, such as an icon, associated with the resource 1020 for display in the computing environment 1004. The first agent 1002 may alternatively receive the graphical representation from the second agent 1010 for display in the computing environment 1004. In still another of these embodiments, a user requests execution of a resource 1020 by interacting with a program execution shortcut or a graphical representation displayed in the local version of the computing environment 1004.
[0213] In one embodiment, the first agent 1002 receives the request for execution of a resource 1020, responsive to the user interaction with a program execution shortcut or a graphical representation, and sends the request for execution to the second agent 1010. In another embodiment, a user requests execution of a resource by interacting with the local display 1012 of the computing environment 1004; for example, by selecting a graphical representation of a file, document, uniform resource locator, or other resource, displayed by the desktop environment.
[0214] The first agent 1002 generates a proxy window 1008 for integration into the computing environment 1004. In one embodiment, the first agent 1002 is referred to as a proxy window management component 1002. In another embodiment, the proxy window management component 1002 is responsible for making the proxy window 1008 conform to client-initiated changes to window attribute data. In still another embodiment, the proxy window management component 1002 is responsible for monitoring any desktop-initiated changes to window attribute data and communicating them to the second agent 1010, which applies them to the corresponding local application window 1014.
[0215] In yet another embodiment (not shown in FIG. 10A), the proxy window management component 1002 provides additional functionality. In one of these embodiments, such functionality may include a MICROSOFT Active Accessibility (MSAA) hook and other monitoring mechanisms to detect new application windows and window attribute data changes initiated on the desktop. In another of these embodiments, the proxy window management component 1002 may further include support for launching published applications. In still another of these embodiments, the proxy window management component 1002 may send updates initiated from the remote machine 106 [desktop] regarding changes to window attribute data to the second agent 1010 where the updates will be applied to the local window 1014, and to the local displays of the plurality of windows 1006a-n. In still even another of these embodiments, the proxy management component 1002 may be enabled to apply client-initiated updates to window attribute data associated with proxy window 1008 and to the plurality of data objects 1006a-n. In yet another of these embodiments, the proxy management component 1002 may remove proxy windows when the client connection disappears.
[0216] In some embodiments, the first agent 1002 and the second agent 1010 include a hooking component for intercepting window-related messages. For example, and in some embodiments, an agent executing on a machine 102 or 106 on which the MICROSOFT WINDOWS operating system executes may use a version of the MICROSOFT Active Accessibility hook to monitor relevant window events. For example, and in other embodiments, an agent may include a MICROSOFT Computer-based Training (CBT) window hook, or other hooking mechanisms, to intercept and monitor window events.
[0217] The proxy window 1008 represents a window 1014 on the second client 102b that displays output data generated by the resource 1020 provided by the second machine.
The proxy window 1008 has a z-order entry in a z-order list associated with the plurality of windows 1006a-n in the computing environment 1004. In one embodiment, the proxy window 1008 has a traditional window visual state - position, size, Z-order, focus state, minimized/normal/maximized state - and a taskbar button, and appears in the remote desktop AIt-TAB selection dialog. In another embodiment, the proxy window 1008 responds to requests, initiated by the remote desktop, to change visual state or to invoke its taskbar menu. In still another embodiment, the visual state of the local application window 1014 is synchronized with the visual state of the corresponding proxy window 1008. In still even another embodiment, the proxy window 1008 can pass visual state change requests to the second agent 1010, via the first agent 1002, without needing to apply them to itself.
[0218] In one embodiment, the first agent 1002 does not transmit the proxy window
1008 to the second agent 1010, only window attribute data associated with the proxy window 1008. In another embodiment, the proxy window 1008 does not need to be painted on the computing environment 1004, or can be painted very efficiently e.g. in a uniform color. In still another embodiment, the first agent 1002 is aware of the clipping regions associated with the proxy window 1008 and does not send window attribute data for those regions.
[0219] The second agent 1010 on the second client 102b receives, from the first agent
1002, at least one entry in the z-order list associated with the computing environment 1004. The second agent 1010 transmits, to the first agent 1002, an identification of a change to window attribute data associated with the window 1014 on the second client 102b for modification, by the first agent 1002, of the computing environment 1004. In one embodiment, the second agent 1010 is an integrated window management component 1010 executing on the second client 102b that enumerates, monitors, and manages local windows 1014 that are integrated into the remote desktop 1004. In another embodiment, the integrated window management component 1010 manages all local windows 1014 that are integrated into the remote desktop 1004. In still another embodiment, the integrated window management component 1010 communicates window existence and client-initiated visual state changes over a virtual channel to the first agent on the first client 102a.
[0220] In yet another embodiment, the integrated window management component
1010 provides additional functionality. In one of these embodiments, such functionality may include a MICROSOFT Active Accessibility (MSAA) hook and other monitoring mechanisms to detect new application windows and window attribute data changes initiated
on the client. In still another of these embodiments, the proxy window management component 1002 may send updates initiated from the client 102a regarding changes to window attribute data to the first agent 1002 where the updates will be applied to the proxy window 1008, and to the local displays of the plurality of windows 1006a-n. In still even another of these embodiments, the proxy management component 1002 may be enabled to apply desktop-initiated updates to window attribute data associated with local window 1014 and to the local displays of the plurality of windows 1006a-n.
[0221] In some embodiments, a first client 102a and a second client 102b communicate using a presentation layer protocol, for example, by communicating via the ICA protocol, or the RDP protocol. In one of these embodiments, the first agent 1002 and the second agent 1010 exchange graphical data, i.e., the data actually displayed in each window on the desktop environment, via a first virtual channel. In another of these embodiments, the first virtual channel is an ICA virtual channel. In still another of these embodiments, information about window positioning, window size, z-ordering of window and other such information is communicated between the first client 102a and the second client 102b via a second virtual channel. In yet another of these embodiments, the second virtual channel is an ICA virtual channel. In other embodiments, the first agent 1002 and the second agent 1010 exchange window attribute data. In one of these embodiments, the first agent 1002 and the second agent 1010 also exchange graphical data. In another of these embodiments, a third agent executing on the first client 102a and the second agent 1010 exchange graphical data. In still another of these embodiments, a third agent executing on the first client 102a and a fourth agent executing on the second client 102b exchange graphical data.
[0222] Referring now to FIG. 1 IA, a flow diagram depicts one embodiment of the steps taken in a method for generating a computing environment on a remote machine for display on a local machine. In a networked computing system including a remote machine and a local machine used by a user, a method for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access both to resources provided by the local machine and to resources provided by the remote machine, includes the step of monitoring, by a first agent executing on a first machine, a computing environment for display on a second machine, the computing environment comprising a plurality of windows and providing access to i) a resource provided by the first machine that is available to a user of the second machine and ii) a
resource provided by the second machine that is available to the user of the second machine (step 1102). The first agent generates a proxy window representing a window on the second machine that displays output data generated by the resources provided by the second machine, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1104). The first agent transmits, to a second agent on the second machine, at least one entry in the z-order list associated with a window in the plurality of windows in the computing environment (step 1106). The first agent receives an identification of a change to window attribute data associated with the window on the second machine (step 1108). The first agent modifies the computing environment, responsive to the identification of the change (step 1110).
[0223] A first agent executing on a first machine monitors a computing environment for display on a second machine, the computing environment comprising a plurality of windows and providing access to i) a resource provided by the first machine that is available to a user of the second machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1102). In one embodiment, the first agent 1002 monitors the generation and modification of the computing environment 1004 and of a plurality of windows 1006a-n providing access to a plurality of resources 1015 executing on remote servers, such as the first client 102a, and to at least one resource 1020 provided by the second client 102b.
[0224] The first agent generates a proxy window representing a window on the second machine that displays output data generated by the resources provided by the second machine, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1104). In one embodiment, the local window 1014 is reflected into the remote desktop window list using the proxy window 1008, which has dimensions, position, Z-order value and focus state in the computing environment 1004. These window attributes are kept in synch between the second client 102b and the first client 102a.
[0225] In some embodiments, the second agent 1010 receives an identification of a region in the computing environment corresponding to a region on the local display 1012 associated with a local resource 1020. In one of these embodiments, the second agent 1010 receives the identification from the first agent 1002. In another of these embodiments, described in additional detail below, the second agent 1010 identifies the region responsive to
information received from the first agent 1002. In still another of these embodiments, as described in additional detail below, the second agent 1010 identifies the region responsive to identifying a color key in a communication from the first agent 1002.
[0226] In some embodiments, the local display 1012 is a single window that displays the graphical output data of the computing environment 1004, including the window output for the plurality of windows 1006a-n. In one of these embodiments, an individual window 1006a does not need to have a corresponding seamless window on the client. In another of these embodiments, the local display 1012 is maintained as the topmost window, thus keeping local applications underneath the local display 1012 in the client Z-order. In still another of these embodiments, a region of local display 1012 is shown as transparent to allow the correct parts of a local window 1014 to show through the local display 1012, as if the local window 1014 were on the computing environment 1004. In still even another of these embodiments, the proxy window 1008 is displayed on a region of the computing environment 1004 corresponding to the region of local display 1012 which is to be transparently displayed. In yet another of these embodiments, the transparent region is referred to as a clipping region.
[0227] In some embodiments, to identify the areas to be made transparent, the first agent 1002 uses the proxy window 1008 to identify a clipping region. In one of these embodiments, the first agent 1002 intercepts a paint request on the computing environment 1004. In another of these embodiments, the first agent 1002 transmits, to the second agent 1010, an identification of a clipping region.
[0228] In other embodiments, the second agent 1010 identifies the areas of local display 1012 to be displayed by the local resource 1020. In one of these embodiments, the second agent 1010 identifies the areas responsive to information received from the first agent 1002. In another of these embodiments, the second agent 1010 identifies a key received from the first agent 1002, such as a color key or an identifiable pattern or tag identifying a clipping region.
[0229] In still other embodiments, the second agent 1010 ensures that the resource
1020 paints output data to the appropriate local window 1014, which is located in a region corresponding to the clipping region on the computing environment. In one of these embodiments, the second agent 1010, in communication with a window management component, ensures that the local display 1012 is kept topmost and displays the graphical
output data associated with the computing environment 1004, other than the output data that would be displayed in a clipping region. In another of these embodiments, the second agent 1010 instructs a window management component to regard an instruction to paint a particular color or pattern to a region (for example, to a region of the local display 1012) as an instruction to keep the region transparent. This color key may include an alpha value (e.g., a 32-bit {R,G,B,alpha} rather than a 104-bit {R,G,B} value) which can be used to distinguish it from all non-transparent colors (for resources that don't use partial transparency).
[0230] In yet other embodiments, the agent 1002 paints a clipping region for a proxy window 1008 by responding to a paint request from a window management component to the proxy window by painting the window in a color key. In one of these embodiments, the agent 1010 associates a paint request in a color key with the appropriate local application window. In another of these embodiments, the agent 1010 also needs to use paint requests in a different color for a (part of) an existing clipping region to remove the painted region from the associated clipping region.
[0231] In some embodiments, the first agent 1002 may paint a clipping region in the color key rather than send a z-order entry to the second agent 1010. In other embodiments, the first agent 1002 sends at least one z-order entry for a proxy window 1008 associated with a local window 1014. In one of these embodiments, the second agent 1010 ensures that a local window 1014 has the same relative Z-order relationship on the client as the corresponding proxy window 1008 does on the first client 102a, i.e. if proxy window 1008 is below proxy window 1008' on the first client 102a (regardless of how many server windows there are, and whether they are above or below either A or B), then the second agent 1010 ensures that a local window 1014 is below a local window 1014'.
[0232] In other embodiments, the windows underneath the local display 1012 show through wherever the local display 1012 is painted in the color key. In one of these embodiments, the second agent 1010 maintains the local windows in the appropriate relative Z-order so that they will paint correctly and a window that's above another will correctly occlude the other even though both occupy the entire clipping region. In another of these embodiments, the second agent 1010 also ensures that user input will be directed to the correct window - for example a mouse click on a transparent region will be sent to the underlying window, not the local display 1012.
[0233] In some embodiments, the first agent 1002 transmits to the second agent 1010 an identification of a clipping region for each proxy window 1008 in a computing environment 1004. In one of these embodiments, the second agent 1010 directs the local resource 1020 to paint output data to the region of local display 1012 corresponding to the clipping region. In another of these embodiments, directing the local resource 1020 to paint output data to a region of the local display 1012 avoids the need for transparency. In still another of these embodiments, the first agent 1002 identifies a clipping region on the computing environment 1004 that corresponds to a region displaying local window 1014. In still even another of these embodiments, the first agent 1002 uses a color key or identifiable pattern or tag as described above to tag a clipping region. In one of these embodiments, the agent 1002 or the agent 1010 associates a paint request in a color key with the appropriate local application window. In another of these embodiments, the agent 1002 or the agent 1010 responds to paint requests in a different color for a (part of) an existing clipping region for an application window by removing the newly painted region from the associated clipping region. In still another of these embodiments, the agent 1002 or agent 1010 associates a different color key for each local window. In yet another of these embodiments, the second agent 1010 identifies a clipping region responsive to information received from the first agent 1002.
[0234] In some embodiments, the first agent 1002 may paint a clipping region in a color key rather than send a z-order entry to the second agent 1010. In other embodiments, the first agent 1002 may send an identification of a clipping region to the second agent 1010 rather than send a z-order entry to the second agent 1010. In still other embodiments, the first agent 1002 may send information other than a z-order entry to agent 1010 that allows agent 1010 to identify a clipping region rather than send a z-order entry to the second agent 1010. In yet other embodiments, the first agent 1002 does not send a z-order entry to the second agent 1010 at all.
[0235] In other embodiments, the second agent 1010 ensures that the local display
1012 is maintained topmost in the local desktop environment. In one of these embodiments, the local display 1012 is used to show all of the remote desktop graphics output including areas belonging to local window 1014. In another of these embodiments, local windows never show through. In still another of these embodiments, when the second agent 1010 receives output data not containing the color key for a portion of a clipping region associated
with a local window, the second agent 1010 removes the output region from the clipping region for that window.
[0236] In still other embodiments, the second agent 1010 directs the local resource
1020 to paint the output data it generates to a region in the local display 1012 corresponding to the region identified by the first agent 1002 as a clipping region. In one of these embodiments, the second agent 1010, in communication with a window management component, sends the local resource 1020 a WM PAINT message for a display context that references a clipping region on the local display 1012. In another of these embodiments, the second agent 1010 sends the messages for any updated portions of a local window's associated clipping region whenever portions of the clipping region change. In still another of these embodiments, the second agent 1010 sends the paint messages in a periodic refresh loop as local windows are always covered by the local display 1012 and can no longer asynchronously update their regions that are visible on the client display. In still even another of these embodiments, the second agent 1010 ensures that user input in clipping regions (including mouse clicks and, where appropriate, keyboard events) are redirected by the local display 1012 to the corresponding local application window. This means detecting mouse events in clipping regions, determining which local window is associated with the mouse event coordinates and sending the event to that window at those coordinates.
[0237] As described above, in some embodiments, a clipping region is identified. In one of these embodiments, an agent 1002 or 1010 calculates a clipping region responsive to information associated with the computing environment, including a list of windows, their Z- order, size & position (and, for non-rectangular windows, their shape). In another of these embodiments, the agent 1002 or 1010 acquires the information from one or more sources including, but not limited to, the seamless virtual channel, data used by the Headless Client, subclassing windows, Microsoft Active Accessibility (e.g. using WinEvents and MSAA hooking) and by hooking various Windows APIs. In still another embodiment, the agent 1002 or 1010 identifies a clipping region, or portion of a clipping region, by hooking or responding to window messages (including WM_PAINT, WM_NCPAINT, WM ERASEBKGND) which allow the agent 1002 or 1010 to see requests by Windows for a window to perform a paint operation within a region. Each time the agent 1002 or 1010 identifies one of these messages, the corresponding region is added to the clipping region for the associated window. The agent 1002 or 1010 uses knowledge of which windows receive
which paint requests to maintain a record of which regions of the screen "belong" to each application window. Each time the agent 1002 or 1010 intercepts a paint request for a different window covering (all or part of) an existing clipping region for another window, that part is removed from that window's clipping region. The agent 1002 or 1010 can also directly retrieve the clipping region using Windows API functions such as GetRandomRgn() which allows the agent 1002 or 1010 to directly interrogate a window for clipping regions. In some embodiments, the agents 1002 and 1010 communicate as described above to exchange information required by the second agent 1010 to modify the local display 1012 or local windows 1014 or local windows corresponding to the plurality of windows 1006a-n, as described below in connection with FIG. 1 IB.
[0238] The first agent transmits, to a second agent on the second machine, at least one entry in the z-order list associated with a window in the plurality of windows in the computing environment (step 1106). In some embodiments, the first agent 1002 transmits to the second agent 1010 the entire z-order list. In other embodiments, the first agent 1002 transmits to the second agent 1010 a partial z-order list. In one embodiment, the first agent 1002 transmits the at least one entry in the z-order list to the second agent 1010 via a virtual channel coupled to the computing environment 1004. In another embodiment, the first agent 1002 transmits, to the second agent 1010, via a virtual channel conveying window attribute data associated with the proxy window 1008, messages directing the modification of a corresponding local window 1014, which displays, in accordance with the window attribute data conveyed by the virtual channel, graphical output data generated by a resource 1020 executing on the second client 102b.
[0239] In one embodiment, the second client 102b displays a local version of each of the plurality of windows 1006a-n. In another embodiment, the second agent 1010 directs a window management component on the second client 102b to assign each of the locally- displayed windows 1006a-n a z-order entry in a z-order list of windows displayed on the second client 102b. Each of the assigned z-order entries for each of the locally-displayed windows 1006a-n result in the display of the locally-displays windows 1006a-n above a local window associated with a local application on the second client 102b other than the local display of a local application displayed in local window 1014. In this embodiment, the local window is kept below the local display 1012 of the computing environment 1004 in the client- side Z-order - that is, it is kept below each of the windows in the plurality of windows
1006a-n and below the local window 1014, which is integrated with the plurality of windows 1006a-n. In still another embodiment, the plurality of windows 1006a-n and the local window 1014 are displayed on the second client 102b according to the z-order list maintained on the first client 102a.
[0240] The first agent receives an identification of a change to window attribute data associated with the window on the second machine (step 1108). In one embodiment, the first agent 1002 receives the identification of the change via a virtual channel. In another embodiment, the first agent 1002 receives a message that a user minimized the local window 1014. In still another embodiment, the first agent 1002 receives a message that a user restored a minimized local window 1014. In still even another embodiment, the first agent 1002 receives a message that a user changed a size or position of the local window 1014. In yet another embodiment, the first agent 1002 receives a message that a user maximized a local window 1014.
[0241] The first agent modifies the computing environment, responsive to the identification of the change (step 1110). In some embodiments, the first agent 1002 directs a window management component of an operating system executing on the first client 102a to modify a window displayed in the computing environment. In other embodiments, the first agent 1002 sends a message to the second agent 1010 to restore the local window 1014. In one of these embodiments, the second agent 1010 sends a message to the resource 1020 to restore the minimized local window 1014. In another of these embodiments, the first agent 1002 sends a message to a window management component of an operating system executing on the first client 102a to restore the proxy window 1008. In still other embodiments, the first agent 1002 sends a message to the second agent 1010 to terminate the execution of a local resource 1020 when the first agent 1002 terminates the generation and display of the computing environment 1004. In some embodiments, the first agent 1002 sends a message to the second agent 1010 to terminate an operating system executing on the second client 102b. In one of these embodiments, the first agent 1002 sends a message to the second agent 1010 to shut down the second client 102b.
[0242] In one embodiment, when a window 1006a associated with a remote resource
1015 is selected by a user and the local window 1014 loses focus, the second agent 1010 transmits a message to the first agent 1002 instructing the first agent 1002 to direct a window management component to remove focus from the proxy window 1008 and to focus on a
window 1006a in the plurality of windows 1006a-n. In another embodiment, when a window 1006a associated with a remote resource 1015 is selected by a user and the local window 1014 loses focus, the second agent 1010 allows the resource 1020 to paint inside its window, resulting in a completely transparent view of local and remote resources.
[0243] Referring now to FIG. 1 IB, a flow diagram depicts one embodiment of the steps taken in a method for displaying, on the local machine, a desktop environment that is generated by the remote machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by the remote machine. In brief overview, the method includes the step of displaying, by an agent executing on a local machine, a computing environment generated by a remote machine and providing access to i) a remote resource available to a user of the local machine, and to ii) a local resource available to the user of the local machine (step 1120). The agent receives at least one entry in a z-order list including a z-order entry of at least one window in the computing environment and including the z-order entry of a proxy window corresponding to a local window on the local machine displaying output data generated by the local resource (step 1122). The agent modifies window attribute data for the displayed computing environment, responsive to the received z-order entry (step 1124). The agent executing on the local machine transmits, to an agent on the remote machine, an identification of a change to a z-order entry of a local window on the local machine displaying output data generated by the local resource (step 1126). The agent on the remote machine modifies a z-order entry for a proxy window representing the local window in a z-order list on the remote machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1128).
[0244] An agent executing on a local machine displays a computing environment generated by a remote machine and providing access to i) a remote resource available to a user of the local machine, and to ii) a local resource available to the user of the local machine (step 1120). In one embodiment, the agent 1010 receives graphical data and window attribute data from the agent 1002 and displays the graphical data in a window formed according to the window attribute data. In another embodiment, the agent 1010 receives graphical data and corresponding window attribute data for a plurality of windows forming a computing environment and displays the graphical data in a plurality of windows formed according to
the window attribute data. In other embodiments, the first agent 1002 and the second agent 1010 exchange window attribute data. In one of these embodiments, the first agent 1002 and the second agent 1010 also exchange graphical data. In another of these embodiments, a third agent executing on the first client 102a and the second agent 1010 exchange graphical data. In still another of these embodiments, a third agent executing on the first client 102a and a fourth agent executing on the second client 102b exchange graphical data. In yet another of these embodiments, the first agent 1002 on the first client 102a and a fourth agent executing on the second client 102b exchange graphical data.
[0245] The agent receives at least one entry in a z-order list including a z-order entry of at least one window in the computing environment and including the z-order entry of a proxy window corresponding to a local window on the local machine displaying output data generated by the local resource (step 1122). In one embodiment, the agent 1010 displays a local window 1014 displaying output data, generated by an execution of the resource 1020, and a plurality of windows 1006a-n formed in accordance with received window attribute data, in accordance with the at least one entry in the z-order. In another embodiment, the agent 1010 monitors the local display of the plurality of windows 1006a-n and of the local window 1014. In another embodiment, the agent 1010, in communication with a window management component of an operating system executing on the second client 102b, manages the local display of the plurality of windows 1006a-n and of the local window.
[0246] The agent modifies window attribute data for the displayed computing environment, responsive to the received z-order entry (step 1124). In some embodiments, the agent modifies a z-order entry for a local version of a window 1006a in the plurality of windows 1006a-n. In other embodiments, the agent modifies window attribute data for a local version of a window 1006a in the plurality of windows 1006a-n. In one of these embodiments, the agent resizes the local version of the window 1006a in the plurality of windows 1006a-n. In one of these embodiments, the agent repositions the local version of the window 1006a in the plurality of windows 1006a-n. For example, the agent may modify window attribute data of the local window 1014 responsive to the received z-order entry sent as a result of a window management component implementing a window cascade on the server.
[0247] In some embodiments, as described above in connection with FIG. 1 IA (step
1106), rather than receiving a z-order entry, the second agent 1010 receives an identification
of a clipping region in the computing environment corresponding to a region on the local display 1012 associated with a local resource 1020. In one of these embodiments, the second agent 1010 receives the identification from the first agent 1002. In another of these embodiments, described in additional detail above, the second agent 1010 identifies the region responsive to information received from the first agent 1002. In still another of these embodiments, as described in additional detail above, the second agent 1010 identifies the region responsive to identifying a color key in a communication from the first agent 1002.
[0248] The agent executing on the local machine transmits, to an agent on the remote machine, an identification of a change to a z-order entry of a local window on the local machine displaying output data generated by the local resource (step 1126). In one embodiment, when a user of the second client 102b makes a change to a local window 1014, for example, by minimizing, maximizing, or resizing the window, the agent 1010 transmits a message to the agent 1002 to make a corresponding change to the proxy window 1008 and to its z-order entry in the z-order list ordering the plurality of windows 1006a-n and the proxy window 1008.
[0249] The agent on the remote machine modifies a z-order entry for a proxy window representing the local window in a z-order list on the remote machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1128). In one embodiment, the agent 1010 assigns the local window 1014 a z-order entry synchronized with the z-order entry of the proxy window 1008, which has a z-order entry in a z-order list including z-order entries for a plurality of windows 1006a-n. In another embodiment, generating a z-order list including entries both for windows generated by a resource executing on a server 106 and for windows generated by a resource executing on a client 102 allows for integrated and overlapping displays of local and remote windows.
[0250] Referring now to FIG. 12A, a block diagram depicts one embodiment of a system for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by a second remote machine. In brief overview, the system includes a first client 102a, a first agent 1002, a second client 102b, a second agent 1010, and a remote machine 106'. The first agent 1002 receives an identification of a remote resource 1210 available to a user of the second client 102b and
provided by the remote machine 106'. The first agent 1002, executing on the first client 102a, generates a computing environment 1004, which includes a plurality of windows 1006a-206n and provides access to i) a resource 1210 available to a user of the second client 102b and provided by the remote machine 106', and ii) a resource 1020 provided by the second client 102b that is available to the user of the second client 102b. The first agent 1002 receives a request for execution of the remote resource 1210 provided by the remote machine 106'. The first agent 1002 directs the second agent 1010 on the second client 102b to request execution of the remote resource 1210 provided by the remote machine 106'. The second agent 1010 transmits to the first agent 1002 an identification of a change to a z-order entry of a local window 1014 on the second client 102b, the local window 1014 displaying output data generated by the remote resource 1210 provided by the remote machine 106'. The first agent 1002 modifies a z-order entry for a proxy window 1008 representing the local window 1014 in a z-order list on the first client 102a, responsive to the identification of the change, the proxy window 1008 having a z-order entry in a z-order list associated with the plurality of windows 1006 in the computing environment 1004.
[0251] The second agent 1010 provides access to the computing environment 1004.
In some embodiments, establishment of a seamless pass-through presentation layer protocol connection between the computing environment 1004 to the remote machine 106' is requested. In one of these embodiments, a second client agent 1010' is executed on the second client 102b to instead establish the connection between the second client 102b and the remote machine 106'. In another of these embodiments, the second client agent 1010' looks like a client-side application that is accordingly integrated into the remote desktop by the second agent 1010 and the first agent 1002 using the methods described above in connection with FIGs. 1OA, 1OB, 1 IA, and 1 IB. In one embodiment, the second client agent 1010' is an RDP client. In another embodiment, the second client agent 1010' is an ICA client.
[0252] Referring now to FIG. 12B, a screen shot depicts one embodiment of a system for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by a second remote machine. As depicted in FIG. 12B, two presentation layer protocol sessions are executed on the second client 102b. Session 1220, with the bold boundary, is an ICA session providing access to a desktop environment 1004. Session 1230, with the dashed boundary, is an ICA session providing
access to a resource (in FIG. 12B, the resource is a word processing program, MICROSOFT WORD). The WORD application window, which is local window 1014, has been merged with the presentation of the desktop session, which is represented by the computing environment 1004. If the first client 102a had established the session, the first client 102a would have received the output data and would have then forwarded the output data to the second client 102b for display. Instead, in this embodiment, the second client 102b established the presentation layer protocol session required to access WORD, instead of the first client 102a establishing a presentation layer protocol session, and the second client 102b receives the output data directly from the remote machine 106.
[0253] Referring now to FIG. 13, a flow diagram depicts one embodiment of the steps taken in a method for generating a computing environment on the remote machine for display on the local machine, the computing environment providing integrated access to both resources provided by the local machine and to resources provided by a second remote machine, includes the step of receiving, by a first agent executing on a first machine, an identification of a remote resource available to a user of a second machine, the remote resource provided by a third machine (step 1302). The first agent monitors a computing environment on the first machine, the computing environment comprising a plurality of windows and providing access to i) the remote resource provided by the third machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1304). The first agent receives a request for execution of the remote resource provided by the third machine (step 1306). The first agent directs a second agent on the second machine to request execution of the remote resource provided by the third machine (step 1308). The second agent transmits, to the first agent, an identification of a change to a z- order entry of a local window on the second machine displaying output data generated by the remote resource provided by the third machine (step 1310). The first agent modifies a z- order entry for a proxy window representing the local window in a z-order list on the first machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1312).
[0254] Referring now to FIG. 13, and in greater detail, a first agent executing on a first machine receives an identification of a remote resource available to a user of a second machine, the remote resource provided by a third machine (step 1302). In one embodiment,
the first agent 1002 receives the identification from the remote machine 106'. In another embodiment, the first agent 1002 receives the identification from the second client 102b. In still another embodiment, the first agent 1002 receives the identification from an agent on one of the remote machine 106' and the second client 102b. In yet another embodiment, the first agent 1002 receives the identification from a fourth machine 106'". In one embodiment, a resource 1210 comprises a program, an application, a document, a file, a plurality of applications, a plurality of files, an executable program file, a desktop environment, a computing environment, or other resource made available to a user of the second client 102b. The resource 1210 may be delivered to the remote machine 106 via a plurality of access methods including, but not limited to, conventional installation directly on the remote machine 106, delivery to the remote machine 106 via a method for application streaming, or execution from a removable storage device connected to the remote machine 106, such as a USB device.
[0255] The first agent monitors a computing environment on the first machine, the computing environment comprising a plurality of windows and providing access to i) the remote resource provided by the third machine and ii) a resource provided by the second machine that is available to the user of the second machine (step 1304). In some embodiments, the first agent monitors the computing environment 1004 as described above in connection with FIGs. 1OA, 1OB, HA, and HB.
[0256] The first agent receives a request for execution of the remote resource provided by the third machine (step 1306). In one embodiment, the first agent 1002 receives the request via a communications channel, such as an ICA channel, between the first agent 1002 and the second agent 1010. In another embodiment, the first agent 1002 receives the request when a user of the second client 102b selects a graphical representation of the remote resource 1210 depicted in the local display 1012 of the computing environment 1004. For example, the first agent 1002 may generate a graphical representation of the remote resource 1210, such as an icon, and incorporate the icon into the computing environment 1004 upon receiving the identification of the remote resource 1210 from the remote machine 106'. In still another embodiment, the first agent 1002 receives a request for execution of the remote resource 1210 when a resource executing on one of the first client 102a and the second client 102b attempts to access a resource, such as a file, that requires the execution of the remote resource 1210.
[0257] The first agent directs a second agent on the second machine to request execution of the remote resource provided by the third machine (step 1308). In one embodiment, the first agent 1002 directs the second agent 1010 to execute another agent 1010' on the second client 102b and to instruct the second agent 1010' to request execution of the remote resource 1210 provided by the remote machine 106'. In another embodiment, the second agent 1010' is a presentation layer protocol client, such as an ICA or RDP client. In still another embodiment, the second agent 1010' establishes a connection to the remote machine 106' and requests execution of the remote resource 1210. In yet another embodiment, the second agent 1010' receives output data generated by the execution of the remote resource 1210 and displays the output data in the local window 1014. In some embodiments, the second agent 1010 directs the first agent 1002 to request execution of the remote resource 1210 on server 106 upon making a determination the requested resource is unavailable or not executable from the second client 102b.
[0258] The second agent transmits, to the first agent, an identification of a change to a z-order entry of a local window on the second machine displaying output data generated by the remote resource provided by the third machine (step 1310). In one embodiment the second agent 1010 transmits the identification of the change to the z-order entry of the local window 1014 to the first agent 1002. In another embodiment the second agent 1010 transmits the identification of the change to the z-order entry of the local window 1014 to the first agent 1002 as described above in connection with FIG. 1OA, 1OB, 1 IA, and 1 IB.
[0259] The first agent modifies a z-order entry for a proxy window representing the local window in a z-order list on the first machine, responsive to the identification of the change, the proxy window having a z-order entry in a z-order list associated with the plurality of windows in the computing environment (step 1312). In some embodiments, the first agent 1002 modifies a z-order entry for a proxy window 1008 as described above in connection with FIG. 1OA, 1OB, HA, and HB.
[0260] In some embodiments, methods and systems described herein provide functionality for addressing limitations on displaying output data on mobile devices, including limitations resulting from the relatively small display form factor and the limited computing capacity. In one of these embodiments, methods and systems are provided for turning the mobile device into an application delivery channel. In another of these embodiments, no changes are required of existing desktop applications to implement these
methods and systems, and a wide variety of applications can be accessed on such an application-delivery enabled mobile device.
[0261] Referring now to FIG. 14A, a block diagram depicts one embodiment of a system for executing on a mobile computing device a virtual input device for interacting with a resource, wherein output data for the resource is displayed on an external display device. In brief overview, the system includes a mobile computing device 102, a native display 201 on the mobile computing device 102, an external display device 202, and a virtual input device 1402. The mobile computing device 102 executes a first resource 1404. The external display device 202 displays output data generated by the execution of the first resource 1404. The mobile computing device 102 executes a second resource 1406. The second resource 1406 generates a virtual input device 1402. The virtual input device 1402 leverages one or more capabilities of the mobile computing device 102. The virtual input device 1402 allows interaction by a user of the mobile computing device 102 with the display of output data generated by the execution of the first resource 1404.
[0262] Referring now to FIG. 14A, and in greater detail, the system includes a mobile computing device 102. In one embodiment, the system is a mobile device 102 as described above in connection with FIGs. 1-13. In another embodiment, the system includes a native display 201 on the mobile computing device 102, an external display device 202, and a virtual input device 1402. The mobile computing device 102 executes a first resource 1404.
[0263] The external display device 202 displays output data generated by the execution of the first resource 1404. In one embodiment, the external display device 202 is a monitor. In another embodiment, the external display device 202 is a projector. In still another embodiment, the mobile computing device 102 is connected or attached to an external display device 202, such as a projector that transmits output data to an external display surface. In various embodiments, the external display device 202 may be integrated into the mobile computing device 102. For example, the mobile computing device 102 may include a pico projector, which directs a display of output data generated by a resource executing on the mobile computing device 102 to an external display surface such as a projection screen, wall, or other display surface. In some embodiments, the mobile computing device 102 may include a projection module based on a microelectromechanical system (MEMS), such as, for example and without limitation, a PicoP Display Engine manufactured by Microvision, Inc., of Redmond, WA.
[0264] The mobile computing device 102 executes a second resource 1406. The second resource 1406 generates a virtual input device 1402 for interaction by a user of the mobile computing device with the display of output data generated by the execution of the first resource. In various embodiments, the mobile computing device 102 may execute the second resource 1406 according to the input needs of the first resource 1404, the identification of the mobile computing device 102, or both. In one embodiment, a virtual input device 1402 includes a graphical representation of a keyboard. In another embodiment, a virtual input device 1402 includes a virtual track pad. In still another embodiments, a virtual input device 1402 includes a virtual pointing device, such as a cursor which may be manipulated by interacting with the virtual input device 1402.
[0265] The virtual input device 1402 may be displayed on the native display 201 on the mobile computing device 102. The virtual input device 1402 allows interaction by a user of the mobile computing device 102 with the display of output data generated by the execution of the first resource 1404. In some embodiments, the native display 201 on the mobile computing device 102 includes a touch-screen. In one of these embodiments, a touchscreen sensor includes a touch-responsive surface that detects touch input from a user of the mobile computing device 102. In another of these embodiments, the virtual input device 140 is displayed on the touch-responsive surface. In still another of these embodiments, the touch-screen sensor redirects an identification of a location of the touch input on the touch- responsive surface to an operating system on the mobile computing device 102. In another of these embodiments, the operating system redirects the identification of the location of the touch input to the second resource 1406 for processing. In still another of these embodiments, the second resource 1406 correlates the location of the touch input with a displayed user interaction element. In yet another of these embodiments, the second resource 1406 transmits an identification of a user interaction with the displayed user interaction element to the first resource 1404.
[0266] In one embodiment, the virtual input device 1402 includes both a virtual keyboard and a virtual pointing device and interactions with these virtual components are redirected to the mobile device capabilities or to secondary external devices. For example, in another embodiments, mobile devices that have full QWERTY keyboards and cursor positioning buttons or joystick keys can have input received by the virtual keyboard and mouse redirected to those capabilities. In still another embodiment, mobile devices with
touch screens can be repurposed as a "soft keyboard" and "soft mouse" and the input received by the virtual keyboard and mouse can be redirected directly to an executing resource. In still even another embodiment, input data received by a mobile device with an externally connected keyboard or mouse can be redirected to the virtual keyboard and mouse.
[0267] In some embodiments, a first resource 1404 may also accept input from other devices, such as camera, microphone, point-of-sale (POS) device, etc. In one of these embodiments, a virtual input device 1402 may provide a virtualized version of any such devices and can redirect data communication for these devices. For instance, and in another of these embodiments, if a mobile computing device 102 has a built-in camera, an application displayed on an external display device 202 and accepting input from a camera could be directed to use the mobile device camera. In still another of these embodiments, and by way of example, an application that has audio output could be directed, for example, to the mobile device's speaker or headphone jack.
[0268] Referring now to FIG. 14B, a screen shot depicts an embodiment of a mobile computing device 102 displaying a virtual input device 1402 on the native display 201, wherein the virtual input device 1402 permits the user to interact with the output data associated with a resource 1406, such as a word processing application. In this embodiment, the resource 1406 may be executing on the mobile computing device 102. When the mobile computing device 102 is connected to the external display device 202, the window management service 206 may detect the connection and display output data associated with the resource 1406 on the external display device 202 instead of the native display 201 of the mobile computing device 102. In this embodiment, the external display device 202 is a projector that displays the output data on a surface, such as a wall.
[0269] In response to output data being transferred to the external display device 202, the mobile computing device 102 may execute a resource that generates a virtual input device 1402. The mobile computing device 102 may evaluate the resource 1406 to determine its input needs. In some embodiments, the mobile computing device 102 may determine that a resource accepts alphanumeric input, audio input, video input, or any other type of input or combinations thereof. In this embodiment, the resource 1402 is a word processing application that needs at least alphanumeric input. In response, the mobile computing device 102 executes a resource to generate a virtual or "soft" keyboard for a virtual input device 1402. The virtual keyboard may be displayed on the native display 201, such as a touch-
screen. The touch-screen may detect user interactions and redirect information associated with the user interactions to the virtual input device 1402. The virtual input device 1402 may communicate the resources to the resource 1402.
[0270] Referring now to FIG. 14C, a screen shot depicts an embodiment of a mobile computing device displaying a virtual input device 1402 for interaction by a user with a display of output data associated with a resource. In this embodiment, the mobile computing device 102 executes a resource to generate at least a virtual trackpad. The virtual trackpad may be displayed on the native display 201, such as a touch-screen, of the mobile computing device 102. When the touch-screen detects a user interaction, the touch-screen may redirect information associated with the user interaction to the resource generating the virtual input device 1402. The virtual input device 1402 may process the information and determine a user command. The virtual input device 1402 may communicate the user command to the resource being displayed on an external display device (not shown), and the resource being displayed may output data corresponding to the user command.
[0271] The embodiment in FIG. 14C also depicts a virtual input device 1402 that repurposes a keyboard on the mobile computing device 102 for the resource being displayed on an external display device. In this embodiment, the mobile computing device 102 displays a keyboard generated by an application installed on the device. The touch-screen may detect a user interaction and transmit information associated with the user interaction to the keyboard application. The keyboard application may interpret the user interaction to determine a user command, such as entering or deleting alphanumeric characters. The keyboard application may communicate the user command to the virtual input device 1402, which may transmit information about the user command to the resource whose output data is being displayed on an external display device. In various embodiments, one resource may generate a virtual trackpad and another resource may repurpose the keyboard on the mobile computing device 102. In other embodiments, the same resource may generate a virtual input device 1402 that generates the virtual trackpad and repurposes the keyboard.
[0272] Referring now to FIG. 15, a flow diagram depicts one embodiment of a method for displaying, on a screen of a mobile computing device, a virtual input device for interacting with a resource generating output data displayed on a display surface external to the mobile computing device. The method includes executing, by a mobile computing device, a first resource (1502). The method includes directing a display, on an external
display device, of output data generated by the executing first resource (1504). The method includes executing, on the mobile computing device, a second resource generating a virtual input device (1506). The method includes displaying, on a screen of the mobile computing device, the virtual input device (1508). The method includes transmitting, by the second resource to the first resource, an indication of an interaction by a user of the mobile computing device with the virtual input device (1510). In some embodiments, the mobile device 102 is repurposed as a virtual input device that transmits user input to either a locally- executing application or, via presentation-layer protocols, to a remotely-executing application.
[0273] Referring now to FIG. 15, and in greater detail, a mobile computing device
102 executes a first resource (1502). In one embodiment, the mobile computing device 102 executes an application generating output data for display to a user of the mobile computing device. In some embodiments, the first resource requests, from a remote computing device, execution of a remote Iy- located resource. In one of these embodiments, a first resource communicates with the remote computing device according to a presentation layer protocol, for example, by communicating via the ICA protocol, or the RDP protocol. In another of these embodiments, the first resource directs the display of remotely-generated output data on a display device associated with the mobile computing device. In other embodiments, the first resource is stored on the mobile computing device, executes locally, and directs the display of locally-generated output data on a display device associated with the mobile computing device.
[0274] A display, by an external display device, of output data generated by the executing first resource is directed (1504). In one embodiment, a window management service directs the display of output data to the external display device 124b. In one embodiment, an operating system executing on the mobile computing device 102 and in communication with the external display device 124b directs the display of output data to the external display device 124b. In another embodiment, the external display device projects the output data to an external display surface, such as a wall, screen, or other surface. In still another embodiment, the display of the output data occurs without modification to the output data generated by the execution of the first resource. In some embodiments, the output data is rendered on a secondary physical display, such as an embedded or external micro projector, heads-up display, laptop, HDTV, or other display. In one of these embodiments, displayed
output from the first resource is shown via the secondary display and is, therefore, not limited to only the display capabilities of the mobile computing device display.
[0275] The mobile computing device executes a second resource generating a virtual input device (1506). In one embodiment, the second resource communicates with the first resource. In another embodiment, the second resource identifies a type of virtual input device to display; for example, the second resource may select a type of virtual input device to display based on a type of interaction supported by the first resource.
[0276] The virtual input device is displayed on a screen of the mobile computing device (1508). In one embodiment, the virtual input device is displayed in conjunction with an existing virtual input device; for example, the mobile computing device may display a virtual (or "soft") keyboard and the second resource may generate a second virtual input device, such as a virtual trackpad displayed in conjunction with the virtual keyboard. In another embodiment, the second resource may generate a customized virtual input device displayed on the mobile computing device in place of the display of output data generated by the first resource. In still another embodiment, the display of the output data generated by the first resource is separated from the display of the virtual input device, instead of combined into a singled display surface, such as either an internal or an external display device.
[0277] The second resource transmits, to the first resource, an indication of an interaction by a user of the mobile computing device with the virtual input device (1510). In some embodiments, the second resource communicates with the first resource, allowing a user to interact with the first resource via the virtual input device generated by the second resource and displayed on a separate display device than the first resource.
[0278] In one embodiment, the first resource is a client application executing on the mobile computing device 102 and communicating with a remote computing device 106 according to a presentation layer protocol. In another embodiment, the first resource requests execution of a remote resource by the remote computing device. In still another embodiment, the first resource receives, from the remote computing device, output data generated by an execution of the remote resource on the remote computing device. In still even another embodiment, the first resource transmits, to the remote computing device, execution of a remote resource, and receives, from a second remote computing device, output data generated by an execution of the remote resource on the second remote computing device. In yet
another embodiment, the mobile computing device directs a display of the remotely- generated output data on the external display device.
[0279] In some embodiments, the methods and systems described herein provide functionality for smart translation of user interaction with endpoint devices. In one of these embodiments, methods and systems are provided for translating a user interaction with an endpoint device into a command for processing by a remotely-executing resource. In another of these embodiments, for example, a resource may execute on a first computing device and generate output data that is displayed on a second computing device, which may be referred to as an endpoint device; a user interaction with the endpoint device may be translated into a command that is transmitted to the resource on the first computing device for processing.
[0280] In some embodiments, a resource executing on a first computing device provides a user interface allowing a user to interact with output data generated by a resource. In one of these embodiments, for example, output data may include one or more graphical user interface elements, such as a cursor or a scroll bar, displayed with data, such as the text in a word processing application or an image in an image viewing or editing software, and the graphical user interface elements provide a mechanism by which a user of the resource may interact with data generated by the resource. In another of these embodiments, the type of user interface varies based upon the computing environment from which the user views the output data. As described above, a first type of user interface may be provided when a user accesses output data from a mobile computing device and a second type of user interface may be provided when a user accesses output data from a different type of computing device, such as a laptop, or when the user has access to a different type of display device.
[0281] In some embodiments, methods for interacting with a user interface are different across different user interfaces - for example, users may access a pointing device (such as a mouse) to manipulate an image of a cursor on a screen in order to interact with a graphical user interface element, or users may interact with a touch screen on a display device and use their fingers to tap on or drag graphical user interface elements from one portion of the display to another. In other embodiments, a method for interacting with a first type of user interface (for example, connecting a mouse to a computing device and using the mouse to control a cursor shown on a display device to interact with the first type of user interface) does not work when applied to a second type of user interface (for example, if a second type of user interface accepts as input a user touch to a screen, the second type of user interface
may not provide functionality for displaying a cursor or for accepting input from a pointing device); even if the method does work, it may be a suboptimal method for interacting with the second type of user interface or it may only provide limited functionality for interacting with the second type of user interface. In one of these embodiments, a first type of user interface may be preferable for use on a first type of computing device - for example, a user interface that supports a user entering commands by touching a screen may be better suited for use on a computing device that includes a touch screen than on one that does not. In still other embodiments, a remotely-executing application generates and displays a first type of user interface intended for access via a first type of computing device but which is displayed to a user accessing the user interface from a second type of computing device. For example, a user may access a first type of user interface that was designed for access from a computing device including a pointing device from a computing device that includes a touch screen but not a pointing device.
[0282] In some embodiments, the methods and systems described herein provide functionality for translating input of a first type into a second type. In one of these embodiments, input provided by a user via a first type of interface is translated into a command that a second type of interface can process. In another of these embodiments, and by way of example, a user may view data including a user interface displaying interface elements such as a scroll bar and accepting input commands from the user via a pointing device; however, the user may enter a command via a touch screen and the command is translated into a format that the user interface can process. In still another of these embodiments, and as another example, a user viewing a first section of data may have requested a display of a second section of data, for example by using a touch screen to scroll through the data; the command to scroll from the first section of data to the second may be translated into a command to select a graphical user interface element for scrolling down (such as an arrow in a scroll bar) and transmitted to the resource generating the data. In yet another of these embodiments, implementation of this functionality allows a user of a first type of computing device to interact with a display of a first type of user interface using a second type of user interface element optimized for use on the first type of computing device.
[0283] Referring now to FIG. 16, a flow diagram depicts one embodiment of a method for translating a user interaction with an input/output device of a first type into a command for processing by a user interface supporting interaction with an input/output
device of a second type. In brief overview, the method includes providing an input/output device of a first type for interaction with a user interface of a first type (1602). The method includes identifying a user interaction with a user interface of a second type via the input/output device of the first type (1604). The method includes generating, responsive to the identified user interaction, a command formatted for processing by the user interface of the second type according to a protocol for interacting with an input/output device of a second type (1606). The method includes processing the command by the user interface of the second type (1608).
[0284] Referring now to FIG. 16, and in greater detail, an input/output device of a first type is provided for interaction with a user interface of a first type (1602). In one embodiment, the input/output device of the first type is a touch screen. In another embodiment, the user interface of the first type includes functionality for processing touch screen-based interactions with user interface elements displayed by the user interface of the first type. In still another embodiment, the input/output device of the first type is an input/output device such as a keyboard or pointing device. In still even another embodiment, the user interface of the first type includes a graphical element (such as a cursor) displayed to a user and with which the user interacts via the input/output device of the first type. In yet another embodiment, the user interface of the first type provides functionality for processing interactions with the graphical element received via input/output devices such as keyboards or pointing devices.
[0285] In some embodiments, a user interface of a second type is displayed to a user.
In one of these embodiments, a user of a mobile computing device 102 views a user interface of the second type generated by a resource executing on the mobile computing device 102. In another of these embodiments, a user of a mobile computing device 102 views a user interface of the second type generated by a resource executing on a remote computing device 106. In still another of these embodiments, the user interface of the second type does not include functionality for processing user input generated by user interactions with the input/output device of the first type. In yet another of these embodiments, and by way of example, a user interface of the second type may be a windowing environment that displays widgets such as pull-down menus, scroll bars, and task bars and that includes functionality for processing user input generated by a user interaction with a pointing device or keyboard
(e.g., a user clicking on a scroll bar with a mouse), but may not include functionality for processing user input generated by a user interaction with a touch screen.
[0286] A user interaction with a user interface of a second type via the input/output device of the first type is identified (1604). In one embodiment, an interaction management component identifies the user interaction. In another embodiment, the user interaction is transmitted to an interaction management component for processing. In an embodiment in which the input/output device of the first type is a touch screen, a user may touch the screen of a mobile computing device in an attempt to manipulate data displayed by the user interface of the second type, such as a windowing environment displaying a pull-down menu.
[0287] A command is generated responsive to the identified user interaction, the command formatted for processing by the user interface of the second type according to a protocol for interacting with an input/output device of a second type (1606). In one embodiment, an interaction management component accesses a mapping, table, or other data structure storing configuration data to generate the command. In another embodiment, the interaction management component analyzes the user interaction with input/output device of the first type to identify a request or command intended by the user interaction; for example, the interaction management component may determine that a user touching a section of a touch screen intending to request access to data or to generate a command to modify an existing display of data. In still another embodiment, the interaction management component generates a type of command that is typically a result of a user interaction with the user interface of the second type. In yet another embodiment, and by way of example, the interaction management component determines that a user has interacted with a touch screen in order to request a modification of an existing display of data and the interaction management component generates a command equivalent to a command that would have been generated if the user had accessed an input/output device such as a mouse to select a graphical user interface element.
[0288] The command is processed by the user interface of the second type (1608). In one embodiment, the interaction management component transmits the command to a resource generating the user interface of the second type. In another embodiment, the interaction management component transmits the command to a resource executing on the mobile computing device 102. In another embodiment, the interaction management
component transmits the command to a resource executing on a second computing device 106.
[0289] The systems and methods described above may be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture. The article of manufacture may be a floppy disk, a hard disk, a CD-ROM, a flash memory card, a PROM, a RAM, a ROM, or a magnetic tape. In general, the computer-readable programs may be implemented in any programming language, LISP, PERL, C, C++, PROLOG, or any byte code language such as JAVA. The software programs may be stored on or in one or more articles of manufacture as object code.
[0290] Having described certain embodiments of methods and systems for displaying and activating disconnected application sessions, it will now become apparent to one of skill in the art that other embodiments incorporating the concepts of the invention may be used. Therefore, the invention should not be limited to certain embodiments.
Claims
1. A method for displaying, on an external display device connected to a mobile computing device, output data associated with at least one resource comprising: receiving, by a virtual graphics driver on the mobile computing device, an identification of an external display device connected to the mobile computing device; selecting, by a window management service on the mobile computing device according to the identification, at least one resource to display on the external display device; positioning, by the window management service, output data associated with the at least one resource in a virtual screen space; and transmitting, by the virtual graphics driver, the output data to the external display device according to the virtual screen space.
2. The method of claim 1, further comprising allocating, by the window management service, a screen space corresponding to the external display device in the virtual screen space according to the identification.
3. The method of claim 2, further comprising allocating, by the window management service, the screen space corresponding to the external display device according to a resolution of the external display device.
4. The method of claim 1, further comprising removing, by the window management service, output data associated with the at least one resource from a position in the virtual screen space corresponding to a native display of the mobile computing device.
5. The method of claim 1, further comprising positioning, by the window management service, the output data in the virtual screen space according to a type of the resource.
6. The method of claim 1, further comprising positioning, by the window management service, the output data in the virtual screen space according to a user association between the resource and a cell in a grid display layout.
7. The method of claim 1, further comprising positioning, by the window management service, the output data in the virtual screen space according to a policy associated with cells in a grid display layout.
8. The method of claim 1, further comprising re-positioning, by the window management service, the output data in the virtual screen space according to an alteration of a grid display layout.
9. The method of claim 1, further comprising positioning, by the window management service, the output data in the virtual screen space according to a user command to display the output data in an empty cell.
10. The method of claim 1, further comprising positioning, by the window management service, the output data in the virtual screen space according to a policy associated with a dynamic display layout.
11. The method of claim 1 , further comprising positioning, by the window management service, the output data in the virtual screen space according to a default position associated with the resource.
12. The method of claim 1 , further comprising receiving, by the window management service, information associated with a user interaction with the output data.
13. The method of claim 12, further comprising re-positioning, by the window management service, the output data according to the information associated with the user interaction.
14. The method of claim 13, further comprising re-positioning, by the window management service, the output data to a position in the virtual screen space corresponding to a native display of the mobile computing device or a different external display device.
15. The method of claim 13, further comprising obscuring, by the window management service, output data associated with a different resource.
16. The method of claim 12, further comprising re-sizing, by the window management service, the output data associated with the resource.
17. A mobile computing device for displaying output data associated with at least one resource on an external display device comprising: a window management service that selects, according to an identification of the external display device, the at least one resource to display on the external display device and positions output data associated with the at least one resource in a virtual screen space; and a virtual graphics driver that transmits the output data to the external display device according to the virtual screen space.
18. The device of claim 17, wherein the window management service allocates a screen space corresponding to the external display device in the virtual screen space according to the identification.
19. The device of claim 17, wherein the window management service allocates a screen space corresponding to the external display device in the virtual screen space according to a resolution of the external display device.
20. The device of claim 17, wherein the window management service removes output data associated with the at least one resource from a position in the virtual screen space corresponding to a native display of the mobile computing device.
21. A method for displaying, on an external display device connected to a mobile computing device, a user interface for a resource comprising: receiving from the resource, by a window management service on the mobile computing device, a first user interface for display on a native display of the mobile computing device; transmitting, by a virtual graphics driver on the mobile computing device, the first user interface to the native display; transmitting, by the window management service, an identification of the external display device to the resource; receiving from the resource, by the window management service, a second user interface according to the identification, the second user interface being distinct from the first user interface; and transmitting, by the virtual graphics driver, the second user interface to the external display device.
22. The method of claim 21, further comprising receiving from the resource, by the window management service, the second user interface optimized for the external display device.
23. The method of claim 21, further comprising transmitting, by the window management service, at least one property of the external display device to the resource.
24. The method of claim 22, further comprising transmitting, by the window management service, an indication that a display on the external display device has a larger resolution or size than the native display.
25. The method of claim 21, further comprising receiving from the resource, by the window management service, a default user interface in the absence of a user interface optimized for the external display device.
26. The method of claim 21, further comprising receiving from the resource, by the window management service, a communication that a user interface optimized for the external display device is unavailable.
27. The method of claim 21, further comprising transmitting, by the window management system, an identification of the second user interface and an instruction to display the second user interface to the external display device.
28. A method for transferring execution of a virtualized resource from a mobile computing device to an external display device comprising: receiving, by a server from the mobile computing device, an identification of an external display device connected to the mobile computing device; identifying, by the server, a virtual machine being delivered to the external display device; disconnecting, by the server, the virtualized resource from a virtual machine being delivered to the mobile computing device; and connecting, by the server, the virtualized resource to the virtual machine being delivered to the external display device.
29. The method of claim 28, wherein the virtualized resource is a virtualized application.
30. The method of claim 28, further comprising transferring, by the server, information related to the state of execution of the virtualized resource from the virtual machine being to delivered to the mobile computing device to the virtual machine being delivered to the external display device.
31. A method of repurposing a mobile computing device as a virtual input device for a first resource being displayed on an external display device comprising: evaluating, by the server, input capabilities of the mobile computing device and input needs of the first resource; and delivering according to the evaluation, by the server to the mobile computing device, a second resource that generates a virtual input device.
32. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that generates a virtual keyboard, virtual track pad, virtual pointing device, virtual camera, virtual microphone, virtual point-of-sale device, virtual speaker, virtual headphone jack, or any combination thereof.
33. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that allows user interaction with output data of the first resource.
34. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that transmits an identification of a user interaction with the virtual input device to the first resource.
35. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that generates at least one user interaction element.
36. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that redirects user interactions with the virtual input device to the input capabilities of the mobile computing device.
37. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that correlates a location of a touch input with a displayed user interaction element.
38. The method of claim 31, further comprising delivering, by the server to the mobile computing device, a third resource that generates another virtual input device in conjunction with the virtual input device of the second resource.
39. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource for display in conjunction with the first resource on a single display surface.
40. The method of claim 31, further comprising delivering, by the server to the mobile computing device, the second resource that generates a virtual input device for a second external device connected to the mobile computing device.
41. A system for repurposing a mobile computing device as a virtual input device for a first resource being displayed on an external display device comprising: a server that evaluates input capabilities of the mobile computing device and input needs of the first resource and delivers to the mobile computing device, according to the evaluation, a second resource that generates a virtual input device.
42. The system of claim 41, wherein the second resource generates a virtual keyboard, virtual track pad, virtual pointing device, virtual camera, virtual microphone, virtual point-of- sale device, virtual speaker, virtual headphone jack, or any combination thereof.
43. The system of claim 41, wherein the second resource allows user interaction with output data of the first resource.
44. The system of claim 41 , wherein the second resource transmits an identification of a user interaction with the virtual input device to the first resource.
45. The system of claim 41, wherein the second resource generates at least one user interaction element.
46. The system of claim 41, wherein the second resource redirects user interactions with the virtual input device to the input capabilities of the mobile computing device.
47. The system of claim 41, wherein the second resource correlates a location of a touch input with a displayed user interaction element.
48. The system of claim 41, wherein the server delivers a third resource that generates another virtual input device in conjunction with the virtual input device of the second resource.
49. The system of claim 41, wherein the second resource is displayed in conjunction with the first resource on a single display surface.
50. The system of claim 41, wherein the second resource generates a virtual input device for a second external device connected to the mobile computing device.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN200980117721.8A CN102027450B (en) | 2008-05-20 | 2009-05-20 | Methods and systems for using external display devices with a mobile computing device |
EP09751522.5A EP2283421B1 (en) | 2008-05-20 | 2009-05-20 | Methods and systems for using external display devices with a mobile computing device |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US5455108P | 2008-05-20 | 2008-05-20 | |
US61/054,551 | 2008-05-20 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2009143294A2 true WO2009143294A2 (en) | 2009-11-26 |
WO2009143294A3 WO2009143294A3 (en) | 2010-01-21 |
Family
ID=41016978
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2009/044741 WO2009143294A2 (en) | 2008-05-20 | 2009-05-20 | Methods and systems for using external display devices with a mobile computing device |
Country Status (4)
Country | Link |
---|---|
US (1) | US20100138780A1 (en) |
EP (1) | EP2283421B1 (en) |
CN (1) | CN102027450B (en) |
WO (1) | WO2009143294A2 (en) |
Cited By (80)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2011084857A1 (en) * | 2010-01-06 | 2011-07-14 | Apple Inc. | Apparatus and method having multiple application display modes including mode with display resolution of another apparatus |
GB2478994A (en) * | 2010-03-26 | 2011-09-28 | Nec Corp | Displaying content delivered from a remote server via a mobile radio communication device in combination with the content from the mobile communication device |
EP2369468A1 (en) * | 2010-02-26 | 2011-09-28 | Research In Motion Limited | Computer to handheld device virtualization system |
WO2011123840A2 (en) | 2010-04-01 | 2011-10-06 | Citrix Systems, Inc. | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
EP2385689A1 (en) * | 2010-05-07 | 2011-11-09 | LG Electronics Inc. | Mobile terminal and controlling method thereof |
WO2012027640A1 (en) * | 2010-08-27 | 2012-03-01 | Google Inc. | Switching display modes based on connection state |
EP2463772A1 (en) * | 2010-11-29 | 2012-06-13 | NEC CASIO Mobile Communications, Ltd. | Method for dynamically allocating an external peripheral to device application |
WO2012092316A1 (en) * | 2010-12-31 | 2012-07-05 | Motorola Mobility, Inc. | Method and system for adapting the usage of external display with mobile device |
US20120238214A1 (en) * | 2011-03-15 | 2012-09-20 | Nokia Corporation | Mobile User Apparatus and a Mobile Accessory Apparatus for the Mobile User Apparatus |
CN102754067A (en) * | 2010-02-12 | 2012-10-24 | 三星电子株式会社 | Data operation method for terminal including three-piece display units and terminal supporting the same |
US8301723B2 (en) | 2010-02-26 | 2012-10-30 | Research In Motion Limited | Computer to handheld device virtualization system |
CN102779063A (en) * | 2011-05-10 | 2012-11-14 | 北京磊友信息科技有限公司 | Method for providing application invocation and server |
EP2544082A1 (en) * | 2010-03-05 | 2013-01-09 | Fujitsu Limited | Image display system, information processing apparatus, display apparatus, and image display method |
WO2013012415A1 (en) * | 2011-07-18 | 2013-01-24 | Research In Motion Limited | Electronic device and method of controlling same |
US8363009B1 (en) | 2010-12-30 | 2013-01-29 | Google Inc. | Enhanced input using touch screen |
EP2562638A1 (en) * | 2011-08-23 | 2013-02-27 | Samsung Electronics Co., Ltd. | Display method and apparatus in portable terminal |
EP2565781A1 (en) * | 2011-08-30 | 2013-03-06 | Casio Computer Co., Ltd. | Image display terminal, image display control method and recording medium recording image display control program |
EP2639689A1 (en) * | 2012-03-15 | 2013-09-18 | Habook Information Technology Inc. | A multi-source multiple documents comparison display (presentation or discussion) system |
EP2645232A1 (en) * | 2012-03-30 | 2013-10-02 | Bixolon Co., Ltd. | Multi-display POS system using portable computer |
US8698845B2 (en) | 2010-01-06 | 2014-04-15 | Apple Inc. | Device, method, and graphical user interface with interactive popup views |
EP2761495A1 (en) * | 2011-09-26 | 2014-08-06 | Intel Corporation | Simulation of web applications and secondary devices in a web browser, web application development tools, and methods using the same |
US8806369B2 (en) | 2011-08-26 | 2014-08-12 | Apple Inc. | Device, method, and graphical user interface for managing and interacting with concurrently open software applications |
EP2799978A1 (en) * | 2013-04-30 | 2014-11-05 | Konica Minolta, Inc. | Image processing system, image processing apparatus, portable information terminal, program |
US9052925B2 (en) | 2010-04-07 | 2015-06-09 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
EP2992527A1 (en) * | 2013-04-30 | 2016-03-09 | Microsoft Technology Licensing, LLC | Second screen view with multitasking |
WO2016048809A1 (en) * | 2014-09-24 | 2016-03-31 | Microsoft Technology Licensing, Llc | Presentation of computing environment on multiple devices |
WO2016073137A1 (en) * | 2014-11-07 | 2016-05-12 | Qualcomm Incorporated | Systems and methods for performing display mirroring |
WO2016133522A1 (en) * | 2015-02-19 | 2016-08-25 | Citrix Systems, Inc. | Systems and methods for providing adapted multi-monitor topology support in a virtualization environment |
RU2603278C2 (en) * | 2011-11-08 | 2016-11-27 | Сони Корпорейшн | Transmitting device, display control device, method of transmitting content and recording medium |
US9513801B2 (en) | 2010-04-07 | 2016-12-06 | Apple Inc. | Accessing electronic notifications and settings icons with gestures |
EP3126956A1 (en) * | 2014-03-31 | 2017-02-08 | Thales | Display method and device for multi-window application software |
US9602729B2 (en) | 2015-06-07 | 2017-03-21 | Apple Inc. | Devices and methods for capturing and interacting with enhanced digital images |
EP3143492A1 (en) * | 2014-05-14 | 2017-03-22 | Microsoft Technology Licensing, LLC | Transferring content between graphical user interfaces |
US9619076B2 (en) | 2012-05-09 | 2017-04-11 | Apple Inc. | Device, method, and graphical user interface for transitioning between display states in response to a gesture |
US9645732B2 (en) | 2015-03-08 | 2017-05-09 | Apple Inc. | Devices, methods, and graphical user interfaces for displaying and using menus |
WO2017079030A1 (en) * | 2015-11-05 | 2017-05-11 | Topcon Positioning Systems, Inc. | Monitoring and control display system and method using multiple displays in a work environment |
WO2017090916A1 (en) | 2015-11-25 | 2017-06-01 | Samsung Electronics Co., Ltd. | Managing display of information on multiple devices based on context for a user task |
US9674426B2 (en) | 2015-06-07 | 2017-06-06 | Apple Inc. | Devices and methods for capturing and interacting with enhanced digital images |
US9678640B2 (en) | 2014-09-24 | 2017-06-13 | Microsoft Technology Licensing, Llc | View management architecture |
US9684429B2 (en) | 2013-03-15 | 2017-06-20 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
US9753639B2 (en) | 2012-05-09 | 2017-09-05 | Apple Inc. | Device, method, and graphical user interface for displaying content associated with a corresponding affordance |
US9778771B2 (en) | 2012-12-29 | 2017-10-03 | Apple Inc. | Device, method, and graphical user interface for transitioning between touch input to display output relationships |
US9785305B2 (en) | 2015-03-19 | 2017-10-10 | Apple Inc. | Touch input cursor manipulation |
US9823839B2 (en) | 2012-05-09 | 2017-11-21 | Apple Inc. | Device, method, and graphical user interface for displaying additional information in response to a user contact |
US9823831B2 (en) | 2010-04-07 | 2017-11-21 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
EP3186955A4 (en) * | 2015-07-09 | 2017-12-27 | Genetec Inc. | Security video monitoring client |
US9860306B2 (en) | 2014-09-24 | 2018-01-02 | Microsoft Technology Licensing, Llc | Component-specific application presentation histories |
US9870192B2 (en) | 2015-02-19 | 2018-01-16 | Citrix Systems, Inc. | Systems and methods for providing adapted multi-monitor topology support in a virtualization environment |
US9886184B2 (en) | 2012-05-09 | 2018-02-06 | Apple Inc. | Device, method, and graphical user interface for providing feedback for changing activation states of a user interface object |
US9959025B2 (en) | 2012-12-29 | 2018-05-01 | Apple Inc. | Device, method, and graphical user interface for navigating user interface hierarchies |
US9990121B2 (en) | 2012-05-09 | 2018-06-05 | Apple Inc. | Device, method, and graphical user interface for moving a user interface object based on an intensity of a press input |
US9996231B2 (en) | 2012-05-09 | 2018-06-12 | Apple Inc. | Device, method, and graphical user interface for manipulating framed graphical objects |
US10007400B2 (en) | 2010-12-20 | 2018-06-26 | Apple Inc. | Device, method, and graphical user interface for navigation of concurrently open software applications |
US10025684B2 (en) | 2014-09-24 | 2018-07-17 | Microsoft Technology Licensing, Llc | Lending target device resources to host device computing environment |
US10042542B2 (en) | 2012-05-09 | 2018-08-07 | Apple Inc. | Device, method, and graphical user interface for moving and dropping a user interface object |
US10048757B2 (en) | 2015-03-08 | 2018-08-14 | Apple Inc. | Devices and methods for controlling media presentation |
US10073963B2 (en) | 2014-05-14 | 2018-09-11 | Microsoft Technology Licensing, Llc | Claiming data from a virtual whiteboard |
US10078442B2 (en) | 2012-12-29 | 2018-09-18 | Apple Inc. | Device, method, and graphical user interface for determining whether to scroll or select content based on an intensity theshold |
US10095396B2 (en) | 2015-03-08 | 2018-10-09 | Apple Inc. | Devices, methods, and graphical user interfaces for interacting with a control object while dragging another object |
US10095391B2 (en) | 2012-05-09 | 2018-10-09 | Apple Inc. | Device, method, and graphical user interface for selecting user interface objects |
WO2018202728A1 (en) * | 2017-05-02 | 2018-11-08 | Barco N.V. | Presentation server, data relay method and method for generating virtual pointer |
US10126930B2 (en) | 2012-05-09 | 2018-11-13 | Apple Inc. | Device, method, and graphical user interface for scrolling nested regions |
US10162452B2 (en) | 2015-08-10 | 2018-12-25 | Apple Inc. | Devices and methods for processing touch inputs based on their intensities |
US10175864B2 (en) | 2012-05-09 | 2019-01-08 | Apple Inc. | Device, method, and graphical user interface for selecting object within a group of objects in accordance with contact intensity |
US10175757B2 (en) | 2012-05-09 | 2019-01-08 | Apple Inc. | Device, method, and graphical user interface for providing tactile feedback for touch-based operations performed and reversed in a user interface |
US10270819B2 (en) | 2014-05-14 | 2019-04-23 | Microsoft Technology Licensing, Llc | System and method providing collaborative interaction |
US10310732B2 (en) | 2013-03-15 | 2019-06-04 | Apple Inc. | Device, method, and graphical user interface for concurrently displaying a plurality of settings controls |
CN109923568A (en) * | 2016-11-08 | 2019-06-21 | 微软技术许可有限责任公司 | Mobile data for data analysis sees clearly platform |
WO2019147303A1 (en) * | 2018-01-26 | 2019-08-01 | Citrix Systems, Inc. | Virtual computing system providing local screen sharing from hosted collaboration applications |
US10387100B2 (en) | 2013-03-15 | 2019-08-20 | Topcon Positioning Systems, Inc. | Monitoring and control display system and method |
US10437333B2 (en) | 2012-12-29 | 2019-10-08 | Apple Inc. | Device, method, and graphical user interface for forgoing generation of tactile output for a multi-contact gesture |
US10448111B2 (en) | 2014-09-24 | 2019-10-15 | Microsoft Technology Licensing, Llc | Content projection |
US10496260B2 (en) | 2012-05-09 | 2019-12-03 | Apple Inc. | Device, method, and graphical user interface for pressure-based alteration of controls in a user interface |
US10620781B2 (en) | 2012-12-29 | 2020-04-14 | Apple Inc. | Device, method, and graphical user interface for moving a cursor according to a change in an appearance of a control icon with simulated three-dimensional characteristics |
US10635296B2 (en) | 2014-09-24 | 2020-04-28 | Microsoft Technology Licensing, Llc | Partitioned application presentation across devices |
CN111135557A (en) * | 2019-12-26 | 2020-05-12 | 三星电子(中国)研发中心 | Interaction method and device for multiple screens |
US10740059B2 (en) | 2012-03-22 | 2020-08-11 | Ricoh Company, Ltd. | Communication system, communication device, and computer program |
CN111788819A (en) * | 2018-03-26 | 2020-10-16 | 三星电子株式会社 | Mobile electronic device and method for forwarding user input to an application based on an input device |
US11068153B2 (en) | 2012-05-09 | 2021-07-20 | Apple Inc. | Device, method, and graphical user interface for displaying user interface objects corresponding to an application |
EP4206897A4 (en) * | 2020-09-29 | 2024-03-06 | Huawei Technologies Co., Ltd. | Distributed display method of interface, and electronic device and communication system |
Families Citing this family (191)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9678624B2 (en) * | 2011-09-27 | 2017-06-13 | Z124 | Unified desktop triad control user interface for a phone manager |
US9529494B2 (en) | 2011-09-27 | 2016-12-27 | Z124 | Unified desktop triad control user interface for a browser |
US9268518B2 (en) | 2011-09-27 | 2016-02-23 | Z124 | Unified desktop docking rules |
US9715252B2 (en) | 2011-08-24 | 2017-07-25 | Z124 | Unified desktop docking behavior for window stickiness |
US9405459B2 (en) | 2011-08-24 | 2016-08-02 | Z124 | Unified desktop laptop dock software operation |
US20130080909A1 (en) * | 2011-09-27 | 2013-03-28 | Paul E. Reeves | Unified desktop docking behaviour for an auxillary monitor |
US20130104062A1 (en) * | 2011-09-27 | 2013-04-25 | Z124 | Unified desktop input segregation in an application manager |
US20130080941A1 (en) * | 2011-09-27 | 2013-03-28 | Paul E. Reeves | Unified desktop triad control user interface for an application launcher |
US8990713B2 (en) | 2011-09-27 | 2015-03-24 | Z124 | Unified desktop triad control user interface for an application manager |
US9507930B2 (en) | 2003-04-25 | 2016-11-29 | Z124 | Physical key secure peripheral interconnection |
US20130198867A1 (en) | 2011-12-09 | 2013-08-01 | Z124 | A Docking Station for Portable Devices Providing Authorized Power Transfer and Facility Access |
US9965067B2 (en) | 2007-09-19 | 2018-05-08 | T1V, Inc. | Multimedia, multiuser system and associated methods |
US20130342489A1 (en) * | 2008-08-13 | 2013-12-26 | Michael R. Feldman | Multimedia, multiuser system and associated methods |
US8015232B2 (en) * | 2007-10-11 | 2011-09-06 | Roaming Keyboards Llc | Thin terminal computer architecture utilizing roaming keyboard files |
US8279238B2 (en) * | 2007-10-16 | 2012-10-02 | Panasonic Corporation | Image display device and image display method |
JP5047870B2 (en) * | 2008-04-17 | 2012-10-10 | 株式会社日立製作所 | Master management system, master management method, and master management program |
JP5335287B2 (en) * | 2008-05-29 | 2013-11-06 | キヤノン株式会社 | Display device, control method thereof, and program |
US20100064251A1 (en) * | 2008-09-05 | 2010-03-11 | International Business Machines Corporation | Toggling window display state by screen in a multi-screened desktop environment |
US8594467B2 (en) * | 2008-12-19 | 2013-11-26 | Microsoft Corporation | Interactive virtual display system for ubiquitous devices |
US20100169791A1 (en) * | 2008-12-31 | 2010-07-01 | Trevor Pering | Remote display remote control |
US20160320938A9 (en) * | 2009-03-17 | 2016-11-03 | Litera Technologies, LLC | System and Method for the Auto-Detection and Presentation of Pre-Set Configurations for Multiple Monitor Layout Display |
US20100268762A1 (en) * | 2009-04-15 | 2010-10-21 | Wyse Technology Inc. | System and method for scrolling a remote application |
US8676926B2 (en) | 2009-04-15 | 2014-03-18 | Wyse Technology L.L.C. | System and method for handling remote drawing commands |
US9578113B2 (en) | 2009-04-15 | 2017-02-21 | Wyse Technology L.L.C. | Method and apparatus for transferring remote session data |
US9553953B2 (en) | 2009-04-15 | 2017-01-24 | Dell Products L.P. | Method and apparatus for extending capabilities of a virtualization domain to support features available in a normal desktop application |
US9189124B2 (en) * | 2009-04-15 | 2015-11-17 | Wyse Technology L.L.C. | Custom pointer features for touch-screen on remote client devices |
US9448815B2 (en) * | 2009-04-15 | 2016-09-20 | Wyse Technology L.L.C. | Server-side computing from a remote client device |
US9106696B2 (en) * | 2009-04-15 | 2015-08-11 | Wyse Technology L.L.C. | Method and apparatus for portability of a remote session |
US9479895B2 (en) * | 2009-04-23 | 2016-10-25 | International Business Machines Corporation | Location-oriented services |
US9241062B2 (en) * | 2009-05-20 | 2016-01-19 | Citrix Systems, Inc. | Methods and systems for using external display devices with a mobile computing device |
US9547511B2 (en) | 2009-06-05 | 2017-01-17 | Microsoft Technology Licensing, Llc | Language-based model for asynchronous operations |
US8954872B2 (en) * | 2009-08-24 | 2015-02-10 | Ati Technologies Ulc | Method and apparatus for configuring a plurality of displays into a single large surface display |
US20110112819A1 (en) * | 2009-11-11 | 2011-05-12 | Sony Corporation | User interface systems and methods between a portable device and a computer |
JP5531612B2 (en) * | 2009-12-25 | 2014-06-25 | ソニー株式会社 | Information processing apparatus, information processing method, program, control target device, and information processing system |
JP2011134278A (en) * | 2009-12-25 | 2011-07-07 | Toshiba Corp | Information processing apparatus and pointing control method |
US9715332B1 (en) | 2010-08-26 | 2017-07-25 | Cypress Lake Software, Inc. | Methods, systems, and computer program products for navigating between visual components |
US10397639B1 (en) | 2010-01-29 | 2019-08-27 | Sitting Man, Llc | Hot key systems and methods |
US20110219331A1 (en) * | 2010-03-02 | 2011-09-08 | International Business Machines Corporation | Window resize on remote desktops |
US10009647B2 (en) * | 2010-03-02 | 2018-06-26 | Qualcomm Incorporated | Reducing end-to-end latency for communicating information from a user device to a receiving device via television white space |
US20110225566A1 (en) * | 2010-03-10 | 2011-09-15 | Microsoft Corporation | Testing user interfaces in multiple execution environments |
US8621365B2 (en) * | 2010-04-06 | 2013-12-31 | Asustek Computer Inc. | File sharing method and system |
US9998580B2 (en) * | 2010-04-26 | 2018-06-12 | Hu-Do Ltd. | Computing device operable to work in conjunction with a companion electronic device |
US20110307831A1 (en) * | 2010-06-10 | 2011-12-15 | Microsoft Corporation | User-Controlled Application Access to Resources |
US9001050B2 (en) * | 2010-07-19 | 2015-04-07 | Red Hat Israel, Ltd. | Touch screen emulation for a virtual machine |
US20120030584A1 (en) * | 2010-07-30 | 2012-02-02 | Brian Bian | Method and apparatus for dynamically switching between scalable graphical user interfaces for mobile devices |
JP2012038210A (en) * | 2010-08-10 | 2012-02-23 | Sony Corp | Information processing unit, information processing method, computer program, and content display system |
DE102010033883B4 (en) | 2010-08-10 | 2021-12-23 | Volkswagen Ag | Method for data transfer between a mobile device and a motor vehicle |
WO2012020868A1 (en) * | 2010-08-13 | 2012-02-16 | 엘지전자 주식회사 | Mobile terminal, display device and method for controlling same |
US9043706B2 (en) * | 2010-08-31 | 2015-05-26 | Anders Nancke-Krogh | System and method for using state replication between application instances to provide a collaborative desktop environment |
US9159298B2 (en) * | 2010-09-08 | 2015-10-13 | Lg Electronics Inc. | Terminal and contents sharing method for terminal |
US8659553B1 (en) * | 2010-09-20 | 2014-02-25 | Google Inc. | Orientation-based touchscreen display |
EP2624112B1 (en) * | 2010-09-30 | 2019-06-19 | Rakuten, Inc. | Viewing device, viewing method, non-transitory computer-readable recording medium whereon program is recorded, and script program |
EP2622446A4 (en) | 2010-10-01 | 2015-06-10 | Z124 | Long drag gesture in user interface |
EP2622491A4 (en) * | 2010-10-01 | 2015-05-27 | Z124 | Systems and methods relating to user interfaces for docking portable electronic devices |
US9400585B2 (en) * | 2010-10-05 | 2016-07-26 | Citrix Systems, Inc. | Display management for native user experiences |
CN102096574B (en) * | 2010-12-01 | 2012-10-03 | 广东威创视讯科技股份有限公司 | Multiwindow montage display method |
KR101738527B1 (en) * | 2010-12-07 | 2017-05-22 | 삼성전자 주식회사 | Mobile device and control method thereof |
US8984120B2 (en) * | 2010-12-16 | 2015-03-17 | International Business Machines Corporation | Determining an unexpected disconnect event constraint within a text exchange session |
US8688734B1 (en) | 2011-02-04 | 2014-04-01 | hopTo Inc. | System for and methods of controlling user access and/or visibility to directories and files of a computer |
US8151279B1 (en) * | 2011-03-28 | 2012-04-03 | Google Inc. | Uniform event handling across multiple computing devices |
US8392935B2 (en) | 2011-03-28 | 2013-03-05 | Google Inc. | Uniform event handling across multiple computing devices |
CN103620544B (en) * | 2011-04-13 | 2017-02-15 | 雷蛇(亚太)私人有限公司 | Computer peripheral display and communication device providing an adjunct 3d user interface |
US9767195B2 (en) | 2011-04-21 | 2017-09-19 | Touchstream Technologies, Inc. | Virtualized hosting and displaying of content using a swappable media player |
US20130027613A1 (en) * | 2011-05-03 | 2013-01-31 | Lg Electronics Inc. | Image display apparatus, portable terminal, and methods for operating the same |
US8760395B2 (en) | 2011-05-31 | 2014-06-24 | Microsoft Corporation | Gesture recognition techniques |
GB2492789B (en) * | 2011-07-12 | 2018-01-03 | Denso Corp | Displays |
US20130019179A1 (en) * | 2011-07-14 | 2013-01-17 | Digilink Software, Inc. | Mobile application enhancements |
TWM419969U (en) * | 2011-07-14 | 2012-01-01 | Chuan-Shi Wu | Screen transfer control device for smart phone |
US10564791B2 (en) * | 2011-07-21 | 2020-02-18 | Nokia Technologies Oy | Method and apparatus for triggering a remote data entry interface |
US20130027315A1 (en) * | 2011-07-25 | 2013-01-31 | Arther Sing Hook Teng | Techniques to display an input device on a mobile device |
US9013510B2 (en) * | 2011-07-29 | 2015-04-21 | Google Inc. | Systems and methods for rendering user interface elements in accordance with a device type |
US8782716B2 (en) * | 2011-07-29 | 2014-07-15 | Google Inc. | Systems and methods for rendering user interface objects in accordance with a variable scaling factor |
US20160124698A1 (en) | 2011-08-24 | 2016-05-05 | Z124 | Unified desktop triad control user interface for an application launcher |
US20130104051A1 (en) | 2011-09-27 | 2013-04-25 | Z124 | Unified desktop big brother application pools |
US9244491B2 (en) | 2011-08-31 | 2016-01-26 | Z124 | Smart dock for auxiliary devices |
US9383770B2 (en) | 2011-08-31 | 2016-07-05 | Z124 | Mobile device that docks with multiple types of docks |
KR101932529B1 (en) * | 2011-09-16 | 2018-12-28 | 삼성전자 주식회사 | Integrated Operation Method For Social Network Service function and System supporting the same |
US20130080899A1 (en) * | 2011-09-27 | 2013-03-28 | Paul E. Reeves | Unified desktop big brother applications |
US11416131B2 (en) * | 2011-09-27 | 2022-08-16 | Z124 | Unified desktop input segregation in an application manager |
US9495012B2 (en) * | 2011-09-27 | 2016-11-15 | Z124 | Secondary single screen mode activation through user interface activation |
US9703468B2 (en) * | 2011-09-27 | 2017-07-11 | Z124 | Unified desktop independent focus in an application manager |
KR101855938B1 (en) * | 2011-10-06 | 2018-05-09 | 엘지전자 주식회사 | Mobile terminal and operation control method thereof |
US20130127738A1 (en) * | 2011-11-23 | 2013-05-23 | Microsoft Corporation | Dynamic scaling of touch sensor |
US9612724B2 (en) | 2011-11-29 | 2017-04-04 | Citrix Systems, Inc. | Integrating native user interface components on a mobile device |
US8635637B2 (en) | 2011-12-02 | 2014-01-21 | Microsoft Corporation | User interface presenting an animated avatar performing a media reaction |
US9100685B2 (en) | 2011-12-09 | 2015-08-04 | Microsoft Technology Licensing, Llc | Determining audience state or interest using passive sensor data |
US9164544B2 (en) | 2011-12-09 | 2015-10-20 | Z124 | Unified desktop: laptop dock, hardware configuration |
US20130159555A1 (en) * | 2011-12-20 | 2013-06-20 | Microsoft Corporation | Input commands |
US9654821B2 (en) | 2011-12-30 | 2017-05-16 | Sonos, Inc. | Systems and methods for networked music playback |
JP5230827B1 (en) | 2012-01-25 | 2013-07-10 | キヤノン株式会社 | Information processing apparatus, method thereof, and program |
KR101195520B1 (en) * | 2012-02-22 | 2012-10-29 | 알서포트 주식회사 | Remote control method and remote control apparatus for controlling a mobile terminal |
US9817442B2 (en) * | 2012-02-28 | 2017-11-14 | Razer (Asia-Pacific) Pte. Ltd. | Systems and methods for presenting visual interface content |
US9275142B2 (en) * | 2012-02-29 | 2016-03-01 | Nokia Technologies Oy | Method and apparatus for multi-browser web-based applications |
US8558790B2 (en) * | 2012-03-04 | 2013-10-15 | Lg Electronics Inc. | Portable device and control method thereof |
EP2823385B1 (en) | 2012-03-04 | 2020-03-18 | Microsoft Technology Licensing, LLC | Portable device and control method thereof |
US20130234959A1 (en) | 2012-03-06 | 2013-09-12 | Industry-University Cooperation Foundation Hanyang University | System and method for linking and controlling terminals |
KR101325840B1 (en) * | 2012-03-13 | 2013-11-05 | 주식회사 팬택 | Method for managing sink device, source device and wlan system for the same |
US8898687B2 (en) | 2012-04-04 | 2014-11-25 | Microsoft Corporation | Controlling a media program based on a media reaction |
KR20130116976A (en) * | 2012-04-17 | 2013-10-25 | 엘지전자 주식회사 | Mobile terminal and method for controlling thereof |
CA2775700C (en) | 2012-05-04 | 2013-07-23 | Microsoft Corporation | Determining a future portion of a currently presented media program |
JP2013242357A (en) * | 2012-05-18 | 2013-12-05 | Ricoh Co Ltd | Information processing apparatus, information processing method, and program |
US9419848B1 (en) | 2012-05-25 | 2016-08-16 | hopTo Inc. | System for and method of providing a document sharing service in combination with remote access to document applications |
US8856907B1 (en) | 2012-05-25 | 2014-10-07 | hopTo Inc. | System for and methods of providing single sign-on (SSO) capability in an application publishing and/or document sharing environment |
US8713658B1 (en) | 2012-05-25 | 2014-04-29 | Graphon Corporation | System for and method of providing single sign-on (SSO) capability in an application publishing environment |
KR101919788B1 (en) * | 2012-05-31 | 2018-11-19 | 엘지전자 주식회사 | Mobile terminal and method for controlling thereof |
US8738826B2 (en) * | 2012-06-08 | 2014-05-27 | Apple Inc. | System and method for display mirroring |
TWI533197B (en) * | 2012-06-19 | 2016-05-11 | 緯創資通股份有限公司 | Method for outputting image and electronic device for using the same |
JP5342044B1 (en) * | 2012-06-22 | 2013-11-13 | 株式会社東芝 | Information processing apparatus, information processing method, and program |
US20130346858A1 (en) * | 2012-06-25 | 2013-12-26 | Neyrinck Llc | Remote Control of Audio Application and Associated Sub-Windows |
US9674587B2 (en) | 2012-06-26 | 2017-06-06 | Sonos, Inc. | Systems and methods for networked music playback including remote add to queue |
US9239812B1 (en) | 2012-08-08 | 2016-01-19 | hopTo Inc. | System for and method of providing a universal I/O command translation framework in an application publishing environment |
KR101961860B1 (en) * | 2012-08-28 | 2019-03-25 | 삼성전자주식회사 | User terminal apparatus and contol method thereof |
CN104704461B (en) * | 2012-09-06 | 2018-04-27 | 丰田自动车株式会社 | Mobile communication terminal, car-mounted device and onboard system |
US20140075377A1 (en) * | 2012-09-10 | 2014-03-13 | Samsung Electronics Co. Ltd. | Method for connecting mobile terminal and external display and apparatus implementing the same |
KR102080146B1 (en) * | 2012-09-10 | 2020-02-24 | 삼성전자 주식회사 | Operating Method associated with connected Electronic Device with External Display Device and Electronic Device supporting the same |
EP3355561B1 (en) * | 2012-09-10 | 2021-04-28 | Samsung Electronics Co., Ltd. | Method for connecting mobile terminal and external display and apparatus implementing the same |
TWI637312B (en) * | 2012-09-19 | 2018-10-01 | 三星電子股份有限公司 | Method for displaying information on transparent display device, display device therewith, and computer-readable recording medium therefor |
US9213515B2 (en) * | 2012-09-24 | 2015-12-15 | At&T Intellectual Property I, L.P. | On-demand multi-screen computing |
US9239727B1 (en) | 2012-10-17 | 2016-01-19 | Amazon Technologies, Inc. | Configurable virtual machines |
US9274742B2 (en) * | 2012-10-25 | 2016-03-01 | PixiOnCloud, Inc. | Visual-symbolic control of remote devices having display-based user interfaces |
US20140125554A1 (en) * | 2012-11-07 | 2014-05-08 | Shanghai Powermo Information Tech. Co. Ltd. | Apparatus and algorithm to implement smart mirroring for a multiple display system |
CN103838481B (en) * | 2012-11-27 | 2017-09-29 | 联想(北京)有限公司 | A kind of data processing method and electronic equipment |
US10282088B2 (en) | 2012-12-06 | 2019-05-07 | Samsung Electronics Co., Ltd. | Configuration of application execution spaces and sub-spaces for sharing data on a mobile tough screen device |
EP4213001A1 (en) | 2012-12-06 | 2023-07-19 | Samsung Electronics Co., Ltd. | Display device and method of controlling the same |
US20140184603A1 (en) * | 2012-12-27 | 2014-07-03 | Nvidia Corporation | Method to improve usability of high pixel density displays |
KR102131646B1 (en) * | 2013-01-03 | 2020-07-08 | 삼성전자주식회사 | Display apparatus and control method thereof |
US20140267074A1 (en) * | 2013-03-14 | 2014-09-18 | Qualcomm Incorporated | System and method for virtual user interface controls in multi-display configurations |
US9792014B2 (en) | 2013-03-15 | 2017-10-17 | Microsoft Technology Licensing, Llc | In-place contextual menu for handling actions for a listing of items |
KR102043049B1 (en) * | 2013-04-01 | 2019-11-11 | 삼성전자 주식회사 | Operating Method of Application and Electronic Device, and Outputting Device supporting the same |
US9361371B2 (en) | 2013-04-16 | 2016-06-07 | Sonos, Inc. | Playlist update in a media playback system |
US9501533B2 (en) | 2013-04-16 | 2016-11-22 | Sonos, Inc. | Private queue for a media playback system |
US9247363B2 (en) | 2013-04-16 | 2016-01-26 | Sonos, Inc. | Playback queue transfer in a media playback system |
US9600595B2 (en) * | 2013-05-21 | 2017-03-21 | International Business Machines Corporation | Modification of windows across multiple displays |
US9684484B2 (en) | 2013-05-29 | 2017-06-20 | Sonos, Inc. | Playback zone silent connect |
US9880727B2 (en) * | 2013-05-29 | 2018-01-30 | Microsoft Technology Licensing, Llc | Gesture manipulations for configuring system settings |
EP2816761A1 (en) * | 2013-06-17 | 2014-12-24 | Thomson Licensing | Wifi display compatible network gateway |
WO2014209271A1 (en) * | 2013-06-25 | 2014-12-31 | Thomson Licensing | Multidisciplinary nomad sub-screen system |
KR102208436B1 (en) * | 2013-08-06 | 2021-01-27 | 삼성전자주식회사 | Method for displaying and an electronic device thereof |
CN104427384A (en) * | 2013-08-26 | 2015-03-18 | 联想(北京)有限公司 | Data processing method, image demonstration method, data processing apparatus and demonstration equipment |
US10719829B2 (en) | 2013-09-09 | 2020-07-21 | Touchtunes Music Corporation | Techniques for processing pin-inclusive transactions in connection with an electronic device |
US10451874B2 (en) * | 2013-09-25 | 2019-10-22 | Seiko Epson Corporation | Image display device, method of controlling image display device, computer program, and image display system |
KR102087987B1 (en) * | 2013-10-04 | 2020-03-11 | 삼성전자주식회사 | Master device, client device, and method for screen mirroring thereof |
KR102180810B1 (en) * | 2013-10-16 | 2020-11-19 | 삼성전자주식회사 | Electronic apparatus, method for executing of application and computer-readable recording medium |
KR102107404B1 (en) | 2013-10-30 | 2020-05-07 | 삼성전자주식회사 | Apparatus for sharing application and method for controlling thereof |
WO2015070070A1 (en) * | 2013-11-07 | 2015-05-14 | Touchtunes Music Corporation | Techniques for generating electronic menu graphical user interface layouts for use in connection with electronic devices |
US9550118B2 (en) * | 2013-11-13 | 2017-01-24 | Gaijin Entertainment Corp. | Method for simulating video games on mobile device |
US20150178032A1 (en) * | 2013-12-19 | 2015-06-25 | Qualcomm Incorporated | Apparatuses and methods for using remote multimedia sink devices |
US9641925B2 (en) * | 2014-01-02 | 2017-05-02 | Zippy Technology Corp. | Headphone wireless expansion device capable of switching among multiple targets and voice control method thereof |
US9582825B2 (en) | 2014-03-24 | 2017-02-28 | Touchtunes Music Corporation | Systems, apparatuses, and methods for ordering items from an electronic menu, and servicing thereof |
CN104977982A (en) * | 2014-04-03 | 2015-10-14 | 鸿富锦精密工业(武汉)有限公司 | Universal electronic equipment combination |
AU2015267671B2 (en) | 2014-05-30 | 2018-04-19 | Apple Inc. | Transition from use of one device to another |
WO2015200542A1 (en) | 2014-06-24 | 2015-12-30 | Aeris Communications, Inc. | Communication between display and device utilizing a communication and display protocol |
US9684776B2 (en) | 2014-07-29 | 2017-06-20 | Google Inc. | Allowing access to applications based on user authentication |
US9690919B2 (en) * | 2014-07-29 | 2017-06-27 | Google Inc. | Allowing access to applications based on user capacitance |
US9818174B2 (en) * | 2014-09-24 | 2017-11-14 | Microsoft Technology Licensing, Llc | Streamlined handling of monitor topology changes |
US10055064B2 (en) * | 2014-10-29 | 2018-08-21 | Sony Corporation | Controlling multiple devices with a wearable input device |
US10949075B2 (en) | 2014-11-06 | 2021-03-16 | Microsoft Technology Licensing, Llc | Application command control for small screen display |
US20160132992A1 (en) | 2014-11-06 | 2016-05-12 | Microsoft Technology Licensing, Llc | User interface scaling for devices based on display size |
US10116748B2 (en) | 2014-11-20 | 2018-10-30 | Microsoft Technology Licensing, Llc | Vehicle-based multi-modal interface |
US20160182603A1 (en) * | 2014-12-19 | 2016-06-23 | Microsoft Technology Licensing, Llc | Browser Display Casting Techniques |
US10073599B2 (en) | 2015-01-07 | 2018-09-11 | Microsoft Technology Licensing, Llc | Automatic home screen determination based on display device |
TW201631464A (en) * | 2015-02-26 | 2016-09-01 | 鴻富錦精密工業(武漢)有限公司 | Control system and method for multi-display |
CN105988764A (en) * | 2015-02-26 | 2016-10-05 | 鸿富锦精密工业(武汉)有限公司 | Display control system and method |
US10297002B2 (en) * | 2015-03-10 | 2019-05-21 | Intel Corporation | Virtual touch pad method and apparatus for controlling an external display |
KR102433879B1 (en) * | 2015-08-21 | 2022-08-18 | 삼성전자주식회사 | Display apparatus and control method thereof |
US10200298B2 (en) * | 2015-10-26 | 2019-02-05 | Knowlio, Inc. | Array generation in a resource management system |
US9769596B2 (en) | 2015-11-24 | 2017-09-19 | International Business Machines Corporation | Mobile device output to external device |
US20170200024A1 (en) * | 2016-01-08 | 2017-07-13 | Samsung Electronics Co., Ltd. | Electronic device and method of securing the same |
CN107045508B (en) * | 2016-02-05 | 2020-03-03 | 腾讯科技(深圳)有限公司 | Application program processing method and device |
CN106095273B (en) * | 2016-06-07 | 2019-06-25 | 联想(北京)有限公司 | Information processing method and electronic equipment |
AU2017100667A4 (en) | 2016-06-11 | 2017-07-06 | Apple Inc. | Activity and workout updates |
DK201670622A1 (en) | 2016-06-12 | 2018-02-12 | Apple Inc | User interfaces for transactions |
CN106648497B (en) * | 2016-10-26 | 2020-03-13 | 京东方科技集团股份有限公司 | Multi-screen display management system, display driving method and screen switching method |
US10976984B2 (en) * | 2017-06-08 | 2021-04-13 | T1V, Inc. | Multi-group collaboration system and associated methods |
US11283912B2 (en) * | 2017-06-16 | 2022-03-22 | Huawei Technologies Co., Ltd. | Display method and device |
WO2019028855A1 (en) * | 2017-08-11 | 2019-02-14 | 深圳前海达闼云端智能科技有限公司 | Virtual display device, intelligent interaction method, and cloud server |
KR20190021016A (en) * | 2017-08-22 | 2019-03-05 | 삼성전자주식회사 | Electronic device and control method thereof |
KR102418297B1 (en) * | 2017-11-01 | 2022-07-08 | 삼성전자주식회사 | Electronic device providinging security notification and control method thereof |
KR20200055983A (en) * | 2018-11-14 | 2020-05-22 | 삼성전자주식회사 | Method for estimating electromagnatic signal radiated from device and an electronic device thereof |
US10984062B2 (en) * | 2019-01-07 | 2021-04-20 | Citrix Systems, Inc. | Device augmentation of real time communications |
US11880422B2 (en) | 2019-02-04 | 2024-01-23 | Cloudflare, Inc. | Theft prevention for sensitive information |
US10452868B1 (en) | 2019-02-04 | 2019-10-22 | S2 Systems Corporation | Web browser remoting using network vector rendering |
CN109992125B (en) * | 2019-03-29 | 2022-11-15 | 京东方科技集团股份有限公司 | Information input method, device and system |
CN110022161A (en) * | 2019-04-23 | 2019-07-16 | 京东方科技集团股份有限公司 | Transmitter, transmitting equipment, transmission device and the wireless method for throwing screen |
CN113542825B (en) * | 2020-04-20 | 2022-10-11 | 华为技术有限公司 | Screen projection display method, system, terminal device and storage medium |
CN111741349B (en) * | 2020-07-14 | 2022-08-02 | 北京简元科技有限公司 | Multimedia information special-shaped screen publishing method, device and equipment |
US11392291B2 (en) | 2020-09-25 | 2022-07-19 | Apple Inc. | Methods and interfaces for media control with dynamic feedback |
CN114327324A (en) * | 2020-09-29 | 2022-04-12 | 华为技术有限公司 | Distributed display method of interface, electronic equipment and communication system |
WO2022154784A1 (en) * | 2021-01-13 | 2022-07-21 | Hewlett-Packard Development Company, L.P. | Display settings changes based on locations |
JP7345144B2 (en) * | 2021-03-31 | 2023-09-15 | パナソニックIpマネジメント株式会社 | Vehicle display system, display method and program |
JP2023038507A (en) * | 2021-09-07 | 2023-03-17 | ルネサスエレクトロニクス株式会社 | Display controller and display control method |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2001097014A2 (en) * | 2000-06-12 | 2001-12-20 | Preworx (Proprietary) Limited | System for controlling a display of the user interface of a software application |
WO2002089441A1 (en) * | 2001-05-01 | 2002-11-07 | Meta4Hand Inc. | Wireless network computing |
US20030048275A1 (en) * | 2001-09-14 | 2003-03-13 | Ciolac Alec A. | System for providing multiple display support and method thereof |
EP1455267A2 (en) * | 2003-03-04 | 2004-09-08 | Kabushiki Kaisha Toshiba | Information processing apparatus and program |
US20060158385A1 (en) * | 2005-01-18 | 2006-07-20 | Nokia Corporation | User interface for different displays |
US20070010287A1 (en) * | 2005-06-24 | 2007-01-11 | Fujitsu Limited | Electronic apparatus, screen information output method and computer-readable storage medium |
US20070178891A1 (en) * | 2006-01-30 | 2007-08-02 | Louch John O | Remote control of electronic devices |
JP2007193788A (en) * | 2005-12-21 | 2007-08-02 | Masahiro Izutsu | Portable information communication device and web server exchanging information with it |
EP1858238A2 (en) * | 2006-05-18 | 2007-11-21 | Samsung Electronics Co., Ltd. | Display method and system for portable device using external display device |
WO2007135598A1 (en) * | 2006-05-19 | 2007-11-29 | Sony Ericsson Mobile Communications Ab | Method for configuring a mobile device to remotely control a controllable device and mobile device therefor |
US20080057929A1 (en) * | 2006-09-06 | 2008-03-06 | Byung Woo Min | Cell phone with remote control system |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6724403B1 (en) * | 1999-10-29 | 2004-04-20 | Surfcast, Inc. | System and method for simultaneous display of multiple information sources |
JP4753529B2 (en) * | 2001-02-22 | 2011-08-24 | 富士通株式会社 | Information processing apparatus, display control method for information processing apparatus, storage medium, and program thereof |
JP2003280630A (en) * | 2002-03-20 | 2003-10-02 | Toshiba Corp | Information processor and display control method used for the processor |
US7098868B2 (en) * | 2003-04-08 | 2006-08-29 | Microsoft Corporation | Display source divider |
KR100744380B1 (en) * | 2006-04-17 | 2007-07-30 | 삼성전자주식회사 | Apparatus and method for controlling display in mobile terminal having function of outputting video to an external display device |
US7657227B2 (en) * | 2006-08-03 | 2010-02-02 | International Business Machines Corporation | Method and system for dynamic display connectivity based on configuration information via RFID tag |
US20080115081A1 (en) * | 2006-11-09 | 2008-05-15 | Microsoft Corporation | Enhanced windows management feature |
JP2008211379A (en) * | 2007-02-23 | 2008-09-11 | Fujitsu Ltd | Display control program and portable terminal device |
CA2621744C (en) * | 2007-09-13 | 2016-10-04 | Research In Motion Limited | System and method for interfacing between a mobile device and a personal computer |
-
2009
- 2009-05-20 CN CN200980117721.8A patent/CN102027450B/en not_active Expired - Fee Related
- 2009-05-20 WO PCT/US2009/044741 patent/WO2009143294A2/en active Application Filing
- 2009-05-20 US US12/469,628 patent/US20100138780A1/en not_active Abandoned
- 2009-05-20 EP EP09751522.5A patent/EP2283421B1/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2001097014A2 (en) * | 2000-06-12 | 2001-12-20 | Preworx (Proprietary) Limited | System for controlling a display of the user interface of a software application |
WO2002089441A1 (en) * | 2001-05-01 | 2002-11-07 | Meta4Hand Inc. | Wireless network computing |
US20030048275A1 (en) * | 2001-09-14 | 2003-03-13 | Ciolac Alec A. | System for providing multiple display support and method thereof |
EP1455267A2 (en) * | 2003-03-04 | 2004-09-08 | Kabushiki Kaisha Toshiba | Information processing apparatus and program |
US20060158385A1 (en) * | 2005-01-18 | 2006-07-20 | Nokia Corporation | User interface for different displays |
US20070010287A1 (en) * | 2005-06-24 | 2007-01-11 | Fujitsu Limited | Electronic apparatus, screen information output method and computer-readable storage medium |
JP2007193788A (en) * | 2005-12-21 | 2007-08-02 | Masahiro Izutsu | Portable information communication device and web server exchanging information with it |
US20070178891A1 (en) * | 2006-01-30 | 2007-08-02 | Louch John O | Remote control of electronic devices |
EP1858238A2 (en) * | 2006-05-18 | 2007-11-21 | Samsung Electronics Co., Ltd. | Display method and system for portable device using external display device |
WO2007135598A1 (en) * | 2006-05-19 | 2007-11-29 | Sony Ericsson Mobile Communications Ab | Method for configuring a mobile device to remotely control a controllable device and mobile device therefor |
US20080057929A1 (en) * | 2006-09-06 | 2008-03-06 | Byung Woo Min | Cell phone with remote control system |
Cited By (155)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101483349B1 (en) * | 2010-01-06 | 2015-01-15 | 애플 인크. | Apparatus and method having multiple application display modes including mode with display resolution of another apparatus |
EP2354928A1 (en) * | 2010-01-06 | 2011-08-10 | Apple Inc. | Apparatus and method having multiple application display modes including mode with display resolution of another apparatus |
WO2011084857A1 (en) * | 2010-01-06 | 2011-07-14 | Apple Inc. | Apparatus and method having multiple application display modes including mode with display resolution of another apparatus |
US8698845B2 (en) | 2010-01-06 | 2014-04-15 | Apple Inc. | Device, method, and graphical user interface with interactive popup views |
US9569102B2 (en) | 2010-01-06 | 2017-02-14 | Apple Inc. | Device, method, and graphical user interface with interactive popup views |
CN102754067B (en) * | 2010-02-12 | 2016-04-27 | 三星电子株式会社 | For comprising the data manipulation method of the terminal of three display units and supporting the terminal of described method |
RU2596980C2 (en) * | 2010-02-12 | 2016-09-10 | Самсунг Электроникс Ко., Лтд. | Method for operations on data for terminal including three-section display units and terminal supporting said method |
US9317242B2 (en) | 2010-02-12 | 2016-04-19 | Samsung Electronics Co., Ltd. | Data operation method for terminal including three-piece display units and terminal supporting the same |
CN102754067A (en) * | 2010-02-12 | 2012-10-24 | 三星电子株式会社 | Data operation method for terminal including three-piece display units and terminal supporting the same |
EP2369468A1 (en) * | 2010-02-26 | 2011-09-28 | Research In Motion Limited | Computer to handheld device virtualization system |
US8533263B2 (en) | 2010-02-26 | 2013-09-10 | Blackberry Limited | Computer to handheld device virtualization system |
US8301723B2 (en) | 2010-02-26 | 2012-10-30 | Research In Motion Limited | Computer to handheld device virtualization system |
EP2544082A4 (en) * | 2010-03-05 | 2013-10-30 | Fujitsu Ltd | Image display system, information processing apparatus, display apparatus, and image display method |
EP2544082A1 (en) * | 2010-03-05 | 2013-01-09 | Fujitsu Limited | Image display system, information processing apparatus, display apparatus, and image display method |
GB2478994A (en) * | 2010-03-26 | 2011-09-28 | Nec Corp | Displaying content delivered from a remote server via a mobile radio communication device in combination with the content from the mobile communication device |
CN102918490B (en) * | 2010-04-01 | 2014-07-23 | 思杰系统有限公司 | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
US9740507B2 (en) | 2010-04-01 | 2017-08-22 | Citrix Systems, Inc. | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
EP2553561A4 (en) * | 2010-04-01 | 2016-03-30 | Citrix Systems Inc | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
CN102918490A (en) * | 2010-04-01 | 2013-02-06 | 思杰系统有限公司 | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
WO2011123840A2 (en) | 2010-04-01 | 2011-10-06 | Citrix Systems, Inc. | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
US10156962B2 (en) | 2010-04-07 | 2018-12-18 | Apple Inc. | Device, method and graphical user interface for sliding an application view by a predefined amount of sliding based on a touch input to a predefined button of a multifunction device |
US10101879B2 (en) | 2010-04-07 | 2018-10-16 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications using a three-dimensional stack of images of open applications |
US9052925B2 (en) | 2010-04-07 | 2015-06-09 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
US9513801B2 (en) | 2010-04-07 | 2016-12-06 | Apple Inc. | Accessing electronic notifications and settings icons with gestures |
US10891023B2 (en) | 2010-04-07 | 2021-01-12 | Apple Inc. | Device, method and graphical user interface for shifting a user interface between positions on a touch-sensitive display in response to detected inputs |
US9052926B2 (en) | 2010-04-07 | 2015-06-09 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
US9823831B2 (en) | 2010-04-07 | 2017-11-21 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
US10901601B2 (en) | 2010-04-07 | 2021-01-26 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
US9058186B2 (en) | 2010-04-07 | 2015-06-16 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
KR20110123348A (en) * | 2010-05-07 | 2011-11-15 | 엘지전자 주식회사 | Mobile terminal and method for controlling thereof |
CN102238279A (en) * | 2010-05-07 | 2011-11-09 | Lg电子株式会社 | Mobile terminal and controlling method thereof |
US8718556B2 (en) | 2010-05-07 | 2014-05-06 | Lg Electronics Inc. | Mobile terminal and controlling method thereof |
EP2385689A1 (en) * | 2010-05-07 | 2011-11-09 | LG Electronics Inc. | Mobile terminal and controlling method thereof |
KR101695810B1 (en) * | 2010-05-07 | 2017-01-13 | 엘지전자 주식회사 | Mobile terminal and method for controlling thereof |
WO2012027640A1 (en) * | 2010-08-27 | 2012-03-01 | Google Inc. | Switching display modes based on connection state |
US9715364B2 (en) | 2010-08-27 | 2017-07-25 | Google Inc. | Switching display modes based on connection state |
EP2625617A1 (en) * | 2010-11-29 | 2013-08-14 | NEC CASIO Mobile Communications, Ltd. | Method for dynamically allocating an external peripheral to device application |
EP2463772A1 (en) * | 2010-11-29 | 2012-06-13 | NEC CASIO Mobile Communications, Ltd. | Method for dynamically allocating an external peripheral to device application |
EP2625617A4 (en) * | 2010-11-29 | 2014-06-18 | Nec Casio Mobile Comm Ltd | Method for dynamically allocating an external peripheral to device application |
CN103201727A (en) * | 2010-11-29 | 2013-07-10 | Nec卡西欧移动通信株式会社 | Method for dynamically allocating an external peripheral to device application |
US10852914B2 (en) | 2010-12-20 | 2020-12-01 | Apple Inc. | Device, method, and graphical user interface for navigation of concurrently open software applications |
US11880550B2 (en) | 2010-12-20 | 2024-01-23 | Apple Inc. | Device, method, and graphical user interface for navigation of concurrently open software applications |
US10261668B2 (en) | 2010-12-20 | 2019-04-16 | Apple Inc. | Device, method, and graphical user interface for navigation of concurrently open software applications |
US10007400B2 (en) | 2010-12-20 | 2018-06-26 | Apple Inc. | Device, method, and graphical user interface for navigation of concurrently open software applications |
US11487404B2 (en) | 2010-12-20 | 2022-11-01 | Apple Inc. | Device, method, and graphical user interface for navigation of concurrently open software applications |
US8363009B1 (en) | 2010-12-30 | 2013-01-29 | Google Inc. | Enhanced input using touch screen |
US8446363B1 (en) | 2010-12-30 | 2013-05-21 | Google Inc. | Enhanced input using touch screen |
KR101561155B1 (en) * | 2010-12-31 | 2015-10-26 | 모토로라 모빌리티 엘엘씨 | Method and system for adapting the usage of external display with mobile device |
US8369893B2 (en) | 2010-12-31 | 2013-02-05 | Motorola Mobility Llc | Method and system for adapting mobile device to accommodate external display |
CN103282870A (en) * | 2010-12-31 | 2013-09-04 | 摩托罗拉移动有限责任公司 | Method and system for adapting the usage of external display with mobile device |
WO2012092316A1 (en) * | 2010-12-31 | 2012-07-05 | Motorola Mobility, Inc. | Method and system for adapting the usage of external display with mobile device |
WO2012123915A1 (en) * | 2011-03-15 | 2012-09-20 | Nokia Corporation | A mobile user apparatus and a mobile accessory apparatus for the mobile user apparatus |
US9124715B2 (en) | 2011-03-15 | 2015-09-01 | Nokia Technologies Oy | Mobile user apparatus and a mobile accessory apparatus for the mobile user apparatus |
US20120238214A1 (en) * | 2011-03-15 | 2012-09-20 | Nokia Corporation | Mobile User Apparatus and a Mobile Accessory Apparatus for the Mobile User Apparatus |
CN102779063A (en) * | 2011-05-10 | 2012-11-14 | 北京磊友信息科技有限公司 | Method for providing application invocation and server |
WO2013012415A1 (en) * | 2011-07-18 | 2013-01-24 | Research In Motion Limited | Electronic device and method of controlling same |
EP2562638A1 (en) * | 2011-08-23 | 2013-02-27 | Samsung Electronics Co., Ltd. | Display method and apparatus in portable terminal |
US10846043B2 (en) | 2011-08-23 | 2020-11-24 | Samsung Electronics Co., Ltd | Display method and apparatus in portable terminal |
EP3399403A1 (en) * | 2011-08-23 | 2018-11-07 | Samsung Electronics Co., Ltd. | Display method and apparatus in portable terminal |
US11237789B2 (en) | 2011-08-23 | 2022-02-01 | Samsung Electronics Co., Ltd. | Display method and apparatus in portable terminal |
US10430141B2 (en) | 2011-08-23 | 2019-10-01 | Samsung Electronics Co., Ltd | Display method and apparatus in portable terminal |
US8806369B2 (en) | 2011-08-26 | 2014-08-12 | Apple Inc. | Device, method, and graphical user interface for managing and interacting with concurrently open software applications |
US9207838B2 (en) | 2011-08-26 | 2015-12-08 | Apple Inc. | Device, method, and graphical user interface for managing and interacting with concurrently open software applications |
US9195430B2 (en) | 2011-08-30 | 2015-11-24 | Casio Computer Co., Ltd. | Image display terminal, image display control method and recording medium recording image display control program |
EP2565781A1 (en) * | 2011-08-30 | 2013-03-06 | Casio Computer Co., Ltd. | Image display terminal, image display control method and recording medium recording image display control program |
EP2761495A4 (en) * | 2011-09-26 | 2015-04-29 | Intel Corp | Simulation of web applications and secondary devices in a web browser, web application development tools, and methods using the same |
EP2761495A1 (en) * | 2011-09-26 | 2014-08-06 | Intel Corporation | Simulation of web applications and secondary devices in a web browser, web application development tools, and methods using the same |
US9142192B2 (en) | 2011-09-26 | 2015-09-22 | Intel Corporation | Simulation of web applications and secondary devices in a web browser, web application development tools, and methods using the same |
RU2603278C2 (en) * | 2011-11-08 | 2016-11-27 | Сони Корпорейшн | Transmitting device, display control device, method of transmitting content and recording medium |
EP2639689A1 (en) * | 2012-03-15 | 2013-09-18 | Habook Information Technology Inc. | A multi-source multiple documents comparison display (presentation or discussion) system |
US11256464B2 (en) | 2012-03-22 | 2022-02-22 | Ricoh Company, Ltd. | Communication system, communication device, and computer program |
US10740059B2 (en) | 2012-03-22 | 2020-08-11 | Ricoh Company, Ltd. | Communication system, communication device, and computer program |
EP2645232A1 (en) * | 2012-03-30 | 2013-10-02 | Bixolon Co., Ltd. | Multi-display POS system using portable computer |
US11068153B2 (en) | 2012-05-09 | 2021-07-20 | Apple Inc. | Device, method, and graphical user interface for displaying user interface objects corresponding to an application |
US10481690B2 (en) | 2012-05-09 | 2019-11-19 | Apple Inc. | Device, method, and graphical user interface for providing tactile feedback for media adjustment operations performed in a user interface |
US10126930B2 (en) | 2012-05-09 | 2018-11-13 | Apple Inc. | Device, method, and graphical user interface for scrolling nested regions |
US10095391B2 (en) | 2012-05-09 | 2018-10-09 | Apple Inc. | Device, method, and graphical user interface for selecting user interface objects |
US10175757B2 (en) | 2012-05-09 | 2019-01-08 | Apple Inc. | Device, method, and graphical user interface for providing tactile feedback for touch-based operations performed and reversed in a user interface |
US9823839B2 (en) | 2012-05-09 | 2017-11-21 | Apple Inc. | Device, method, and graphical user interface for displaying additional information in response to a user contact |
US10191627B2 (en) | 2012-05-09 | 2019-01-29 | Apple Inc. | Device, method, and graphical user interface for manipulating framed graphical objects |
US10175864B2 (en) | 2012-05-09 | 2019-01-08 | Apple Inc. | Device, method, and graphical user interface for selecting object within a group of objects in accordance with contact intensity |
US10042542B2 (en) | 2012-05-09 | 2018-08-07 | Apple Inc. | Device, method, and graphical user interface for moving and dropping a user interface object |
US9619076B2 (en) | 2012-05-09 | 2017-04-11 | Apple Inc. | Device, method, and graphical user interface for transitioning between display states in response to a gesture |
US10496260B2 (en) | 2012-05-09 | 2019-12-03 | Apple Inc. | Device, method, and graphical user interface for pressure-based alteration of controls in a user interface |
US9753639B2 (en) | 2012-05-09 | 2017-09-05 | Apple Inc. | Device, method, and graphical user interface for displaying content associated with a corresponding affordance |
US9996231B2 (en) | 2012-05-09 | 2018-06-12 | Apple Inc. | Device, method, and graphical user interface for manipulating framed graphical objects |
US9886184B2 (en) | 2012-05-09 | 2018-02-06 | Apple Inc. | Device, method, and graphical user interface for providing feedback for changing activation states of a user interface object |
US9990121B2 (en) | 2012-05-09 | 2018-06-05 | Apple Inc. | Device, method, and graphical user interface for moving a user interface object based on an intensity of a press input |
US9959025B2 (en) | 2012-12-29 | 2018-05-01 | Apple Inc. | Device, method, and graphical user interface for navigating user interface hierarchies |
US10437333B2 (en) | 2012-12-29 | 2019-10-08 | Apple Inc. | Device, method, and graphical user interface for forgoing generation of tactile output for a multi-contact gesture |
US10078442B2 (en) | 2012-12-29 | 2018-09-18 | Apple Inc. | Device, method, and graphical user interface for determining whether to scroll or select content based on an intensity theshold |
US9857897B2 (en) | 2012-12-29 | 2018-01-02 | Apple Inc. | Device and method for assigning respective portions of an aggregate intensity to a plurality of contacts |
US10101887B2 (en) | 2012-12-29 | 2018-10-16 | Apple Inc. | Device, method, and graphical user interface for navigating user interface hierarchies |
US10620781B2 (en) | 2012-12-29 | 2020-04-14 | Apple Inc. | Device, method, and graphical user interface for moving a cursor according to a change in an appearance of a control icon with simulated three-dimensional characteristics |
US9778771B2 (en) | 2012-12-29 | 2017-10-03 | Apple Inc. | Device, method, and graphical user interface for transitioning between touch input to display output relationships |
US11137898B2 (en) | 2013-03-15 | 2021-10-05 | Apple Inc. | Device, method, and graphical user interface for displaying a plurality of settings controls |
US10310732B2 (en) | 2013-03-15 | 2019-06-04 | Apple Inc. | Device, method, and graphical user interface for concurrently displaying a plurality of settings controls |
US10387100B2 (en) | 2013-03-15 | 2019-08-20 | Topcon Positioning Systems, Inc. | Monitoring and control display system and method |
US11989409B2 (en) | 2013-03-15 | 2024-05-21 | Apple Inc. | Device, method, and graphical user interface for displaying a plurality of settings controls |
US9684429B2 (en) | 2013-03-15 | 2017-06-20 | Apple Inc. | Device, method, and graphical user interface for managing concurrently open software applications |
EP2992527A1 (en) * | 2013-04-30 | 2016-03-09 | Microsoft Technology Licensing, LLC | Second screen view with multitasking |
EP2799978A1 (en) * | 2013-04-30 | 2014-11-05 | Konica Minolta, Inc. | Image processing system, image processing apparatus, portable information terminal, program |
JP2014215961A (en) * | 2013-04-30 | 2014-11-17 | コニカミノルタ株式会社 | Image processing system, image processing apparatus, mobile information terminal, and program |
EP3126956A1 (en) * | 2014-03-31 | 2017-02-08 | Thales | Display method and device for multi-window application software |
US10073963B2 (en) | 2014-05-14 | 2018-09-11 | Microsoft Technology Licensing, Llc | Claiming data from a virtual whiteboard |
US10270819B2 (en) | 2014-05-14 | 2019-04-23 | Microsoft Technology Licensing, Llc | System and method providing collaborative interaction |
EP3143492A1 (en) * | 2014-05-14 | 2017-03-22 | Microsoft Technology Licensing, LLC | Transferring content between graphical user interfaces |
US9769227B2 (en) | 2014-09-24 | 2017-09-19 | Microsoft Technology Licensing, Llc | Presentation of computing environment on multiple devices |
US20180007104A1 (en) | 2014-09-24 | 2018-01-04 | Microsoft Corporation | Presentation of computing environment on multiple devices |
WO2016048809A1 (en) * | 2014-09-24 | 2016-03-31 | Microsoft Technology Licensing, Llc | Presentation of computing environment on multiple devices |
KR20170059474A (en) * | 2014-09-24 | 2017-05-30 | 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 | Presentation of computing environment on multiple devices |
US9678640B2 (en) | 2014-09-24 | 2017-06-13 | Microsoft Technology Licensing, Llc | View management architecture |
US10824531B2 (en) | 2014-09-24 | 2020-11-03 | Microsoft Technology Licensing, Llc | Lending target device resources to host device computing environment |
US10277649B2 (en) | 2014-09-24 | 2019-04-30 | Microsoft Technology Licensing, Llc | Presentation of computing environment on multiple devices |
US10635296B2 (en) | 2014-09-24 | 2020-04-28 | Microsoft Technology Licensing, Llc | Partitioned application presentation across devices |
KR102570799B1 (en) | 2014-09-24 | 2023-08-24 | 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 | Presentation of computing environment on multiple devices |
KR102394296B1 (en) | 2014-09-24 | 2022-05-03 | 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 | Presentation of computing environment on multiple devices |
US10025684B2 (en) | 2014-09-24 | 2018-07-17 | Microsoft Technology Licensing, Llc | Lending target device resources to host device computing environment |
KR20220061273A (en) * | 2014-09-24 | 2022-05-12 | 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 | Presentation of computing environment on multiple devices |
US9860306B2 (en) | 2014-09-24 | 2018-01-02 | Microsoft Technology Licensing, Llc | Component-specific application presentation histories |
US10448111B2 (en) | 2014-09-24 | 2019-10-15 | Microsoft Technology Licensing, Llc | Content projection |
WO2016073137A1 (en) * | 2014-11-07 | 2016-05-12 | Qualcomm Incorporated | Systems and methods for performing display mirroring |
US10114600B2 (en) | 2015-02-19 | 2018-10-30 | Citrix Systems, Inc. | Systems and methods for providing adapted multi-monitor topology support in a virtualization environment |
WO2016133522A1 (en) * | 2015-02-19 | 2016-08-25 | Citrix Systems, Inc. | Systems and methods for providing adapted multi-monitor topology support in a virtualization environment |
US9870192B2 (en) | 2015-02-19 | 2018-01-16 | Citrix Systems, Inc. | Systems and methods for providing adapted multi-monitor topology support in a virtualization environment |
US9645732B2 (en) | 2015-03-08 | 2017-05-09 | Apple Inc. | Devices, methods, and graphical user interfaces for displaying and using menus |
US10095396B2 (en) | 2015-03-08 | 2018-10-09 | Apple Inc. | Devices, methods, and graphical user interfaces for interacting with a control object while dragging another object |
US10048757B2 (en) | 2015-03-08 | 2018-08-14 | Apple Inc. | Devices and methods for controlling media presentation |
US9785305B2 (en) | 2015-03-19 | 2017-10-10 | Apple Inc. | Touch input cursor manipulation |
US9860451B2 (en) | 2015-06-07 | 2018-01-02 | Apple Inc. | Devices and methods for capturing and interacting with enhanced digital images |
US9602729B2 (en) | 2015-06-07 | 2017-03-21 | Apple Inc. | Devices and methods for capturing and interacting with enhanced digital images |
US9674426B2 (en) | 2015-06-07 | 2017-06-06 | Apple Inc. | Devices and methods for capturing and interacting with enhanced digital images |
EP3186955A4 (en) * | 2015-07-09 | 2017-12-27 | Genetec Inc. | Security video monitoring client |
US10162452B2 (en) | 2015-08-10 | 2018-12-25 | Apple Inc. | Devices and methods for processing touch inputs based on their intensities |
US11182017B2 (en) | 2015-08-10 | 2021-11-23 | Apple Inc. | Devices and methods for processing touch inputs based on their intensities |
US10719289B2 (en) | 2015-11-05 | 2020-07-21 | Topcon Positioning Systems, Inc. | Monitoring and control display system and method using multiple displays in a work environment |
WO2017079030A1 (en) * | 2015-11-05 | 2017-05-11 | Topcon Positioning Systems, Inc. | Monitoring and control display system and method using multiple displays in a work environment |
EP3345415A4 (en) * | 2015-11-25 | 2018-07-11 | Samsung Electronics Co., Ltd. | Managing display of information on multiple devices based on context for a user task |
US10078482B2 (en) | 2015-11-25 | 2018-09-18 | Samsung Electronics Co., Ltd. | Managing display of information on multiple devices based on context for a user task |
WO2017090916A1 (en) | 2015-11-25 | 2017-06-01 | Samsung Electronics Co., Ltd. | Managing display of information on multiple devices based on context for a user task |
CN109923568B (en) * | 2016-11-08 | 2023-09-01 | 微软技术许可有限责任公司 | Mobile data insight platform for data analysis |
CN109923568A (en) * | 2016-11-08 | 2019-06-21 | 微软技术许可有限责任公司 | Mobile data for data analysis sees clearly platform |
TWI672057B (en) * | 2017-05-02 | 2019-09-11 | 比利時商巴可公司 | Presentation server, data relay method and method for generating virtual pointer |
US11366629B2 (en) | 2017-05-02 | 2022-06-21 | Barco N.V. | Presentation server, data relay method and method for generating virtual pointer |
WO2018202728A1 (en) * | 2017-05-02 | 2018-11-08 | Barco N.V. | Presentation server, data relay method and method for generating virtual pointer |
US10581939B2 (en) | 2018-01-26 | 2020-03-03 | Citrix Systems, Inc. | Virtual computing system providing local screen sharing from hosted collaboration applications and related methods |
WO2019147303A1 (en) * | 2018-01-26 | 2019-08-01 | Citrix Systems, Inc. | Virtual computing system providing local screen sharing from hosted collaboration applications |
US11190561B2 (en) | 2018-01-26 | 2021-11-30 | Citrix Systems, Inc. | Virtual computing system providing local screen sharing from hosted collaboration applications and related methods |
US11838335B2 (en) | 2018-01-26 | 2023-12-05 | Citrix Systems, Inc. | Virtual computing system providing local screen sharing from hosted collaboration applications and related methods |
US11093198B2 (en) | 2018-03-26 | 2021-08-17 | Samsung Electronics Co., Ltd. | Mobile electronic device and method for forwarding user input to application according to input means |
CN111788819A (en) * | 2018-03-26 | 2020-10-16 | 三星电子株式会社 | Mobile electronic device and method for forwarding user input to an application based on an input device |
CN111135557A (en) * | 2019-12-26 | 2020-05-12 | 三星电子(中国)研发中心 | Interaction method and device for multiple screens |
CN111135557B (en) * | 2019-12-26 | 2024-01-12 | 三星电子(中国)研发中心 | Interaction method and device for multiple screens |
EP4206897A4 (en) * | 2020-09-29 | 2024-03-06 | Huawei Technologies Co., Ltd. | Distributed display method of interface, and electronic device and communication system |
US11995370B2 (en) | 2020-09-29 | 2024-05-28 | Huawei Technologies Co., Ltd. | Distributed interface display method, electronic device, and communication system |
Also Published As
Publication number | Publication date |
---|---|
EP2283421B1 (en) | 2019-08-14 |
EP2283421A2 (en) | 2011-02-16 |
CN102027450B (en) | 2015-05-13 |
CN102027450A (en) | 2011-04-20 |
US20100138780A1 (en) | 2010-06-03 |
WO2009143294A3 (en) | 2010-01-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2283421B1 (en) | Methods and systems for using external display devices with a mobile computing device | |
US9241062B2 (en) | Methods and systems for using external display devices with a mobile computing device | |
US11281360B2 (en) | Display management for native user experiences | |
US9740507B2 (en) | Interacting with remote applications displayed within a virtual desktop of a tablet computing device | |
US11494010B2 (en) | Touch support for remoted applications | |
US10855747B2 (en) | Reverse seamless integration between local and remote computing environments | |
US9467346B2 (en) | Interpreting a gesture-based instruction to selectively display a frame of an application user interface on a mobile computing device | |
US10244039B2 (en) | Shell integration on a mobile device for an application executing remotely on a server | |
US9588637B2 (en) | Transparent user interface integration between local and remote computing environments | |
US10200453B2 (en) | Reverse seamless integration between local and remote computing environments | |
KR101624700B1 (en) | Virtualization client controlling multiple displays under virtual operating system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200980117721.8 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09751522 Country of ref document: EP Kind code of ref document: A2 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2811/KOLNP/2010 Country of ref document: IN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2009751522 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |