US20220179531A1 - Display Management for Native User Experiences - Google Patents
Display Management for Native User Experiences Download PDFInfo
- Publication number
- US20220179531A1 US20220179531A1 US17/678,679 US202217678679A US2022179531A1 US 20220179531 A1 US20220179531 A1 US 20220179531A1 US 202217678679 A US202217678679 A US 202217678679A US 2022179531 A1 US2022179531 A1 US 2022179531A1
- Authority
- US
- United States
- Prior art keywords
- computing device
- application
- native
- display
- control element
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 claims description 30
- 230000004044 response Effects 0.000 claims description 15
- 230000003993 interaction Effects 0.000 abstract description 6
- 238000004091 panning Methods 0.000 abstract description 5
- 238000012545 processing Methods 0.000 description 47
- 239000000872 buffer Substances 0.000 description 16
- 238000010586 diagram Methods 0.000 description 13
- 238000004891 communication Methods 0.000 description 11
- 230000004913 activation Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 238000012790 confirmation Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 4
- 238000009434 installation Methods 0.000 description 4
- 239000008186 active pharmaceutical agent Substances 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- IRLPACMLTUPBCL-KQYNXXCUSA-N 5'-adenylyl sulfate Chemical compound C1=NC=2C(N)=NC=NC=2N1[C@@H]1O[C@H](COP(O)(=O)OS(O)(=O)=O)[C@@H](O)[C@H]1O IRLPACMLTUPBCL-KQYNXXCUSA-N 0.000 description 1
- 241000501754 Astronotus ocellatus Species 0.000 description 1
- 101000666896 Homo sapiens V-type immunoglobulin domain-containing suppressor of T-cell activation Proteins 0.000 description 1
- 241000721662 Juniperus Species 0.000 description 1
- 241000699670 Mus sp. Species 0.000 description 1
- 241001147389 Panthera uncia Species 0.000 description 1
- BQCADISMDOOEFD-UHFFFAOYSA-N Silver Chemical compound [Ag] BQCADISMDOOEFD-UHFFFAOYSA-N 0.000 description 1
- 102100038282 V-type immunoglobulin domain-containing suppressor of T-cell activation Human genes 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- VJYFKVYYMZPMAB-UHFFFAOYSA-N ethoprophos Chemical compound CCCSP(=O)(OCC)SCCC VJYFKVYYMZPMAB-UHFFFAOYSA-N 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000005057 finger movement Effects 0.000 description 1
- IJJVMEJXYNJXOJ-UHFFFAOYSA-N fluquinconazole Chemical compound C=1C=C(Cl)C=C(Cl)C=1N1C(=O)C2=CC(F)=CC=C2N=C1N1C=NC=N1 IJJVMEJXYNJXOJ-UHFFFAOYSA-N 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000008570 general process Effects 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 229910052709 silver Inorganic materials 0.000 description 1
- 239000004332 silver Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000000859 sublimation Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000005641 tunneling Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/14—Display of multiple viewports
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/04—Changes in size, position or resolution of an image
- G09G2340/0464—Positioning
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/14—Solving problems related to the presentation of information to be displayed
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/02—Networking aspects
Definitions
- aspects described herein generally relate to displaying remotely executing applications on devices. For example, aspects relate to providing a native application interaction experience on a device.
- FIG. 4A illustrates an example native mobile device application that permits a user to enter information into an editable field via a virtual keyboard displayed on the screen, and permits a user to select from a drop down box (e.g., combo box) using a virtual selector or picker control.
- the touch screen of the mobile device may be used as the input device.
- an application may be displayed on a device that is not formatted for touch-screen input.
- an application remotely executing on a server may be remotely displayed on a device as shown in FIG. 4B .
- FIG. 4B a virtual keyboard is displayed over the editable control or field such that the user might not be able to see the control while the user is trying to input data into the control, thus making user input difficult.
- the user may be required to manually activate the native control element of the client device, instead of being provided a native control element that is automatically triggered in response to selection of an editable area.
- Other input and viewability issues might also arise from touch compatible input elements being displayed on a remote device.
- a display of a remoted application and native control elements may be managed so that one or more portions of the remoted application display are not obscured or hidden by active native control elements.
- a remote computing device executing the application for a client device may receive a notification when the application or a control element within the application receives focus. Upon receiving the notification, the remote computing device may notify the client device to modify the display of the application as necessary (e.g., to reflect the focus) and/or to determine a type of control element corresponding to the control element within the remoted application receiving focus. Additionally or alternatively, the remote computing device may instruct the client device to activate a native control element corresponding to the determined type of control element.
- the client device may then identify a native control element matching the determined type or category of control element and active the native control element. In some arrangements, the client device may further determine whether the native control element will overlap or otherwise obscure one or more portions of the remoted application display when activated. If so, the client device may modify the local display to avoid obscuring the remoted application display or the area of focus. Modification of the application display may include panning, zooming, scrolling and the like.
- the client device may transmit confirmation and update messages to the remote computing device.
- the confirmation and update messages may include status information for the native control element and notifications of user interaction with the native control element.
- the remote computing device executing the application may determine whether the native control element will obscure the application display and determine a manner in which to modify the local display of the client device.
- a remote computing device executing the application may transmit values of an application control element receiving focus to the client device for populating the native control element.
- the application control element receiving focus is a drop down menu
- the values within the drop down menu may be transmitted to the client device for display in a native picker control element of the client device.
- the remote computing device may generate a virtual control element and remote the control element to the client device similar to remoting an application to which input or interaction is to be registered. For example, if the client device is not capable of showing a local software keyboard, then an in-session keyboard (e.g., remoted by the server) may be displayed instead. The display of the in-session keyboard may be remoted (e.g., graphics transmitted) to the client device for use thereon.
- an in-session keyboard e.g., remoted by the server
- the display of the in-session keyboard may be remoted (e.g., graphics transmitted) to the client device for use thereon.
- FIG. 1A illustrates an example network environment that may be configured to provide remote access to computing devices and application programs executing thereon according to one or more aspects described herein.
- FIG. 1B and FIG. 1C are block diagrams illustrating example computing devices according to one or more aspects described herein.
- FIG. 2A is a block diagram illustrating an example system for displaying a plurality of resources in a user-configurable display layout on an external display device according to one or more aspects described herein.
- FIG. 2B is a block diagram illustrating an example system for mapping a display of one or more resources to one or more display devices according to one or more aspects described herein.
- FIG. 2C illustrates an example interface displaying a plurality of resources in a user-configurable display layout on an external display device, the user-configurable display layout divided into a grid according to one or more aspects described herein.
- FIG. 2D illustrates an example interface displaying a plurality of resources in a user-configurable, dynamic display layout on an external display device according to one or more aspects described herein.
- FIG. 3A is a block diagram illustrating example screen spaces provided by a mobile computing device attached to one or more external display devices according to one or more aspects described herein.
- FIG. 3B is a block diagram illustrating an example mobile computing device providing a plurality of screen spaces according to one or more aspects described herein.
- FIG. 3C is a block diagram illustrating an example logical representation of a plurality of screen spaces managed by a virtual graphics driver according to one or more aspects described herein.
- FIG. 4A is an example known interface displaying an application executing locally on a mobile device.
- FIG. 4B is an example known interface displaying a remoted application on a mobile device.
- FIG. 4C illustrates an example interface displaying a remoted application on a mobile device according to one or more aspects described herein.
- FIG. 5 is a block diagram illustrating an example system for altering the display of a remoted application on a mobile device according to one or more aspects described herein.
- FIG. 6 is a flow diagram illustrating an example method for altering the display of a remoted application on a mobile device according to one or more aspects described herein.
- FIG. 7 is a flowchart illustrating a method for modifying a remoted application display to accommodate the activation of native control elements according to one or more aspects described herein.
- FIG. 8 is a flowchart illustrating a method for managing a display on a device based on the activation of native control elements according to one or more aspects described herein.
- FIG. 9A and FIG. 9B illustrate an example sequence of user interfaces in which a native input element is activated and a display of an application is modified according to one or more aspects described herein.
- FIG. 10A and FIG. 10B illustrate another example sequence of user interfaces in which a native input element is activated and a display of an application is modified according to one or more aspects described herein.
- Applications may be executed remotely by a first computing device for use at a second computing device (e.g., a client device).
- a second computing device e.g., a client device
- the client device may use processing power by the first computing device to execute the functionality and features of the remoted application.
- data and applications may be served at a location that is accessible from multiple remote locations (e.g., through a public and/or private network).
- FIG. 1A illustrates an example computing environment 101 that includes one or more client machines 102 A- 102 N (collectively referred to herein as “client machine(s) 102 ”) that are in communication with one or more servers 106 A- 106 N (generally referred to herein as “server(s) 106 ”). Installed in between the client machine(s) 102 and server(s) 106 is a network.
- the computing environment 101 can include an appliance (e.g., a device or apparatus) installed between the server(s) 106 and client machine(s) 102 .
- This appliance can manage client/server connections, and in some cases can load balance client connections amongst a plurality of backend servers.
- the client machine(s) 102 may, in some examples, be a single client machine 102 or a single group of client machines 102
- server(s) 106 may be a single server 106 or a single group of servers 106
- a single client machine 102 may communicate with more than one server 106
- a single server 106 may communicate with more than one client machine 102
- a single client machine 102 might only communicate with a single server 106 .
- a client machine 102 may, in some examples, be referenced by any one of the following terms: client machine(s) 102 ; client(s); client computer(s); client device(s); client computing device(s); local machine; remote machine; client node(s); endpoint(s); endpoint node(s); or a second machine.
- the server 106 in some examples, may be referenced by any one of the following terms: server(s), local machine; remote machine; server farm(s), host computing device(s), or a first machine(s).
- the client machine 102 may be a virtual machine 102 C.
- the virtual machine 102 C may be any virtual machine, while in some configurations, the virtual machine 102 C may be any virtual machine managed by a hypervisor developed by XenSolutions, Citrix Systems, IBM, VMware, or any other hypervisor. In other examples, the virtual machine 102 C can be managed by any hypervisor, while in still other examples, the virtual machine 102 C can be managed by a hypervisor executing on a server 106 or a hypervisor executing on a client 102 .
- the client machine 102 may, in some examples, execute, operate or otherwise provide an application such as: software; a program; executable instructions; a virtual machine; a hypervisor; a web browser; a web-based client; a client-server application; a thin-client computing client; an ActiveX control; a Java applet; software related to voice over internet protocol (VoIP) communications like a soft IP telephone; an application for streaming video and/or audio; an application for facilitating real-time-data communications; a HTTP client; a FTP client; an Oscar client; a Telnet client; or any other set of executable instructions.
- an application such as: software; a program; executable instructions; a virtual machine; a hypervisor; a web browser; a web-based client; a client-server application; a thin-client computing client; an ActiveX control; a Java applet; software related to voice over internet protocol (VoIP) communications like a soft IP telephone; an application for streaming video and/or audio; an application for facilitating real-
- remoting an application may include a client device 102 receiving and displaying application output generated by an application remotely executing on a server 106 or other remotely located machine or computing device.
- the client device 102 may display the application output in an application window, a browser, or other output window.
- the application is a desktop, while in other examples the application is an application that generates a desktop.
- a desktop may include a graphical shell providing a user interface for an instance of an operating system in which local and/or remote applications can be integrated.
- Applications as used herein, are programs that execute after an instance of an operating system (and, optionally, also the desktop) has been loaded.
- Each instance of the operating system may be physical (e.g., one operating system per device) or virtual (e.g., many instances of an OS running on a single device).
- Each application may be executed on a local device, or executed on a remotely located device (e.g., remoted). Applications may be remoted in multiple ways. In one example, applications may be remoted in a seamless manner in which windows are created on the client device so as to make the application display seem as if it were running locally on the client device and the desktop thereof. In another example, a remoted application may be provided in a windowed mode where a desktop is remoted to the client device and the application is displayed as an application executing in the remoted desktop. Various other remoting methods and techniques may also be implemented or used.
- the server 106 may execute a remote presentation client or other client or program that uses a thin-client or remote-display protocol to capture display output generated by an application executing on a server 106 and to subsequently transmit the application display output to a remote client 102 .
- the thin-client or remote-display protocol can be any one of the following protocols (or other suitable protocol): the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla.; or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Wash.
- ICA Independent Computing Architecture
- RDP Remote Desktop Protocol
- the computing environment can include more than one server 106 A- 106 N such that the servers 106 A- 106 N are logically and/or physically grouped together into a server farm 106 .
- the server farm 106 can include servers 106 that are geographically dispersed and logically grouped together in a server farm 106 , or, in another example, servers 106 that are located proximate to each other and logically grouped together in a server farm 106 .
- Geographically dispersed servers 106 A- 106 N within a server farm 106 can, in some examples, communicate using a WAN, MAN, or LAN, where different geographic regions can be characterized as: different continents; different regions of a continent; different countries; different states; different cities; different zip codes; different neighborhoods; different campuses; different rooms; or any combination of the preceding geographical locations.
- the server farm 106 may be administered as a single entity, while in other examples, the server farm 106 can include multiple server farms 106 .
- a server farm 106 may include servers 106 that execute a substantially similar type of operating system platform (e.g., WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Wash., UNIX, LINUX, or SNOW LEOPARD.)
- the server farm 106 can include a first group of servers 106 that execute a first type of operating system platform, and a second group of servers 106 that execute a second type of operating system platform.
- the server farm 106 in other examples, can include servers 106 that execute different types of operating system platforms.
- the server 106 may be of any server type.
- the server 106 can be any of the following server types: a file server; an application server; a web server; a proxy server; an appliance; a network appliance; a gateway; an application gateway; a gateway server; a virtualization server; a deployment server; a SSL VPN server; a firewall; a web server; an application server or as a master application server; a server 106 executing an active directory; or a server 106 executing an application acceleration program that provides firewall functionality, application functionality, or load balancing functionality.
- a server 106 may be a RADIUS server that includes a remote authentication dial-in user service.
- the server 106 may be an appliance manufactured by any one of the following manufacturers: the Citrix Application Networking Group; Silver Peak Systems, Inc; Riverbed Technology, Inc.; F5 Networks, Inc.; or Juniper Networks, Inc.
- Some examples include a first server 106 A that receives requests from a client machine 102 , forwards the request to a second server 106 B, and responds to the request generated by the client machine 102 with a response from the second server 106 B.
- the first server 106 A may acquire an enumeration of applications available to the client machine 102 as well as address information associated with an application server 106 hosting an application identified within the enumeration of applications.
- the first server 106 A may then present a response to the client's request using a web interface, and communicate directly with the client 102 to provide the client 102 with access to an identified application.
- the server 106 may, in some examples, execute any one of the following applications: a thin-client application using a thin-client protocol to transmit application display data to a client; a remote display presentation application; any portion of the CITRIX ACCESS SUITE by Citrix Systems, Inc. like the METAFRAME, CITRIX PRESENTATION SERVER, CITRIX XENAPP or CITRIX XENDESKTOP server; MICROSOFT WINDOWS Terminal Services manufactured by the Microsoft Corporation; or an ICA client, developed by Citrix Systems, Inc.
- server 106 that is an application server such as: an email server that provides email services such as MICROSOFT EXCHANGE manufactured by the Microsoft Corporation; a web or Internet server; a desktop sharing server; a collaboration server; or any other type of application server.
- server 106 that executes any one of the following types of hosted servers applications: GOTOMEETING provided by Citrix Online Division, Inc.; WEBEX provided by WebEx, Inc. of Santa Clara, Calif.; or Microsoft Office LIVE MEETING provided by Microsoft Corporation.
- Client machines 102 may, in some examples, be a client node that seeks access to resources provided by a server 106 .
- the server 106 may provide clients 102 or client nodes with access to hosted resources.
- the server 106 functions as a master node such that it communicates with one or more clients 102 or servers 106 .
- the master node may identify and provide address information associated with a server 106 hosting a requested application, to one or more clients 102 or servers 106 .
- the master node may be a server farm 106 , a client 102 , a cluster of client nodes 102 , or an appliance.
- One or more clients 102 and/or one or more servers 106 may transmit data over a network 104 installed between machines and appliances within the computing environment 101 .
- the network 104 may comprise one or more sub-networks, and may be installed between any combination of the clients 102 , servers 106 , computing machines and appliances included within the computing environment 101 .
- the network 104 may be: a local-area network (LAN); a metropolitan area network (MAN); a wide area network (WAN); a primary network 104 comprised of multiple sub-networks 104 located between the client machines 102 and the servers 106 ; a primary public network 104 with a private sub-network 104 ; a primary private network 104 with a public sub-network 104 ; or a primary private network 104 with a private sub-network 104 .
- LAN local-area network
- MAN metropolitan area network
- WAN wide area network
- a primary network 104 comprised of multiple sub-networks 104 located between the client machines 102 and the servers 106 ; a primary public network 104 with a private sub-network 104 ; a primary private network 104 with a public sub-network 104 ; or a primary private network 104 with a private sub-network 104 .
- Still further examples include a network 104 that may be any of the following network types: a point to point network; a broadcast network; a telecommunications network; a data communication network; a computer network; an ATM (Asynchronous Transfer Mode) network; a SONET (Synchronous Optical Network) network; a SDH (Synchronous Digital Hierarchy) network; a wireless network; a wireline network; or a network 104 that includes a wireless link where the wireless link may be an infrared channel or satellite band.
- a network 104 may be any of the following network types: a point to point network; a broadcast network; a telecommunications network; a data communication network; a computer network; an ATM (Asynchronous Transfer Mode) network; a SONET (Synchronous Optical Network) network; a SDH (Synchronous Digital Hierarchy) network; a wireless network; a wireline network; or a network 104 that includes a wireless link where the wireless link may be an
- the network topology of the network 104 may differ within different examples, possible network topologies include: a bus network topology; a star network topology; a ring network topology; a repeater-based network topology; or a tiered-star network topology. Additional examples may include a network 104 of mobile telephone networks that use a protocol to communicate among mobile devices, where the protocol may be any one of the following: AMPS; TDMA; CDMA; GSM; GPRS UMTS; or any other protocol able to transmit data among mobile devices.
- FIG. 1B illustrates an example computing device 100 .
- the client machine 102 and server 106 illustrated in FIG. 1A may be deployed as and/or executed on any example of computing device 100 illustrated and described herein or any other computing device.
- Included within the computing device 100 is a system bus 150 that communicates with the following components: a central processing unit 121 ; a main memory 122 ; storage memory 128 ; an input/output (I/O) controller 123 ; display devices 124 A- 124 N; an installation device 116 ; and a network interface 118 .
- the storage memory 128 includes: an operating system, software routines, and a client agent 120 .
- the I/O controller 123 in some examples, is further connected to a key board 126 , and a pointing device 127 . Other examples may include an I/O controller 123 connected to one or more of input/output devices 130 A- 130 N.
- FIG. 1C illustrates an example computing device 100 , where the client machine 102 and server 106 illustrated in FIG. 1A may be deployed as and/or executed on any example of the computing device 100 illustrated and described herein or any other computing device.
- a system bus 150 that communicates with the following components: a bridge 170 , and a first I/O device 130 A.
- the bridge 170 is in further communication with the main central processing unit 121 , where the central processing unit 121 may further communicate with a second I/O device 130 B, a main memory 122 , and a cache memory 140 .
- I/O ports I/O ports, a memory port 103 , and a main processor.
- Examples of the computing machine 100 may include a central processing unit 121 characterized by any one of the following component configurations: logic circuits that respond to and process instructions fetched from the main memory unit 122 ; a microprocessor unit, such as: those manufactured by Intel Corporation; those manufactured by Motorola Corporation; those manufactured by Transmeta Corporation of Santa Clara, Calif.; the RS/6000 processor such as those manufactured by International Business Machines; a processor such as those manufactured by Advanced Micro Devices; or any other combination of logic circuits. Still other examples of the central processing unit 122 may include any combination of the following: a microprocessor, a microcontroller, a central processing unit with a single processing core, a central processing unit with two processing cores, or a central processing unit with more than one processing core.
- FIG. 1C illustrates a computing device 100 that includes a single central processing unit 121
- the computing device 100 may include one or more processing units 121 .
- the computing device 100 may store and execute firmware or other executable instructions that, when executed, direct the one or more processing units 121 to simultaneously execute instructions or to simultaneously execute instructions on a single piece of data.
- the computing device 100 may store and execute firmware or other executable instructions that, when executed, direct the one or more processing units to each execute a section of a group of instructions.
- each processing unit 121 may be instructed to execute a portion of a program or a particular module within a program.
- the processing unit 121 may include one or more processing cores.
- the processing unit 121 may have two cores, four cores, eight cores, etc.
- the processing unit 121 may comprise one or more parallel processing cores.
- the processing cores of the processing unit 121 may in some examples access available memory as a global address space, or in other examples, memory within the computing device 100 may be segmented and assigned to a particular core within the processing unit 121 .
- the one or more processing cores or processors in the computing device 100 may each access local memory.
- memory within the computing device 100 may be shared amongst one or more processors or processing cores, while other memory may be accessed by particular processors or subsets of processors.
- the multiple processing units may be included in a single integrated circuit (IC). These multiple processors, in some examples, may be linked together by an internal high speed bus, which may be referred to as an element interconnect bus.
- the processors may execute a single instruction simultaneously on multiple pieces of data (SIMD), or in other examples may execute multiple instructions simultaneously on multiple pieces of data (MIMD).
- SIMD single instruction simultaneously on multiple pieces of data
- MIMD multiple instructions simultaneously on multiple pieces of data
- the computing device 100 may include any number of SIMD and MIMD processors.
- the computing device 100 may include a graphics processor or a graphics processing unit (Not Shown).
- the graphics processing unit may include any combination of software and hardware, and may further input graphics data and graphics instructions, render a graphic from the inputted data and instructions, and output the rendered graphic.
- the graphics processing unit may be included within the processing unit 121 .
- the computing device 100 may include one or more processing units 121 , where at least one processing unit 121 is dedicated to processing and rendering graphics.
- computing machine 100 may include a central processing unit 121 that communicates with cache memory 140 via a secondary bus also known as a backside bus, while another example of the computing machine 100 includes a central processing unit 121 that communicates with cache memory via the system bus 150 .
- the local system bus 150 may, in some examples, also be used by the central processing unit to communicate with more than one type of I/O device 130 A- 130 N.
- the local system bus 150 may be any one of the following types of buses: a VESA VL bus; an ISA bus; an EISA bus; a MicroChannel Architecture (MCA) bus; a PCI bus; a PCI-X bus; a PCI-Express bus; or a NuBus.
- MCA MicroChannel Architecture
- computing machine 100 may include an I/O device 130 A- 130 N that is a video display 124 that communicates with the central processing unit 121 .
- Still other versions of the computing machine 100 include a processor 121 connected to an I/O device 130 A- 130 N via any one of the following connections: HyperTransport, Rapid I/O, or InfiniBand.
- Further examples of the computing machine 100 include a processor 121 that communicates with one I/O device 130 A using a local interconnect bus and a second I/O device 130 B using a direct connection.
- the computing device 100 includes a main memory unit 122 and cache memory 140 .
- the cache memory 140 may be any memory type, and in some examples may be any one of the following types of memory: SRAM; BSRAM; or EDRAM.
- Other examples include cache memory 140 and a main memory unit 122 that may be any one of the following types of memory: Static random access memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM); Dynamic random access memory (DRAM); Fast Page Mode DRAM (FPM DRAM); Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM); Extended Data Output DRAM (EDO DRAM); Burst Extended Data Output DRAM (BEDO DRAM); Enhanced DRAM (EDRAM); synchronous DRAM (SDRAM); JEDEC SRAM; PC100 SDRAM; Double Data Rate SDRAM (DDR SDRAM); Enhanced SDRAM (ESDRAM); SyncLink DRAM (SLDRAM); Direct Rambus DRAM (DRDRAM); Ferroelectric RAM (FRAM); or any combination
- computing device 100 may provide support for any one of the following installation devices 116 : a CD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, tape drives of various formats, USB device, a bootable medium, a bootable CD, a bootable CD for GNU/Linux distribution such as KNOPPIX®, a hard-drive or any other device suitable for installing applications or software.
- Applications may, in some examples, include a client agent 120 , or any portion of a client agent 120 .
- the computing device 100 may further include a storage device 128 that may be either one or more hard disk drives, or one or more redundant arrays of independent disks; where the storage device is configured to store an operating system, software, programs applications, or at least a portion of the client agent 120 .
- Computing device 100 may also include an installation device 116 that is used as the storage device 128 .
- the computing device 100 may further include a network interface 118 to interface to a Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or some combination of any or all of the above.
- LAN Local Area Network
- WAN Wide Area Network
- the Internet may further include a network interface 118 to interface to a Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over
- Connections may also be established using a variety of communication protocols (e.g., TCP/IP, IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), RS232, RS485, IEEE 802.11, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, CDMA, GSM, WiMax and direct asynchronous connections).
- One version of the computing device 100 includes a network interface 118 able to communicate with additional computing devices 100 ′ via any type and/or form of gateway or tunneling protocol such as Secure Socket Layer (SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems, Inc.
- SSL Secure Socket Layer
- TLS Transport Layer Security
- Versions of the network interface 118 may comprise any one of: a built-in network adapter; a network interface card; a PCMCIA network card; a card bus network adapter; a wireless network adapter; a USB network adapter; a modem; or any other device suitable for interfacing the computing device 100 to a network capable of communicating and performing the methods and systems described herein.
- computing device 100 may include any one of the following I/O devices 130 A- 130 N: a keyboard 126 ; a pointing device 127 ; mice; trackpads; an optical pen; trackballs; microphones; drawing tablets; video displays; speakers; inkjet printers; laser printers; and dye-sublimation printers; or any other input/output device able to perform the methods and systems described herein.
- An I/O controller 123 may in some examples connect to multiple I/O devices 103 A- 130 N to control the one or more I/O devices.
- I/O devices 130 A- 130 N may be configured to provide storage or an installation medium 116 , while others may provide a universal serial bus (USB) interface for receiving USB storage devices such as the USB Flash Drive line of devices manufactured by Twintech Industry, Inc.
- computing device 100 may include I/O device 130 that may be a bridge between the system bus 150 and an external communication bus, such as: a USB bus; an Apple Desktop Bus; an RS-232 serial connection; a SCSI bus; a FireWire bus; a FireWire 800 bus; an Ethernet bus; an AppleTalk bus; a Gigabit Ethernet bus; an Asynchronous Transfer Mode bus; a HIPPI bus; a Super HIPPI bus; a SerialPlus bus; a SCI/LAMP bus; a FibreChannel bus; or a Serial Attached small computer system interface bus.
- USB universal serial bus
- the computing machine 100 may connect to multiple display devices 124 A- 124 N, in other examples the computing device 100 may connect to a single display device 124 , while in still other examples the computing device 100 connects to display devices 124 A- 124 N that are the same type or form of display, or to display devices that are different types or forms.
- Examples of the display devices 124 A- 124 N may be supported and enabled by the following: one or multiple I/O devices 130 A- 130 N; the I/O controller 123 ; a combination of I/O device(s) 130 A- 130 N and the I/O controller 123 ; any combination of hardware and software able to support a display device 124 A- 124 N; any type and/or form of video adapter, video card, driver, and/or library to interface, communicate, connect or otherwise use the display devices 124 A- 124 N.
- the computing device 100 may in some examples be configured to use one or multiple display devices 124 A- 124 N, these configurations include: having multiple connectors to interface to multiple display devices 124 A- 124 N; having multiple video adapters, with each video adapter connected to one or more of the display devices 124 A- 124 N; having an operating system configured to support multiple displays 124 A- 124 N; using circuits and software included within the computing device 100 to connect to and use multiple display devices 124 A- 124 N; and executing software on the main computing device 100 and multiple secondary computing devices to enable the main computing device 100 to use a secondary computing device's display as a display device 124 A- 124 N for the main computing device 100 . Still other examples of the computing device 100 may include multiple display devices 124 A- 124 N provided by multiple secondary computing devices and connected to the main computing device 100 via a network.
- the computing machine 100 may execute any operating system including: versions of the MICROSOFT WINDOWS operating systems such as WINDOWS 3.x; WINDOWS 95; WINDOWS 98; WINDOWS 2000; WINDOWS NT 3.51; WINDOWS NT 4.0; WINDOWS CE; WINDOWS XP; and WINDOWS VISTA; the different releases of the Unix and Linux operating systems; any version of the MAC OS manufactured by Apple Computer; OS/2, manufactured by International Business Machines; any embedded operating system; any real-time operating system; any open source operating system; any proprietary operating system; any operating systems for mobile computing devices; or any other operating system.
- the computing machine 100 may execute multiple operating systems.
- the computing machine 100 may execute PARALLELS or another virtualization platform that may execute or manage a virtual machine executing a first operating system, while the computing machine 100 executes a second operating system different from the first operating system.
- the computing machine 100 may be embodied in any one of the following computing devices: a computing workstation; a desktop computer; a laptop or notebook computer; a server; a handheld computer (e.g., a tablet computer such as the iPad and iPad 2 manufactured by Apple Computer); a mobile telephone; a portable telecommunication device; a media playing device; a gaming system; a mobile computing device; a netbook; a device of the IPOD family of devices manufactured by Apple Computer; any one of the PLAYSTATION family of devices manufactured by the Sony Corporation; any one of the Nintendo family of devices manufactured by Nintendo Co; any one of the XBOX family of devices manufactured by the Microsoft Corporation; or any other type and/or form of computing, telecommunications or media device that is capable of communication and that has sufficient processor power and memory capacity to perform the methods and systems described herein.
- a computing workstation e.g., a desktop computer such as the iPad and iPad 2 manufactured by Apple Computer
- a handheld computer e.g., a tablet computer such as the iPad and
- the computing machine 100 may be a mobile device such as any one of the following mobile devices: a JAVA-enabled cellular telephone or personal digital assistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the im1100, all of which are manufactured by Motorola Corp; the 6035 or the 7135, manufactured by Kyocera; the i300 or i330, manufactured by Samsung Electronics Co., Ltd; the TREO 180, 270, 600, 650, 680, 700p, 700w, or 750 smart phone manufactured by Palm, Inc; any computing device that has different processors, operating systems, and input devices consistent with the device; or any other mobile computing device capable of performing the methods and systems described herein.
- PDA personal digital assistant
- the computing device 100 may be any one of the following mobile computing devices: any one series of Blackberry, or other handheld device manufactured by Research In Motion Limited; the iPhone manufactured by Apple Computer; Palm Pre; a Pocket PC; a Pocket PC Phone; or any other handheld mobile device.
- the computing device 100 may have different processors, operating systems, and input devices consistent with the device.
- the computing device 100 is a TREO 180, 270, 600, 650, 680, 700p, 700w, or 750 smart phone manufactured by Palm, Inc.
- the TREO smart phone is operated under the control of the PalmOS operating system and includes a stylus input device as well as a five-way navigator device.
- the computing device 100 may be a mobile device, such as a JAVA-enabled cellular telephone or personal digital assistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the im1100, all of which are manufactured by Motorola Corp. of Schaumburg, Ill., the 6035 or the 7135, manufactured by Kyocera of Kyoto, Japan, or the i300 or i330, manufactured by Samsung Electronics Co., Ltd., of Seoul, Korea.
- the computing device 100 is a mobile device manufactured by Nokia of Finland, or by Sony Ericsson Mobile Communications AB of Lund, Sweden.
- the computing device 100 may be a Blackberry handheld or smart phone, such as the devices manufactured by Research In Motion Limited, including the Blackberry 7100 series, 8700 series, 7700 series, 7200 series, the Blackberry 7520, or the Blackberry Pearl 8100.
- the computing device 100 is a smart phone, Pocket PC, Pocket PC Phone, or other handheld mobile device supporting Microsoft Windows Mobile Software.
- the computing device 100 may be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile telephone, any other computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.
- the computing device 100 may be a digital audio player.
- the computing device 100 is a digital audio player such as the Apple IPOD, IPOD Touch, IPOD NANO, and IPOD SHUFFLE lines of devices, manufactured by Apple Computer of Cupertino, Calif.
- the digital audio player may function as both a portable media player and as a mass storage device.
- the computing device 100 is a digital audio player such as the DigitalAudioPlayer Select MP3 players, manufactured by Samsung Electronics America, of Ridgefield Park, N.J., or the Motorola m500 or m25 Digital Audio Players, manufactured by Motorola Inc. of Schaumburg, Ill.
- the computing device 100 is a portable media player, such as the Zen Vision W, the Zen Vision series, the Zen Portable Media Center devices, or the Digital MP3 line of MP3 players, manufactured by Creative Technologies Ltd.
- the computing device 100 is a portable media player or digital audio player supporting file formats including, but not limited to, MP3, WAV, M4A/AAC, WMA Protected AAC, AIFF, Audible audiobook, Apple Lossless audio file formats and .mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats.
- the computing device 100 may, alternatively or additionally, include a combination of devices, such as a mobile phone combined with a digital audio player or portable media player.
- the computing device 100 is a Motorola RAZR or Motorola ROKR line of combination digital audio players and mobile phones.
- the computing device 100 is an iPhone smartphone, manufactured by Apple Computer of Cupertino, Calif.
- FIG. 2A the block diagram illustrates an example system for displaying a plurality of resources on one or more display devices.
- the system includes a mobile computing device 102 that may communicate with one or more external display devices 202 a - n .
- the example in FIG. 2A shows a mobile computing device 102 with a native display 201 , although devices without native displays may also be used.
- the mobile computing device 102 executes a plurality of resources 204 a - n (collectively, 204 ).
- the window management system 206 and virtual graphics driver 208 manage the locations and sizes of the display of output data associated with each of the plurality of resources in a user-configurable display layout.
- the mobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to an external display device 202 . In some of these examples, the mobile computing device 102 transmits the output data upon establishing a connection with the external display device 202 . Alternatively or additionally, the mobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to the device's native display 201 . For example, the mobile computing device 102 might only transmit the output data to the device's native display or only transmit the output data to one or more of the external display devices 202 a - n .
- the mobile computing device 102 transmits the output data associated with certain of the plurality of resources 204 to the native display 201 and transmits the output data associated with other ones of the plurality of resources 204 to the external display devices 202 a - n.
- Mobile computing device 102 may be configured to execute a plurality of resources 204 .
- the mobile computing device 102 is a client 102 as described above in connection with FIGS. 1A-1C .
- the mobile computing device 102 displays the output data associated with a resource 204 a in a plurality of resources 204 a - n executed by the mobile computing device 102 .
- the mobile computing device 102 displays the output data associated with each of the plurality of resources 204 .
- a resource in the plurality of resources 204 may include, without limitation, a data file, an executable file, configuration files, an application, a desktop environment (which may itself include a plurality of applications for execution by the user), a computing environment image (such as a virtual machine image), and/or operating system software or other applications needed to execute a computing environment image.
- mobile computing device 102 includes a window management service 206 allowing an external display device 202 to display the output data associated with each of a plurality of resources 204 executed on the mobile computing device 102 .
- the window management service 206 may also allow multiple resources running on the mobile computing device 102 to be viewed on an external display device 202 or the native display 201 at substantially the same time, as opposed to allowing the output of one resource 204 to be viewed exclusively on the native display 201 or external display device 202 .
- the window management service 206 in conjunction with a virtual graphics driver 208 , manages the display layout of the windows displayed on the external display device 202 and the native display 201 .
- the virtual graphics driver 208 is a driver-level component that manages a virtual screen frame buffer storing output data that will be displayed by the native display 201 on the mobile computing device 102 or an external display device 202 .
- the window management service 206 in conjunction with the virtual graphics driver 208 , may further manage the boundaries and size of a screen space used to display output data and on which display device the output data is displayed.
- an external display device 202 receives output data associated with each of the plurality of resources 204 and displays the output data in a user-configurable display layout.
- the external display device 202 includes a dock to which the mobile computing device 102 connects.
- the external display device 202 includes a receiver for communicating with the mobile computing device 102 wirelessly, for example, via BLUETOOTH, Wi-Fi or other networking protocols, as described above in connection with FIGS. 1A-1C .
- the external display device 202 may be a display device 124 as described above in connection with FIGS. 1B and 1C .
- the block diagram illustrates an example system for mapping the display of one or more resources 204 of the mobile computing device 102 on one or more display devices 201 and/or 202 .
- the window management service 206 manages a virtual screen space 210 .
- the virtual screen space 210 may map to the native display 201 and one or more external display devices 202 .
- the window management service 206 may position output data associated with the user interfaces of one or more resources 204 on the virtual screen space 210 to specify where each user interface will be displayed.
- the window management service 206 positions the output data according to a user preference.
- the window management service 206 positions the output data according to a policy.
- the window management service 206 positions the output data based on the resource 204 associated with the output data.
- the window management service 206 communicates with the virtual graphics driver 208 to transmit output data associated with user interfaces of resources 204 to the native display 201 and one or more external display devices 202 .
- the window management service 206 may transmit output data and associated coordinates from the virtual screen space 210 to the virtual graphics driver 208 .
- the virtual graphics driver 208 stores the output data in a virtual screen frame buffer.
- the virtual graphics driver 208 transmits the entries in the virtual screen frame buffer to the native display 201 and external display devices 202 .
- the virtual graphics driver 208 transmits an entry in the virtual screen frame buffer to a native display 201 or an external display device 202 based on the position of the entry in the frame buffer.
- the user interface illustrates an example user-configurable display layout in which the external display device 202 displays the output data of the resources 204 .
- the output data of the resources 204 is displayed in a grid display layout.
- the grid display layout may include one or more cells in an arrangement.
- a cell may display output data associated with a resource. In some examples, more than one cell displays output data associated with the same resource.
- the cells are uniformly sized, whereas in other examples, the cells have different sizes.
- the cells may be arranged in any configuration. In some examples, the cells may be arranged in rows, columns, or both.
- a cell may have a descriptor associated with the cell's position in the grid. The descriptor may indicate the position of a cell within a row. In the example depicted in FIG.
- the cell for resource 204 a may have the descriptor “1-1,” the cell for resource 204 b may have the descriptor “1-2,” the cell for resource 204 c may have the descriptor “1-3,” the cell for resource 204 d may have the descriptor “2-1,” and the cell for resource 204 d may have the descriptor “2-2.”
- the cells may be numbered, e.g., “Cell 1,” “Cell 2,” etc. However, any system of choosing descriptors known to those of ordinary skill in the art may be used.
- the window management service 206 configures a grid display layout according to the resources 204 being displayed on the native display 201 or the external display device 202 . In some examples, the service 206 configures a grid display layout according to the number of resources 204 being displayed. In other examples, the service 206 configures a grid display layout according to the size or amount of content in the user interfaces of the resources 204 . For example, if an external display device 202 will display four resources with comparable amounts of content, the window management service 206 may configure a grid display layout with four uniform cells.
- the window management service 206 may configure a grid display layout with three uniform cells in a first row and a single cell in a second row.
- the single cell in the second row may be three times as wide as the cells in the first row.
- the window management service 206 may configure a grid display layout to reserve a cell for displaying information about the resources being displayed, such as a menu of the resources.
- the window management service 206 may configure a grid display layout to reserve a cell for allowing a user to configure the grid display layout.
- a user interface illustrates an example user-configurable display layout providing a dynamic display layout in which the external display device 202 displays the output data associated with the plurality of resources 204 .
- windows on the external display device 202 that display output data for resources 204 may be dynamically positioned and sized.
- the window management service 206 may position a user interface for a resource at a default position and with a default size chosen according to a policy, the resource 204 , or any other method.
- the window management service 206 may order overlapping user interfaces such that higher-order user interfaces obscure lower-order user interfaces.
- the window management service 206 may transmit output data to the virtual graphics driver 208 reflecting the obfuscation.
- the user may re-position or re-size a window by, for example, clicking and dragging the window or a window edge.
- the virtual graphics driver 208 may detect the user's change to the window, and transmit information about the user's change to the window management service 206 .
- the window management service 206 may process the change and transmit updated output data to the virtual graphics driver 208 .
- the user moves the user interface for a resource 204 to any location on a native display 201 or external display device 202 .
- the user moves the user interface for a resource 204 to a different display device.
- the updated output data indicates that one user interface's size has been increased or location has been adjusted to obscure another user interface.
- the updated output data indicates that one user interface's size has been decreased or location has been adjusted such that more of another user interface shall be visible.
- FIG. 3A a block diagram illustrates example virtual screen spaces 210 provided by a mobile computing device attached to one or more external display devices.
- the mobile computing device 102 includes a virtual graphics driver 208 and a virtual screen 210 .
- the virtual screen 210 includes a plurality of virtual screen spaces 310 and 312 a - n .
- Virtual screen space 310 may be a native display screen space for the native display 201 on the mobile computing device 102 .
- the other virtual screen spaces 312 a - n may be extended screen spaces that correspond to the displays of external display devices 202 .
- the window management service 206 and virtual graphics driver 208 manage the virtual screen 210 .
- the virtual graphics driver 208 uses a virtual screen frame buffer to manage the mobile computing device's native display 201 and change the native display's 201 screen resolution. In another example, the virtual graphics driver 208 uses a virtual screen frame buffer to manage an extended screen space 312 and to change a resolution of the extended screen 312 .
- the virtual graphics driver 208 allocates and manages a plurality of virtual screen spaces 310 , 312 a - n and virtual screen frame buffers.
- each virtual screen space and virtual screen frame buffer has a resolution independent of the other screen spaces and frame buffers.
- output data associated with each of the plurality of resources 204 may reside within any of the virtual screen spaces 310 , 312 a - n .
- each of the extended screen spaces 312 a - n is associated with at least one external display device 202 , depending on the capabilities of the device.
- the window management service 206 and the virtual graphics driver 208 allocate and manage the display, on a plurality of external display devices 202 , of output data associated with a plurality of resources. For example, output data associated with a resource 204 a displays on a mobile computing device 102 , output data associated with a resource 204 b displays on one external display device 202 a , and output data associated with a resource 204 c display on another external display device 202 b .
- the window management device 206 identifies one of the external display devices 202 for displaying output data generated by a resource 204 a based upon a type of the resource 204 a . For example, the window management service 206 may determine that a type of resource rendering a video may display on a television screen, while a type of resource rendering a word processing application may render on a display of a laptop computer.
- FIG. 3B a block diagram illustrates an example mobile computing device 102 providing a virtual screen 210 with virtual screen spaces 310 , 312 a - n of varying resolutions.
- the virtual screen 210 includes a native display screen space 310 corresponding to the native display 201 of the mobile computing device 102 with a resolution of 320 pixels ⁇ 240 pixels.
- the virtual screen 210 also includes an extended screen 312 a corresponding to the display of an external display device 202 with a resolution of 1024 pixels ⁇ 768 pixels, an extended screen 312 n -1 corresponding to the display of an external display device 202 with a resolution of 800 pixels ⁇ 600 pixels, and an extended screen 312 n corresponding to the display of an external display device 202 with a resolution of 640 pixels ⁇ 480 pixels.
- the virtual screen 210 may include a native display screen space 310 and any number of extended screens 312 of any resolution.
- the entire virtual screen space 210 may be mapped into a single virtual screen frame buffer, although examples that map into multiple buffers may be used.
- the block diagram illustrates an example logical representation of a plurality of virtual screen spaces managed by a virtual graphics driver.
- the virtual graphics driver 208 manages multiple virtual screen spaces with different resolutions in a virtual screen frame buffer.
- the native display 201 of the mobile computing device is the primary display and the external display device 202 , corresponding to the extended screen 312 a , is a secondary display.
- output data associated with resources 204 on the native display screen space 310 will be displayed on the native display 201 and output data associated with resources 204 on the extended screen space 312 a will be displayed on the external display device 202 associated with the extended screen spaces 312 a .
- all output data associated with resources 204 may be displayed on the native display 201 .
- all output data associated with resources 204 may be displayed on an external display device such as external display device 202 .
- FIG. 5 illustrates an example system in which applications may be remotely executed at a server while providing display and user interaction through a client device such as mobile device 102 .
- a user may view and interact with an application remoted from (e.g., remotely executed by the server (e.g., remote computer 501 )) at a client device (e.g., mobile device 102 ) without requiring the application and/or data to be installed or stored on the client device.
- the system of FIG. 5 may include a mobile computing device 102 (e.g., similar to that shown in FIG. 2B ) configured to execute and/or store one or more resources 204 such as applications executing on the mobile computing device 102 .
- At least one resource may be an operating system executing on the mobile device 102 and one or more other resources may execute within the context of the operating system.
- the mobile device 102 may further execute a windows management service 206 that may communicate with other applications executing on the mobile device 102 and, in some examples, may communicate with a virtual graphics driver 208 and the resources or applications 204 executing on the mobile device 102 .
- the mobile computing device 102 may further execute a virtual graphics driver 208 , and may store a buffer for generating a virtual screen space 210 that may include a buffer for a native display screen space 210 a .
- the mobile device 102 may communicate with a native display 201 of the mobile device 102 on which application output generated by a resource 204 may be displayed.
- the mobile device 102 may further communicate with a remote computer 501 that may execute one or more resources 505 A- 505 N (generally referred to as remote resources 505 ) which may in some examples be remote applications 505 .
- the remote computer 501 may further execute a remoting client 510 .
- the remote applications 505 may be executed by the remote computer for use at the mobile device 102 in one or more arrangements as discussed in further detail below. When one or more remote applications 505 are executed by the remote computer for use at the mobile device 102 , those applications may be considered remoted applications.
- resources 204 executing on the mobile device 102 and/or resources 505 A- 505 N may be applications configured to accept data inputted by a touch screen.
- the applications may be modified versions of applications that typically do not receive data from a touch screen.
- These modified applications 204 may receive data inputted by a user via a touch screen of the mobile device 102 , and may be modified to accommodate typical touch screen input functions like a virtual keyboard and a virtual scroll menu.
- a modified version of SAFARI a web browser published by APPLE, may be modified to pan up when a user selects or gives focus to an edit control such as a text box. SAFARI pans the application upwards to accommodate the virtual keyboard displayed over the browser and so that a user may view the text box while typing on the virtual keyboard.
- Touch input may include a variety of hand or finger movements and actions including touching a point on the screen, swiping movements, flicking movements, multi-finger or multi-touch actions (e.g., pinching, expansion) and gestures and the like.
- the touch input may trigger different functionalities depending on the type of application receiving the input. For example, some applications may receive a flicking action as closing the application while other application may interpret a flicking input as moving the application window in a certain direction.
- Each applications may define its own interpretation of different types of touch input or may use a standard definition provided by an underlying operating system.
- the mobile device 102 may communicate with remote computer 501 in a variety of ways.
- the remote computer 501 may be a server, a client or any other computing machine.
- the remote computer 501 may be a remote application server that executes one or more applications for mobile device 102 .
- the mobile device 102 may communicate with the remote computer 501 over one or more virtual channels.
- virtual channels may be established over a network and may be referred to as a control virtual channel.
- the virtual channel may be a seamless virtual channel.
- a control virtual channel may be used to remote control commands and other miscellaneous commands (e.g., for an application) while the seamless virtual channel may be used to remote application windows, a taskbar, a systray, etc.
- each virtual channel may remote different functions and content.
- the virtual channel may be established by a window management service 206 executing on the mobile device 102 , or the virtual channel may be established by both the window management service 206 and a remoting client 510 executing on remote computer 501 .
- the virtual channel may in some examples facilitate communication sent using the ICA protocol.
- the remote computer 501 may execute a remoting client 510 .
- the remoting client 510 may be a control virtual channel or a seamless virtual channel and may be a remoting application that corresponds to the virtual channel used by the client 510 to transmit data to and receive data from the mobile device 102 .
- the remoting client 510 may collaborate with the window management service 206 (of the mobile computing device 102 ) to modify remote applications 505 for remote display on the mobile device 102 .
- the window management service 206 may be, for example, a CITRIX RECEIVER published by CITRIX SYSTEMS.
- the remoting client 510 may communicate with the remote applications 505 to intercept event providing notifications and data generated by the remote applications 505 .
- the remoting client 510 may perform event-based detection of one or more controls using application programming interfaces provided by one or more of the remote applications 505 .
- at least one remote application 505 may be an operating system executing on the remote computer 501 .
- Another remote application 505 B may be executing on the remote computer 501 and within the context of the operating system 505 .
- the remoting client 510 may intercept events generated by controls within the remote application 505 B using application program interfaces made available by the operating system 505 .
- the controls may be selected by the remoting client 510 based on whether the control has focus or is on a predetermined list of controls of interest.
- a control in some examples, may be an object within an application that a user interacts with, e.g., a text box, drop down menu, radio button, button, check box, edit box, combo box etc. The control may further be referred to as a field or interactive element.
- the remoting client 510 may intercept the control-generated events by registering with a communication interface associated with the application 505 B to receive notifications when a focus-change event occurs within the application 505 B. For example, the remoting client 510 may receive a notification when an object or control receives focus, e.g., indicating that a user selected a text box within the application.
- the remoting client 510 may communicate with the window management service 206 over the virtual channel. In a particular example, the remoting client 510 may send intercepted event notifications over the virtual channel to the window management service 206 . In other examples, the remoting client 510 may transmit location coordinates for a control, the type of control, the contents of a control, the window handle of a control, and/or parent window information for a control. Window handles or other identification information for elements within an interface or application may include a globally unique identifier (GUID).
- GUID globally unique identifier
- the window management service 206 may receive information from the remoting client 510 and adjust or modify display of an application 204 on the native display 201 using the received information. Adjusting the display may include panning, zooming (in or out), scrolling or otherwise modifying the display of the application 204 .
- the remote computer 501 may execute one or more resources 505 . These resources, in some examples, may be applications. In other examples, at least one resource may be an operating system executing on the remote computer 501 . In those examples, the other applications 505 may execute within the context of the operating system. In some instances, the applications 505 may correspond to remote applications 505 from the perspective of device 102 and one or more of applications 505 may be remoted to mobile device 102 for use by the user at mobile device 102 .
- FIG. 6 illustrates a general method 600 for altering or modifying the display of a remoted application on a mobile device.
- an application executing on a computer remote from the mobile device may intercept a focus event notification generated in response to a control within a remoted application receiving focus.
- the remote computer may forward the received or intercepted focus event notification to a client device through which a user is viewing and interacting with the remoted application (e.g., mobile computing device 102 of FIG. 5 ) in step 605 .
- the remoted application e.g., mobile computing device 102 of FIG. 5
- an application executing on the mobile computing device may modify a display of the remoted application in response to receiving the notification and according to information associated with the event (Step 610 ).
- FIG. 7 is a flowchart illustrating a method that may be performed by a computer executing an application for a remote client device (e.g., within the general process of method 600 ).
- a remoting management application executing on the computer may intercept or otherwise receive a focus event notification.
- the remoting management application may, in some examples, be a remoting client 510 executing on a remote computer 501 .
- the application may be any application executing on a remote computer 501 and able to hook into function calls issued by an application executing on the remote computer 501 .
- the remoting management application executing on the remote computer 501 may be a component inside WFSHELL.EXE (ICACTLS.DLL) or another service of Citrix XenApp or XenDesktop server.
- the remoting management application may be an application managing a control or seamless virtual channel established between the remote computer 501 and the mobile device 102 .
- the focus event notification may be a call issued by a remote application executing on the remote computer 501 when a control or field within the remote application receives focus, e.g., a user selects or otherwise highlights the control or field through mobile device 102 .
- control may be any object or control or user interactive or input element, such as: an edit box, a textbox, a memo field, a combo box, a drop-down menu, a slider, a list box, or any other similar object or control.
- the function may, for example, be a function such as ‘focus combo box” or any other function that generates a notification indicating a control receiving focus.
- the remoting management application may use event-based detection facilitated by WINDOWS 7 UI AUTOMATION application program interfaces. In other examples, the remoting management application may use event-based detection facilitated by any API. In these examples, the application may register for notifications provided by these APIs using a COM interface associated with a particular control and/or a particular function.
- the remoting management application may send additional information to the mobile device 102 , e.g., separately or as part of the notification message, including: the type of control or interface element that received focus (e.g., combo box, textbox); the name or an identifier of the control that received focus; the location or coordinates of the control within the application (e.g., where in the application output the control is located); the content of the control (e.g., values assigned to the control, strings associated with the control, etc.); a window handle of the control; a window identifier of the window displaying the control; the location or coordinates of the parent window displaying the control; or any additional information that could be used to determine the location of the control within the application output displayed by the mobile device 102 .
- the type of control or interface element that received focus e.g., combo box, textbox
- the name or an identifier of the control that received focus e.g., the location or coordinates of the control within the application (e.g., where in the application
- the remote computer 501 may determine the type of object or control element receiving focus and/or a position thereof. For example, the computer may determine whether the control element is a text entry field, a menu of selectable values, a radio button or the like. The computer may also determine position information based on position data specified in the notification message, including the coordinates and size of the control element on which focus is placed. In step 710 , the remote computer 501 may then determine a general type of user control element to activate on the mobile device based on the determined type of control element of the remoted application receiving focus.
- the computer may determine that a software keyboard of type standard alphanumeric would be appropriate for interacting with the text entry field in a touch-input environment.
- the computer may determine that a software keyboard of a number pad type may be most appropriate for interacting with the number entry field.
- the computer may transmit a command to the mobile device to activate a native control element corresponding to the determined type of control element locally on the mobile device.
- the command may be transmitted through a virtual channel between the remoting application and a remoting client application executing on the mobile device.
- the computer may transmit the command identifying the type of the control element and identification of an editable area or field for which the control element is to be activated.
- the identification of the editable area or field may include coordinates, size information and the like.
- the computer may transmit the command along with an identifier of the picker control, an area or field definition (e.g., coordinates, size information), contents of the picker control (e.g., the selectable items), current selection index, a title of the picker control and/or window and parent window handles.
- an identifier of the picker control e.g., a menu or scroll selection bar
- an area or field definition e.g., coordinates, size information
- contents of the picker control e.g., the selectable items
- current selection index e.g., a title of the picker control and/or window and parent window handles.
- the remote computer 501 might not determine a mobile device-native control element or type thereof to activate or use. Instead, the user of the mobile device may manually select or activate a native control element at the mobile device.
- the remote computer may receive an indication (e.g., a notification or confirmation) from the mobile device of the activation of a user control or input element native to the mobile device.
- the indication may include, for example, a confirmation of the state of the control element (e.g., show/hide), success or failure from execution of the command and/or an identifier of the local control element, e.g., mirroring the control identifier from the original command from the remote computer.
- the remote computer 501 may receive a return operation status message from the client device (e.g., mobile device 102 ) in step 725 , indicating an operation status of an activated local control element such as a picker control.
- the return operation status message may include the identifier of the control, the operation status of the control and a selection index if the operation status indicates a selection made by the user.
- the selection index may identify the selected item.
- the operation status may include success (indicating that a selection has been made by the user) or cancel/abort (e.g., if the user has deactivated the control element without selection).
- the remote computer 501 may further update the application display to reflect a selection made by a user (e.g., in a picker control element) in response to a notification from the client of the operation status and the selection.
- the selection field may be updated with the selected value and the updated application display may be transmitted to the client device (e.g., mobile device 102 ) for display.
- the application display update may then be transmitted to the client device in step 735 .
- the remote computer 501 may receive another notification indicating that focus has left the control element and/or the remoted application.
- the remote computer 501 may issue an instruction to the mobile device 102 (or a client application executing thereon) to return the application display to an original state and/or to deactivate the native control element in step 745 (e.g., if the application display was previously modified upon the remoted application receiving focus).
- the remote computer 501 may issue the instruction only upon receiving notification that the native control element is no longer active (e.g., being displayed) or in use.
- FIG. 8 is a flowchart illustrating a process by which a device such as mobile device 102 ( FIG. 5 ) may receive output data and instructions from a remote computing device executing a remoted application.
- the device e.g., a client device
- the management application may modify the display of the remoted application on the mobile device 102 in response to receiving instructions from the remote computing device (e.g., as described with respect to FIG. 7 ).
- the management application may be a window management service 206 executing on the mobile device 102 .
- the management application may be any application able to receive event and control information from a remote computer 501 and use it to modify output displayed on the native display 201 of mobile device 102 .
- the device may receive, from the remote computer executing the application, an instruction to activate a type of control element.
- the instruction may be received from the remote computer in response to the remote computer detecting an editable or control field receiving focus (e.g., a user selecting, through the client device, a text entry field in a remoted application display).
- the device may determine a native control (e.g., user input) element to activate based on the type of control element specified in the instruction in step 810 .
- a native control element e.g., user input
- the device may identify a native control element matching the specified type such as an iOS standard alphanumeric keyboard (if it is an iOS mobile device), or an Android standard alphanumeric keyboard (if it is an Android mobile device).
- the general types of controls supported by or available on the client device may be reported to the remote computer executing the application so that the remote computer may select an appropriate general type.
- the remote computer may process the focus event differently if a corresponding general type of control element is not supported by the client device, as is described in further detail herein.
- the instructions may specify activation of multiple types of controls. Operation of the multiple native controls may be based on the use and specification of the GUID assigned to each native control.
- the device may activate or execute the determined native control in step 815 .
- activation and/or execution of the native control may include modifying the native display interface to display the control.
- Activation and/or execution of the native control might also include extracting control element values (e.g., values within a drop-down menu) and incorporating those values into the native control element (e.g., the virtual picker in the above example).
- the remote computer executing the remoted application may transmit control values included in the application control element receiving focus to the device for display in the native control element.
- the window management service 206 at the mobile or client device may determine whether the control is a type of control having associated values. Upon determining that the control has associated values, the window management service 206 may modify the display of the application to include the received values. In one example, if one or more strings are associated with a combo box having received focus, the window management service 206 may modify the display to display a combo box that includes those strings.
- a client executing on the device, or a window management service 206 may use local controls native to the device to display a remote control or to display graphics representative of a remote control. For example, when a remote Windows Combo Box is presented in an application, the device may display and/or use a local iPad picker control.
- the remote application displayed on the device may adopt the appearance of a device native application and may include at least some native functionality.
- the device may further determine, in step 820 , whether the native control element will (or does) obscure the element of the remoted application receiving focus. For example, the device may determine an amount of the remoted application element being obscured (e.g., a percentage, absolute amount in pixels, inches or other metric), a particular area being obscured (e.g., obscuring a left part of a text entry field may be less important than obscuring a right part of the text entry field for language environments that read left to right), and the like. In some arrangements, the remoted application element might not be considered obscured if the amount obscured or portion obscured does not meet a predefined rule or threshold.
- an amount of the remoted application element being obscured e.g., a percentage, absolute amount in pixels, inches or other metric
- a particular area being obscured e.g., obscuring a left part of a text entry field may be less important than obscuring a right part of the text entry field for language environments that read left to right
- a remoted application element might only be considered obscured if at least 25%, at least 10%, at least 50%, etc. of the element is not visible.
- a remoted application might only be considered obscured if any portion of the left side of a text entry field (e.g., any part of the left 50% of the text entry field) is hidden from view.
- Other rules for determining when a control element is obscured may be defined and used.
- the device may determine, in step 825 , a manner in which to modify the local display such that at least the application control element is no longer obscured. For example, if a mobile device native keyboard is to be displayed at the bottom of the mobile device display and will obscure the application control element, the device may determine that the one or more portions of the local display (desktop interface or application specific window(s) or portions thereof) should be panned upward (e.g., auto-scroll).
- the device may modify the display (entire desktop, portions of the desktop and/or specific windows) to include a scrolling feature so that a remainder of the application display may be viewed once the display is panned upward if there is insufficient display real estate.
- the device may modify the application window to zoom out (e.g., shrink the application display) so that the application display fully fits within the mobile device display.
- step 830 the device may subsequently modify the display in accordance with the determined manner to avoid obscuring the editable/control field. Alternatively, if the editable/control field receiving focus will not be obscured, the device might not modify the display. In some examples, steps 820 - 830 may be performed in conjunction with (e.g., before, during or after) activation of the native control element in step 815 .
- the device may transmit a confirmation message to the remote computing device confirming activation of the native control element as shown in step 835 .
- the device may further transmit a return operation status message to the remote computing device in step 840 , as described above with respect to steps 725 and 730 of FIG. 7 .
- status messages may be provided when the device detects a user selection of an item in the picker control.
- the device may transmit a status message upon detecting a user cancelling or aborting the picker control. These events may be reported to the remote computing device in the update message for appropriate handling by the remote computing device. If the update message includes an event that causes changes to the application display, for instance, those changes to the application display may be received from the remote computing device for updating the local application display.
- the client device may further receive, in step 845 , a message indicating that focus has left the editable/control field and providing instructions to deactivate the corresponding native control element.
- the client device may deactivate the native control element and return the application display to a previous state (e.g., prior to activation of the native control element) in step 850 if modification was performed.
- the processes of FIGS. 7 and 8 may further include transmitting actions performed on or through a native control element of the client device back to the remote computing device and remote application.
- Application A executing on the remote computing device may generate application output which may be displayed in an application output window on the remote computing device.
- the application output generated by Application A is subsequently transmitted to the client device where it is displayed within an application output window thereon (e.g., a local application window).
- a client or a window management service 206 may intercept the on-focus event and may display or invoke a native, virtual picker control, e.g., an iPad picker control.
- the native, virtual picker control is displayed.
- a user may interact with the native control to generate input.
- the client or window management service 206 may transmit the input to the remote computing device for processing within the context of the remoted application.
- the remote computer 501 may inject the control input into an actual control displayed within the actual application output generated by Application A.
- injection of the received control input may be accomplished using a Windows UI Automation API. Changes in the remoted application based on the received input may then be outputted back to the client device. This process may repeat until the remoted application is closed and/or the remoting relationship between the client device and the remote computing device is ended.
- the remote computing device may further be configured to determine whether the client device supports or has local control elements such as a native virtual keyboard or a native picker corresponding to a particular type or category of control element. For example, the client device's capabilities may be specified to the remote computing device and as such, the remote computing device may make the above noted determination. If the client device does not support or does not have a local control element corresponding to a determined type of control element corresponding to the editable/control field, the remote computing device may generate a virtual keyboard or other control element as part of the remoted application or separately from the remoted application and output the display of the virtual keyboard to the client device (similar to the remoting of the application). In one particular example, the virtual control element generated by the remote computing device may be executed and remoted as a separate application. In another example, the virtual control element may be included within and as part of the existing remoted application receiving focus.
- local control elements such as a native virtual keyboard or a native picker corresponding to a particular type or category of control element.
- the activation of a native control element may be switched on or off by the user or by a service providing the application remoting capability. Accordingly, if the user or the service does not wish to use native controls, the device might not automatically trigger a native control element upon receiving a focus event notification for an editable field.
- the remote computing device executing the application may be configured to determine whether an editable/control field will be obscured and to determine and provide instructions for modifying a local display of the client device in response. Accordingly, the remote computing device may transmit instructions for modifying the local display and/or the application display of the client device.
- FIGS. 9A and 9B illustrate example interfaces displaying the use of native controls including a virtual keyboard.
- a native control of the device 900 such as virtual keyboard 901 is activated and displayed for entry of input into a text entry field without modifying a remoted application display 903 .
- the text entry field and other potentially important or relevant portions of the application display 903 become obscured by the virtual keyboard 901 .
- the application display may be panned upward.
- the display of an entire desktop including the remoted application display 903 may be modified.
- Panning may include moving the application display 903 in a particular direction (in this example, towards the top of the display) with or without modifying the proportions or orientation of the application display 903 or desktop.
- panning may result in elements that were previously visible being moved off of the device display. For example, recycle bin icon 907 in FIG. 9A is moved off of the displayable area after the application display 903 is panned upward as shown in FIG. 9B .
- the amount or degree to which the application display 903 is modified may be determined based on various rules and thresholds.
- the application display 903 may be panned until at least 75% of the text entry field is visible.
- the application display 903 may be panned until an entirety of the text entry field is visible.
- FIGS. 10A and 10B illustrate other example interfaces displaying the use of native controls including a scrolling picker.
- a web browser application 1001 is displayed with content having various control elements including a drop down menu 1003 .
- the drop down menu 1003 may include multiple selectable values that, in some arrangements, may be difficult to see because of font size, the number of values, size of the drop down portion of the menu and the like.
- the interface may display a native picker control element 1005 .
- the native picker element 1005 may be populated with the values included in drop down menu 1003 .
- the native picker control element 1005 may be displayed for easier viewing and accessibility (e.g., larger font size, easier selection mechanism, etc.).
- the application display might not be modified. Because the values listed in the drop down menu 1003 are also displayed in native picker control element 1005 , obscuring part or all of the drop down menu 1003 might not have as significant an impact in usability.
- the application display may be modified to insure that the non-expanded drop down menu portion remains visible so that a user may confirm his or her selection once populated into the drop down menu 1003 .
- the application display may zoom in to the drop down menu 1003 or a portion thereof to increase viewability.
- the methods and features recited herein may further be implemented through any number of computer readable media that are able to store computer readable instructions.
- Examples of computer readable media that may be used include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical disk storage, magnetic cassettes, magnetic tape, magnetic storage and the like.
- stationary or fixed location devices may also include the same or similar aspects and features.
- a personal computer or workstation e.g., plugged into a wall socket
- touch input controls e.g., touch input controls and provide access to a remoted application from a remote computing device.
- similar native control elements may be generated and used from the personal computer or workstation in similar fashion to a mobile device such as a tablet computer.
- the methods and features recited herein may be implemented through one or more integrated circuits (ICs).
- An integrated circuit may, for example, be a microprocessor that accesses programming instructions or other data stored in a read only memory (ROM).
- the ROM stores programming instructions that cause the IC to perform operations according to one or more of the methods described herein.
- one or more of the methods described herein are hardwired into an IC.
- the IC is in such cases an application specific integrated circuit (ASIC) having gates and other logic dedicated to the calculations and other operations described herein.
- the IC may perform some operations based on execution of programming instructions read from ROM or RAM, with other operations hardwired into gates and other logic of IC. Further, the IC may output image data to a display buffer.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- User Interface Of Digital Computer (AREA)
- Information Transfer Between Computers (AREA)
Abstract
Description
- This application is a continuation of co-pending U.S. application Ser. No. 16/929,339, entitled “DISPLAY MANAGEMENT FOR NATIVE USER EXPERIENCES,” filed on Jul. 15, 2020, which is a is a continuation of U.S. application Ser. No. 15/215,175, entitled “DISPLAY MANAGEMENT FOR NATIVE USER EXPERIENCES,” filed on Jul. 20, 2016 (now U.S. Pat. No. 10,761,692), which is a continuation of U.S. application Ser. No. 13/253,414, entitled “DISPLAY MANAGEMENT FOR NATIVE USER EXPERIENCES,” filed on Oct. 5, 2011 (now U.S. Pat. No. 9,400,585), which claims the priority benefit of U.S. Provisional Application No. 61/389,975, entitled “NATIVE USER EXPERIENCE FOR REMOTE APPLICATIONS DISPLAYED ON MOBILE DEVICES,” filed on Oct. 5, 2010. Each of the above-mentioned applications is incorporated herein by reference in its entirety.
- Aspects described herein generally relate to displaying remotely executing applications on devices. For example, aspects relate to providing a native application interaction experience on a device.
- Devices, including stationary devices such as personal computers and workstations and mobile devices such as smart phones and tablet computers, may execute operating systems and applications that are touch-friendly (e.g., compatible with touch input). These operating systems and applications often allow users to interact with menus and applications designed to accommodate using a touch screen as an input device. For example, some applications may be designed to handle a touch screen as the primary input such that when an editable control or field is selected or receives focus, the event may cause a virtual keyboard to appear.
FIG. 4A illustrates an example native mobile device application that permits a user to enter information into an editable field via a virtual keyboard displayed on the screen, and permits a user to select from a drop down box (e.g., combo box) using a virtual selector or picker control. In each of these examples, the touch screen of the mobile device may be used as the input device. - In some instances, an application may be displayed on a device that is not formatted for touch-screen input. For example, in some cases an application remotely executing on a server may be remotely displayed on a device as shown in
FIG. 4B . InFIG. 4B , a virtual keyboard is displayed over the editable control or field such that the user might not be able to see the control while the user is trying to input data into the control, thus making user input difficult. Additionally, the user may be required to manually activate the native control element of the client device, instead of being provided a native control element that is automatically triggered in response to selection of an editable area. Other input and viewability issues might also arise from touch compatible input elements being displayed on a remote device. - The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects. It is not intended to identify key or critical elements of the disclosure or to delineate the scope of the disclosure. The following summary merely presents some concepts of the disclosure in a simplified form as a prelude to the more detailed description provided below.
- According to one or more aspects, a display of a remoted application and native control elements may be managed so that one or more portions of the remoted application display are not obscured or hidden by active native control elements. In one example, a remote computing device executing the application for a client device may receive a notification when the application or a control element within the application receives focus. Upon receiving the notification, the remote computing device may notify the client device to modify the display of the application as necessary (e.g., to reflect the focus) and/or to determine a type of control element corresponding to the control element within the remoted application receiving focus. Additionally or alternatively, the remote computing device may instruct the client device to activate a native control element corresponding to the determined type of control element. The client device may then identify a native control element matching the determined type or category of control element and active the native control element. In some arrangements, the client device may further determine whether the native control element will overlap or otherwise obscure one or more portions of the remoted application display when activated. If so, the client device may modify the local display to avoid obscuring the remoted application display or the area of focus. Modification of the application display may include panning, zooming, scrolling and the like.
- According to another aspect, the client device may transmit confirmation and update messages to the remote computing device. The confirmation and update messages may include status information for the native control element and notifications of user interaction with the native control element.
- In some arrangements, the remote computing device executing the application may determine whether the native control element will obscure the application display and determine a manner in which to modify the local display of the client device.
- According to another aspect, a remote computing device executing the application may transmit values of an application control element receiving focus to the client device for populating the native control element. In one example, if the application control element receiving focus is a drop down menu, the values within the drop down menu may be transmitted to the client device for display in a native picker control element of the client device.
- According to yet another aspect, if a client device does not have native control elements, the remote computing device may generate a virtual control element and remote the control element to the client device similar to remoting an application to which input or interaction is to be registered. For example, if the client device is not capable of showing a local software keyboard, then an in-session keyboard (e.g., remoted by the server) may be displayed instead. The display of the in-session keyboard may be remoted (e.g., graphics transmitted) to the client device for use thereon.
- The details of these and other embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the disclosure will be apparent from the description and drawings, and from the claims.
- The present disclosure is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
-
FIG. 1A illustrates an example network environment that may be configured to provide remote access to computing devices and application programs executing thereon according to one or more aspects described herein. -
FIG. 1B andFIG. 1C are block diagrams illustrating example computing devices according to one or more aspects described herein. -
FIG. 2A is a block diagram illustrating an example system for displaying a plurality of resources in a user-configurable display layout on an external display device according to one or more aspects described herein. -
FIG. 2B is a block diagram illustrating an example system for mapping a display of one or more resources to one or more display devices according to one or more aspects described herein. -
FIG. 2C illustrates an example interface displaying a plurality of resources in a user-configurable display layout on an external display device, the user-configurable display layout divided into a grid according to one or more aspects described herein. -
FIG. 2D illustrates an example interface displaying a plurality of resources in a user-configurable, dynamic display layout on an external display device according to one or more aspects described herein. -
FIG. 3A is a block diagram illustrating example screen spaces provided by a mobile computing device attached to one or more external display devices according to one or more aspects described herein. -
FIG. 3B is a block diagram illustrating an example mobile computing device providing a plurality of screen spaces according to one or more aspects described herein. -
FIG. 3C is a block diagram illustrating an example logical representation of a plurality of screen spaces managed by a virtual graphics driver according to one or more aspects described herein. -
FIG. 4A is an example known interface displaying an application executing locally on a mobile device. -
FIG. 4B is an example known interface displaying a remoted application on a mobile device. -
FIG. 4C illustrates an example interface displaying a remoted application on a mobile device according to one or more aspects described herein. -
FIG. 5 is a block diagram illustrating an example system for altering the display of a remoted application on a mobile device according to one or more aspects described herein. -
FIG. 6 is a flow diagram illustrating an example method for altering the display of a remoted application on a mobile device according to one or more aspects described herein. -
FIG. 7 is a flowchart illustrating a method for modifying a remoted application display to accommodate the activation of native control elements according to one or more aspects described herein. -
FIG. 8 is a flowchart illustrating a method for managing a display on a device based on the activation of native control elements according to one or more aspects described herein. -
FIG. 9A andFIG. 9B illustrate an example sequence of user interfaces in which a native input element is activated and a display of an application is modified according to one or more aspects described herein. -
FIG. 10A andFIG. 10B illustrate another example sequence of user interfaces in which a native input element is activated and a display of an application is modified according to one or more aspects described herein. - Applications may be executed remotely by a first computing device for use at a second computing device (e.g., a client device). By using a remoted application, the client device may use processing power by the first computing device to execute the functionality and features of the remoted application. Additionally, data and applications may be served at a location that is accessible from multiple remote locations (e.g., through a public and/or private network).
-
FIG. 1A illustrates an example computing environment 101 that includes one or more client machines 102A-102N (collectively referred to herein as “client machine(s) 102”) that are in communication with one or more servers 106A-106N (generally referred to herein as “server(s) 106”). Installed in between the client machine(s) 102 and server(s) 106 is a network. - In one example, the computing environment 101 can include an appliance (e.g., a device or apparatus) installed between the server(s) 106 and client machine(s) 102. This appliance can manage client/server connections, and in some cases can load balance client connections amongst a plurality of backend servers.
- The client machine(s) 102 may, in some examples, be a
single client machine 102 or a single group ofclient machines 102, while server(s) 106 may be asingle server 106 or a single group ofservers 106. In one example, asingle client machine 102 may communicate with more than oneserver 106, while in another example, asingle server 106 may communicate with more than oneclient machine 102. In yet another example, asingle client machine 102 might only communicate with asingle server 106. - A
client machine 102 may, in some examples, be referenced by any one of the following terms: client machine(s) 102; client(s); client computer(s); client device(s); client computing device(s); local machine; remote machine; client node(s); endpoint(s); endpoint node(s); or a second machine. Theserver 106, in some examples, may be referenced by any one of the following terms: server(s), local machine; remote machine; server farm(s), host computing device(s), or a first machine(s). - According to one or more arrangements, the
client machine 102 may be a virtual machine 102C. The virtual machine 102C may be any virtual machine, while in some configurations, the virtual machine 102C may be any virtual machine managed by a hypervisor developed by XenSolutions, Citrix Systems, IBM, VMware, or any other hypervisor. In other examples, the virtual machine 102C can be managed by any hypervisor, while in still other examples, the virtual machine 102C can be managed by a hypervisor executing on aserver 106 or a hypervisor executing on aclient 102. - The
client machine 102 may, in some examples, execute, operate or otherwise provide an application such as: software; a program; executable instructions; a virtual machine; a hypervisor; a web browser; a web-based client; a client-server application; a thin-client computing client; an ActiveX control; a Java applet; software related to voice over internet protocol (VoIP) communications like a soft IP telephone; an application for streaming video and/or audio; an application for facilitating real-time-data communications; a HTTP client; a FTP client; an Oscar client; a Telnet client; or any other set of executable instructions. Accordingly, in at least some arrangements, remoting an application may include aclient device 102 receiving and displaying application output generated by an application remotely executing on aserver 106 or other remotely located machine or computing device. For example, theclient device 102 may display the application output in an application window, a browser, or other output window. In one example, the application is a desktop, while in other examples the application is an application that generates a desktop. A desktop may include a graphical shell providing a user interface for an instance of an operating system in which local and/or remote applications can be integrated. Applications, as used herein, are programs that execute after an instance of an operating system (and, optionally, also the desktop) has been loaded. Each instance of the operating system may be physical (e.g., one operating system per device) or virtual (e.g., many instances of an OS running on a single device). Each application may be executed on a local device, or executed on a remotely located device (e.g., remoted). Applications may be remoted in multiple ways. In one example, applications may be remoted in a seamless manner in which windows are created on the client device so as to make the application display seem as if it were running locally on the client device and the desktop thereof. In another example, a remoted application may be provided in a windowed mode where a desktop is remoted to the client device and the application is displayed as an application executing in the remoted desktop. Various other remoting methods and techniques may also be implemented or used. - According to some aspects, the
server 106 may execute a remote presentation client or other client or program that uses a thin-client or remote-display protocol to capture display output generated by an application executing on aserver 106 and to subsequently transmit the application display output to aremote client 102. The thin-client or remote-display protocol can be any one of the following protocols (or other suitable protocol): the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla.; or the Remote Desktop Protocol (RDP) manufactured by the Microsoft Corporation of Redmond, Wash. - The computing environment can include more than one server 106A-106N such that the servers 106A-106N are logically and/or physically grouped together into a
server farm 106. In one example, theserver farm 106 can includeservers 106 that are geographically dispersed and logically grouped together in aserver farm 106, or, in another example,servers 106 that are located proximate to each other and logically grouped together in aserver farm 106. Geographically dispersed servers 106A-106N within aserver farm 106 can, in some examples, communicate using a WAN, MAN, or LAN, where different geographic regions can be characterized as: different continents; different regions of a continent; different countries; different states; different cities; different zip codes; different neighborhoods; different campuses; different rooms; or any combination of the preceding geographical locations. In some examples, theserver farm 106 may be administered as a single entity, while in other examples, theserver farm 106 can includemultiple server farms 106. - A
server farm 106 may includeservers 106 that execute a substantially similar type of operating system platform (e.g., WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Wash., UNIX, LINUX, or SNOW LEOPARD.) In other examples, theserver farm 106 can include a first group ofservers 106 that execute a first type of operating system platform, and a second group ofservers 106 that execute a second type of operating system platform. Theserver farm 106, in other examples, can includeservers 106 that execute different types of operating system platforms. - The
server 106, in some examples, may be of any server type. In other examples, theserver 106 can be any of the following server types: a file server; an application server; a web server; a proxy server; an appliance; a network appliance; a gateway; an application gateway; a gateway server; a virtualization server; a deployment server; a SSL VPN server; a firewall; a web server; an application server or as a master application server; aserver 106 executing an active directory; or aserver 106 executing an application acceleration program that provides firewall functionality, application functionality, or load balancing functionality. In some examples, aserver 106 may be a RADIUS server that includes a remote authentication dial-in user service. In arrangements where theserver 106 includes an appliance, theserver 106 may be an appliance manufactured by any one of the following manufacturers: the Citrix Application Networking Group; Silver Peak Systems, Inc; Riverbed Technology, Inc.; F5 Networks, Inc.; or Juniper Networks, Inc. Some examples include a first server 106A that receives requests from aclient machine 102, forwards the request to a second server 106B, and responds to the request generated by theclient machine 102 with a response from the second server 106B. The first server 106A may acquire an enumeration of applications available to theclient machine 102 as well as address information associated with anapplication server 106 hosting an application identified within the enumeration of applications. The first server 106A may then present a response to the client's request using a web interface, and communicate directly with theclient 102 to provide theclient 102 with access to an identified application. - The
server 106 may, in some examples, execute any one of the following applications: a thin-client application using a thin-client protocol to transmit application display data to a client; a remote display presentation application; any portion of the CITRIX ACCESS SUITE by Citrix Systems, Inc. like the METAFRAME, CITRIX PRESENTATION SERVER, CITRIX XENAPP or CITRIX XENDESKTOP server; MICROSOFT WINDOWS Terminal Services manufactured by the Microsoft Corporation; or an ICA client, developed by Citrix Systems, Inc. Another example includes aserver 106 that is an application server such as: an email server that provides email services such as MICROSOFT EXCHANGE manufactured by the Microsoft Corporation; a web or Internet server; a desktop sharing server; a collaboration server; or any other type of application server. Still other examples include aserver 106 that executes any one of the following types of hosted servers applications: GOTOMEETING provided by Citrix Online Division, Inc.; WEBEX provided by WebEx, Inc. of Santa Clara, Calif.; or Microsoft Office LIVE MEETING provided by Microsoft Corporation. -
Client machines 102 may, in some examples, be a client node that seeks access to resources provided by aserver 106. In other examples, theserver 106 may provideclients 102 or client nodes with access to hosted resources. Theserver 106, in some examples, functions as a master node such that it communicates with one ormore clients 102 orservers 106. In some examples, the master node may identify and provide address information associated with aserver 106 hosting a requested application, to one ormore clients 102 orservers 106. In still other examples, the master node may be aserver farm 106, aclient 102, a cluster ofclient nodes 102, or an appliance. - One or
more clients 102 and/or one ormore servers 106 may transmit data over anetwork 104 installed between machines and appliances within the computing environment 101. Thenetwork 104 may comprise one or more sub-networks, and may be installed between any combination of theclients 102,servers 106, computing machines and appliances included within the computing environment 101. In some examples, thenetwork 104 may be: a local-area network (LAN); a metropolitan area network (MAN); a wide area network (WAN); aprimary network 104 comprised ofmultiple sub-networks 104 located between theclient machines 102 and theservers 106; a primarypublic network 104 with aprivate sub-network 104; a primaryprivate network 104 with apublic sub-network 104; or a primaryprivate network 104 with aprivate sub-network 104. Still further examples include anetwork 104 that may be any of the following network types: a point to point network; a broadcast network; a telecommunications network; a data communication network; a computer network; an ATM (Asynchronous Transfer Mode) network; a SONET (Synchronous Optical Network) network; a SDH (Synchronous Digital Hierarchy) network; a wireless network; a wireline network; or anetwork 104 that includes a wireless link where the wireless link may be an infrared channel or satellite band. The network topology of thenetwork 104 may differ within different examples, possible network topologies include: a bus network topology; a star network topology; a ring network topology; a repeater-based network topology; or a tiered-star network topology. Additional examples may include anetwork 104 of mobile telephone networks that use a protocol to communicate among mobile devices, where the protocol may be any one of the following: AMPS; TDMA; CDMA; GSM; GPRS UMTS; or any other protocol able to transmit data among mobile devices. -
FIG. 1B illustrates anexample computing device 100. Theclient machine 102 andserver 106 illustrated inFIG. 1A may be deployed as and/or executed on any example ofcomputing device 100 illustrated and described herein or any other computing device. Included within thecomputing device 100 is asystem bus 150 that communicates with the following components: acentral processing unit 121; amain memory 122;storage memory 128; an input/output (I/O)controller 123; display devices 124A-124N; aninstallation device 116; and anetwork interface 118. In one example, thestorage memory 128 includes: an operating system, software routines, and aclient agent 120. The I/O controller 123, in some examples, is further connected to akey board 126, and apointing device 127. Other examples may include an I/O controller 123 connected to one or more of input/output devices 130A-130N. -
FIG. 1C illustrates anexample computing device 100, where theclient machine 102 andserver 106 illustrated inFIG. 1A may be deployed as and/or executed on any example of thecomputing device 100 illustrated and described herein or any other computing device. Included within thecomputing device 100 is asystem bus 150 that communicates with the following components: abridge 170, and a first I/O device 130A. In another example, thebridge 170 is in further communication with the maincentral processing unit 121, where thecentral processing unit 121 may further communicate with a second I/O device 130B, amain memory 122, and acache memory 140. Included within thecentral processing unit 121, are I/O ports, amemory port 103, and a main processor. - Examples of the
computing machine 100 may include acentral processing unit 121 characterized by any one of the following component configurations: logic circuits that respond to and process instructions fetched from themain memory unit 122; a microprocessor unit, such as: those manufactured by Intel Corporation; those manufactured by Motorola Corporation; those manufactured by Transmeta Corporation of Santa Clara, Calif.; the RS/6000 processor such as those manufactured by International Business Machines; a processor such as those manufactured by Advanced Micro Devices; or any other combination of logic circuits. Still other examples of thecentral processing unit 122 may include any combination of the following: a microprocessor, a microcontroller, a central processing unit with a single processing core, a central processing unit with two processing cores, or a central processing unit with more than one processing core. - While
FIG. 1C illustrates acomputing device 100 that includes a singlecentral processing unit 121, in some examples thecomputing device 100 may include one ormore processing units 121. In these examples, thecomputing device 100 may store and execute firmware or other executable instructions that, when executed, direct the one ormore processing units 121 to simultaneously execute instructions or to simultaneously execute instructions on a single piece of data. In other examples, thecomputing device 100 may store and execute firmware or other executable instructions that, when executed, direct the one or more processing units to each execute a section of a group of instructions. For example, eachprocessing unit 121 may be instructed to execute a portion of a program or a particular module within a program. - In some examples, the
processing unit 121 may include one or more processing cores. For example, theprocessing unit 121 may have two cores, four cores, eight cores, etc. In one example, theprocessing unit 121 may comprise one or more parallel processing cores. The processing cores of theprocessing unit 121, may in some examples access available memory as a global address space, or in other examples, memory within thecomputing device 100 may be segmented and assigned to a particular core within theprocessing unit 121. In one example, the one or more processing cores or processors in thecomputing device 100 may each access local memory. In still another example, memory within thecomputing device 100 may be shared amongst one or more processors or processing cores, while other memory may be accessed by particular processors or subsets of processors. In examples where thecomputing device 100 includes more than one processing unit, the multiple processing units may be included in a single integrated circuit (IC). These multiple processors, in some examples, may be linked together by an internal high speed bus, which may be referred to as an element interconnect bus. - In examples where the
computing device 100 includes one ormore processing units 121, or aprocessing unit 121 including one or more processing cores, the processors may execute a single instruction simultaneously on multiple pieces of data (SIMD), or in other examples may execute multiple instructions simultaneously on multiple pieces of data (MIMD). In some examples, thecomputing device 100 may include any number of SIMD and MIMD processors. - The
computing device 100, in some examples, may include a graphics processor or a graphics processing unit (Not Shown). The graphics processing unit may include any combination of software and hardware, and may further input graphics data and graphics instructions, render a graphic from the inputted data and instructions, and output the rendered graphic. In some examples, the graphics processing unit may be included within theprocessing unit 121. In other examples, thecomputing device 100 may include one ormore processing units 121, where at least oneprocessing unit 121 is dedicated to processing and rendering graphics. - In one example, computing
machine 100 may include acentral processing unit 121 that communicates withcache memory 140 via a secondary bus also known as a backside bus, while another example of thecomputing machine 100 includes acentral processing unit 121 that communicates with cache memory via thesystem bus 150. Thelocal system bus 150 may, in some examples, also be used by the central processing unit to communicate with more than one type of I/O device 130A-130N. In some examples, thelocal system bus 150 may be any one of the following types of buses: a VESA VL bus; an ISA bus; an EISA bus; a MicroChannel Architecture (MCA) bus; a PCI bus; a PCI-X bus; a PCI-Express bus; or a NuBus. Alternatively or additionally, computingmachine 100 may include an I/O device 130A-130N that is a video display 124 that communicates with thecentral processing unit 121. Still other versions of thecomputing machine 100 include aprocessor 121 connected to an I/O device 130A-130N via any one of the following connections: HyperTransport, Rapid I/O, or InfiniBand. Further examples of thecomputing machine 100 include aprocessor 121 that communicates with one I/O device 130A using a local interconnect bus and a second I/O device 130B using a direct connection. - The
computing device 100, in some examples, includes amain memory unit 122 andcache memory 140. Thecache memory 140 may be any memory type, and in some examples may be any one of the following types of memory: SRAM; BSRAM; or EDRAM. Other examples includecache memory 140 and amain memory unit 122 that may be any one of the following types of memory: Static random access memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM); Dynamic random access memory (DRAM); Fast Page Mode DRAM (FPM DRAM); Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM); Extended Data Output DRAM (EDO DRAM); Burst Extended Data Output DRAM (BEDO DRAM); Enhanced DRAM (EDRAM); synchronous DRAM (SDRAM); JEDEC SRAM; PC100 SDRAM; Double Data Rate SDRAM (DDR SDRAM); Enhanced SDRAM (ESDRAM); SyncLink DRAM (SLDRAM); Direct Rambus DRAM (DRDRAM); Ferroelectric RAM (FRAM); or any other type of memory. Further examples include acentral processing unit 121 that may access themain memory 122 via: asystem bus 150; amemory port 103; or any other connection, bus or port that allows theprocessor 121 to accessmemory 122. - In one or more arrangements,
computing device 100 may provide support for any one of the following installation devices 116: a CD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, tape drives of various formats, USB device, a bootable medium, a bootable CD, a bootable CD for GNU/Linux distribution such as KNOPPIX®, a hard-drive or any other device suitable for installing applications or software. Applications may, in some examples, include aclient agent 120, or any portion of aclient agent 120. Thecomputing device 100 may further include astorage device 128 that may be either one or more hard disk drives, or one or more redundant arrays of independent disks; where the storage device is configured to store an operating system, software, programs applications, or at least a portion of theclient agent 120.Computing device 100 may also include aninstallation device 116 that is used as thestorage device 128. - The
computing device 100 may further include anetwork interface 118 to interface to a Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56kb, X.25, SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or some combination of any or all of the above. Connections may also be established using a variety of communication protocols (e.g., TCP/IP, IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), RS232, RS485, IEEE 802.11, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, CDMA, GSM, WiMax and direct asynchronous connections). One version of thecomputing device 100 includes anetwork interface 118 able to communicate withadditional computing devices 100′ via any type and/or form of gateway or tunneling protocol such as Secure Socket Layer (SSL) or Transport Layer Security (TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems, Inc. Versions of thenetwork interface 118 may comprise any one of: a built-in network adapter; a network interface card; a PCMCIA network card; a card bus network adapter; a wireless network adapter; a USB network adapter; a modem; or any other device suitable for interfacing thecomputing device 100 to a network capable of communicating and performing the methods and systems described herein. - According to one or more aspects,
computing device 100 may include any one of the following I/O devices 130A-130N: akeyboard 126; apointing device 127; mice; trackpads; an optical pen; trackballs; microphones; drawing tablets; video displays; speakers; inkjet printers; laser printers; and dye-sublimation printers; or any other input/output device able to perform the methods and systems described herein. An I/O controller 123 may in some examples connect to multiple I/O devices 103A-130N to control the one or more I/O devices. In some examples, I/O devices 130A-130N may be configured to provide storage or aninstallation medium 116, while others may provide a universal serial bus (USB) interface for receiving USB storage devices such as the USB Flash Drive line of devices manufactured by Twintech Industry, Inc. Additionally or alternatively,computing device 100 may include I/O device 130 that may be a bridge between thesystem bus 150 and an external communication bus, such as: a USB bus; an Apple Desktop Bus; an RS-232 serial connection; a SCSI bus; a FireWire bus; aFireWire 800 bus; an Ethernet bus; an AppleTalk bus; a Gigabit Ethernet bus; an Asynchronous Transfer Mode bus; a HIPPI bus; a Super HIPPI bus; a SerialPlus bus; a SCI/LAMP bus; a FibreChannel bus; or a Serial Attached small computer system interface bus. - In some examples, the
computing machine 100 may connect to multiple display devices 124A-124N, in other examples thecomputing device 100 may connect to a single display device 124, while in still other examples thecomputing device 100 connects to display devices 124A-124N that are the same type or form of display, or to display devices that are different types or forms. Examples of the display devices 124A-124N may be supported and enabled by the following: one or multiple I/O devices 130A-130N; the I/O controller 123; a combination of I/O device(s) 130A-130N and the I/O controller 123; any combination of hardware and software able to support a display device 124A-124N; any type and/or form of video adapter, video card, driver, and/or library to interface, communicate, connect or otherwise use the display devices 124A-124N. Thecomputing device 100 may in some examples be configured to use one or multiple display devices 124A-124N, these configurations include: having multiple connectors to interface to multiple display devices 124A-124N; having multiple video adapters, with each video adapter connected to one or more of the display devices 124A-124N; having an operating system configured to support multiple displays 124A-124N; using circuits and software included within thecomputing device 100 to connect to and use multiple display devices 124A-124N; and executing software on themain computing device 100 and multiple secondary computing devices to enable themain computing device 100 to use a secondary computing device's display as a display device 124A-124N for themain computing device 100. Still other examples of thecomputing device 100 may include multiple display devices 124A-124N provided by multiple secondary computing devices and connected to themain computing device 100 via a network. - According to one or more arrangements, the
computing machine 100 may execute any operating system including: versions of the MICROSOFT WINDOWS operating systems such as WINDOWS 3.x; WINDOWS 95; WINDOWS 98; WINDOWS 2000; WINDOWS NT 3.51; WINDOWS NT 4.0; WINDOWS CE; WINDOWS XP; and WINDOWS VISTA; the different releases of the Unix and Linux operating systems; any version of the MAC OS manufactured by Apple Computer; OS/2, manufactured by International Business Machines; any embedded operating system; any real-time operating system; any open source operating system; any proprietary operating system; any operating systems for mobile computing devices; or any other operating system. In some examples, thecomputing machine 100 may execute multiple operating systems. For example, thecomputing machine 100 may execute PARALLELS or another virtualization platform that may execute or manage a virtual machine executing a first operating system, while thecomputing machine 100 executes a second operating system different from the first operating system. - The
computing machine 100 may be embodied in any one of the following computing devices: a computing workstation; a desktop computer; a laptop or notebook computer; a server; a handheld computer (e.g., a tablet computer such as the iPad andiPad 2 manufactured by Apple Computer); a mobile telephone; a portable telecommunication device; a media playing device; a gaming system; a mobile computing device; a netbook; a device of the IPOD family of devices manufactured by Apple Computer; any one of the PLAYSTATION family of devices manufactured by the Sony Corporation; any one of the Nintendo family of devices manufactured by Nintendo Co; any one of the XBOX family of devices manufactured by the Microsoft Corporation; or any other type and/or form of computing, telecommunications or media device that is capable of communication and that has sufficient processor power and memory capacity to perform the methods and systems described herein. In other examples thecomputing machine 100 may be a mobile device such as any one of the following mobile devices: a JAVA-enabled cellular telephone or personal digital assistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the im1100, all of which are manufactured by Motorola Corp; the 6035 or the 7135, manufactured by Kyocera; the i300 or i330, manufactured by Samsung Electronics Co., Ltd; theTREO 180, 270, 600, 650, 680, 700p, 700w, or 750 smart phone manufactured by Palm, Inc; any computing device that has different processors, operating systems, and input devices consistent with the device; or any other mobile computing device capable of performing the methods and systems described herein. In still other examples, thecomputing device 100 may be any one of the following mobile computing devices: any one series of Blackberry, or other handheld device manufactured by Research In Motion Limited; the iPhone manufactured by Apple Computer; Palm Pre; a Pocket PC; a Pocket PC Phone; or any other handheld mobile device. - In some examples, the
computing device 100 may have different processors, operating systems, and input devices consistent with the device. For example, in one example, thecomputing device 100 is aTREO 180, 270, 600, 650, 680, 700p, 700w, or 750 smart phone manufactured by Palm, Inc. In some of these examples, the TREO smart phone is operated under the control of the PalmOS operating system and includes a stylus input device as well as a five-way navigator device. - According to one or more configurations, the
computing device 100 may be a mobile device, such as a JAVA-enabled cellular telephone or personal digital assistant (PDA), such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the im1100, all of which are manufactured by Motorola Corp. of Schaumburg, Ill., the 6035 or the 7135, manufactured by Kyocera of Kyoto, Japan, or the i300 or i330, manufactured by Samsung Electronics Co., Ltd., of Seoul, Korea. In some examples, thecomputing device 100 is a mobile device manufactured by Nokia of Finland, or by Sony Ericsson Mobile Communications AB of Lund, Sweden. - In some examples, the
computing device 100 may be a Blackberry handheld or smart phone, such as the devices manufactured by Research In Motion Limited, including the Blackberry 7100 series, 8700 series, 7700 series, 7200 series, the Blackberry 7520, or the Blackberry Pearl 8100. In yet other examples, thecomputing device 100 is a smart phone, Pocket PC, Pocket PC Phone, or other handheld mobile device supporting Microsoft Windows Mobile Software. Moreover, thecomputing device 100 may be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile telephone, any other computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein. - Additionally or alternatively, the
computing device 100 may be a digital audio player. For example, thecomputing device 100 is a digital audio player such as the Apple IPOD, IPOD Touch, IPOD NANO, and IPOD SHUFFLE lines of devices, manufactured by Apple Computer of Cupertino, Calif. In another of these examples, the digital audio player may function as both a portable media player and as a mass storage device. In other examples, thecomputing device 100 is a digital audio player such as the DigitalAudioPlayer Select MP3 players, manufactured by Samsung Electronics America, of Ridgefield Park, N.J., or the Motorola m500 or m25 Digital Audio Players, manufactured by Motorola Inc. of Schaumburg, Ill. In still other examples, thecomputing device 100 is a portable media player, such as the Zen Vision W, the Zen Vision series, the Zen Portable Media Center devices, or the Digital MP3 line of MP3 players, manufactured by Creative Technologies Ltd. In yet other examples, thecomputing device 100 is a portable media player or digital audio player supporting file formats including, but not limited to, MP3, WAV, M4A/AAC, WMA Protected AAC, AIFF, Audible audiobook, Apple Lossless audio file formats and .mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats. - The
computing device 100 may, alternatively or additionally, include a combination of devices, such as a mobile phone combined with a digital audio player or portable media player. In one of these examples, thecomputing device 100 is a Motorola RAZR or Motorola ROKR line of combination digital audio players and mobile phones. In another of these examples, thecomputing device 100 is an iPhone smartphone, manufactured by Apple Computer of Cupertino, Calif. - Referring now to
FIG. 2A , the block diagram illustrates an example system for displaying a plurality of resources on one or more display devices. The system includes amobile computing device 102 that may communicate with one or moreexternal display devices 202 a-n. The example inFIG. 2A shows amobile computing device 102 with anative display 201, although devices without native displays may also be used. Themobile computing device 102 executes a plurality of resources 204 a-n (collectively, 204). Thewindow management system 206 andvirtual graphics driver 208 manage the locations and sizes of the display of output data associated with each of the plurality of resources in a user-configurable display layout. In many examples, themobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to anexternal display device 202. In some of these examples, themobile computing device 102 transmits the output data upon establishing a connection with theexternal display device 202. Alternatively or additionally, themobile computing device 102 transmits the output data associated with each of the plurality of resources 204 to the device'snative display 201. For example, themobile computing device 102 might only transmit the output data to the device's native display or only transmit the output data to one or more of theexternal display devices 202 a-n. In some examples, themobile computing device 102 transmits the output data associated with certain of the plurality of resources 204 to thenative display 201 and transmits the output data associated with other ones of the plurality of resources 204 to theexternal display devices 202 a-n. -
Mobile computing device 102 may be configured to execute a plurality of resources 204. In one example, themobile computing device 102 is aclient 102 as described above in connection withFIGS. 1A-1C . In another example, themobile computing device 102 displays the output data associated with aresource 204 a in a plurality of resources 204 a-n executed by themobile computing device 102. In some examples, themobile computing device 102 displays the output data associated with each of the plurality of resources 204. - A resource in the plurality of resources 204 may include, without limitation, a data file, an executable file, configuration files, an application, a desktop environment (which may itself include a plurality of applications for execution by the user), a computing environment image (such as a virtual machine image), and/or operating system software or other applications needed to execute a computing environment image.
- According to one or more arrangements,
mobile computing device 102 includes awindow management service 206 allowing anexternal display device 202 to display the output data associated with each of a plurality of resources 204 executed on themobile computing device 102. Thewindow management service 206 may also allow multiple resources running on themobile computing device 102 to be viewed on anexternal display device 202 or thenative display 201 at substantially the same time, as opposed to allowing the output of one resource 204 to be viewed exclusively on thenative display 201 orexternal display device 202. In one example, thewindow management service 206, in conjunction with avirtual graphics driver 208, manages the display layout of the windows displayed on theexternal display device 202 and thenative display 201. In some examples, thevirtual graphics driver 208 is a driver-level component that manages a virtual screen frame buffer storing output data that will be displayed by thenative display 201 on themobile computing device 102 or anexternal display device 202. Thewindow management service 206, in conjunction with thevirtual graphics driver 208, may further manage the boundaries and size of a screen space used to display output data and on which display device the output data is displayed. - In some examples, an
external display device 202 receives output data associated with each of the plurality of resources 204 and displays the output data in a user-configurable display layout. In one example, theexternal display device 202 includes a dock to which themobile computing device 102 connects. In another example, theexternal display device 202 includes a receiver for communicating with themobile computing device 102 wirelessly, for example, via BLUETOOTH, Wi-Fi or other networking protocols, as described above in connection withFIGS. 1A-1C . In still another example, theexternal display device 202 may be a display device 124 as described above in connection withFIGS. 1B and 1C . - Referring now to
FIG. 2B , the block diagram illustrates an example system for mapping the display of one or more resources 204 of themobile computing device 102 on one ormore display devices 201 and/or 202. In various examples, thewindow management service 206 manages avirtual screen space 210. Thevirtual screen space 210 may map to thenative display 201 and one or moreexternal display devices 202. Thewindow management service 206 may position output data associated with the user interfaces of one or more resources 204 on thevirtual screen space 210 to specify where each user interface will be displayed. In some examples, thewindow management service 206 positions the output data according to a user preference. In additional examples, thewindow management service 206 positions the output data according to a policy. In various examples, thewindow management service 206 positions the output data based on the resource 204 associated with the output data. - The
window management service 206 communicates with thevirtual graphics driver 208 to transmit output data associated with user interfaces of resources 204 to thenative display 201 and one or moreexternal display devices 202. In some examples, thewindow management service 206 may transmit output data and associated coordinates from thevirtual screen space 210 to thevirtual graphics driver 208. In various examples, thevirtual graphics driver 208 stores the output data in a virtual screen frame buffer. In many examples, thevirtual graphics driver 208 transmits the entries in the virtual screen frame buffer to thenative display 201 andexternal display devices 202. In many examples, thevirtual graphics driver 208 transmits an entry in the virtual screen frame buffer to anative display 201 or anexternal display device 202 based on the position of the entry in the frame buffer. - Referring now to
FIG. 2C , the user interface illustrates an example user-configurable display layout in which theexternal display device 202 displays the output data of the resources 204. In this example, the output data of the resources 204 is displayed in a grid display layout. The grid display layout may include one or more cells in an arrangement. A cell may display output data associated with a resource. In some examples, more than one cell displays output data associated with the same resource. - Additionally, in some examples, the cells are uniformly sized, whereas in other examples, the cells have different sizes. The cells may be arranged in any configuration. In some examples, the cells may be arranged in rows, columns, or both. A cell may have a descriptor associated with the cell's position in the grid. The descriptor may indicate the position of a cell within a row. In the example depicted in
FIG. 2C , the cell forresource 204 a may have the descriptor “1-1,” the cell forresource 204 b may have the descriptor “1-2,” the cell forresource 204 c may have the descriptor “1-3,” the cell forresource 204 d may have the descriptor “2-1,” and the cell forresource 204 d may have the descriptor “2-2.” In other examples, the cells may be numbered, e.g., “Cell 1,” “Cell 2,” etc. However, any system of choosing descriptors known to those of ordinary skill in the art may be used. - In various examples, the
window management service 206 configures a grid display layout according to the resources 204 being displayed on thenative display 201 or theexternal display device 202. In some examples, theservice 206 configures a grid display layout according to the number of resources 204 being displayed. In other examples, theservice 206 configures a grid display layout according to the size or amount of content in the user interfaces of the resources 204. For example, if anexternal display device 202 will display four resources with comparable amounts of content, thewindow management service 206 may configure a grid display layout with four uniform cells. In another example, if anexternal display device 202 will display four resources and one resource includes three times as much content as the others, thewindow management service 206 may configure a grid display layout with three uniform cells in a first row and a single cell in a second row. The single cell in the second row may be three times as wide as the cells in the first row. In various examples, thewindow management service 206 may configure a grid display layout to reserve a cell for displaying information about the resources being displayed, such as a menu of the resources. In many examples, thewindow management service 206 may configure a grid display layout to reserve a cell for allowing a user to configure the grid display layout. - Referring now to
FIG. 2D , a user interface illustrates an example user-configurable display layout providing a dynamic display layout in which theexternal display device 202 displays the output data associated with the plurality of resources 204. In this example, windows on theexternal display device 202 that display output data for resources 204 may be dynamically positioned and sized. Thewindow management service 206 may position a user interface for a resource at a default position and with a default size chosen according to a policy, the resource 204, or any other method. Thewindow management service 206 may order overlapping user interfaces such that higher-order user interfaces obscure lower-order user interfaces. Thewindow management service 206 may transmit output data to thevirtual graphics driver 208 reflecting the obfuscation. The user may re-position or re-size a window by, for example, clicking and dragging the window or a window edge. In these examples, thevirtual graphics driver 208 may detect the user's change to the window, and transmit information about the user's change to thewindow management service 206. Thewindow management service 206 may process the change and transmit updated output data to thevirtual graphics driver 208. In some examples, the user moves the user interface for a resource 204 to any location on anative display 201 orexternal display device 202. In some examples, the user moves the user interface for a resource 204 to a different display device. In some examples, the updated output data indicates that one user interface's size has been increased or location has been adjusted to obscure another user interface. In other examples, the updated output data indicates that one user interface's size has been decreased or location has been adjusted such that more of another user interface shall be visible. - Referring now to
FIG. 3A , a block diagram illustrates examplevirtual screen spaces 210 provided by a mobile computing device attached to one or more external display devices. As shown inFIG. 3A , themobile computing device 102 includes avirtual graphics driver 208 and avirtual screen 210. Thevirtual screen 210 includes a plurality ofvirtual screen spaces 310 and 312 a-n.Virtual screen space 310 may be a native display screen space for thenative display 201 on themobile computing device 102. The other virtual screen spaces 312 a-n may be extended screen spaces that correspond to the displays ofexternal display devices 202. Thewindow management service 206 andvirtual graphics driver 208 manage thevirtual screen 210. In one example, thevirtual graphics driver 208 uses a virtual screen frame buffer to manage the mobile computing device'snative display 201 and change the native display's 201 screen resolution. In another example, thevirtual graphics driver 208 uses a virtual screen frame buffer to manage an extended screen space 312 and to change a resolution of the extended screen 312. - In some examples, the
virtual graphics driver 208 allocates and manages a plurality ofvirtual screen spaces 310, 312 a-n and virtual screen frame buffers. In some of these examples, each virtual screen space and virtual screen frame buffer has a resolution independent of the other screen spaces and frame buffers. In one of these examples, output data associated with each of the plurality of resources 204 may reside within any of thevirtual screen spaces 310, 312 a-n. In another of these examples, each of the extended screen spaces 312 a-n is associated with at least oneexternal display device 202, depending on the capabilities of the device. - In various examples, the
window management service 206 and thevirtual graphics driver 208 allocate and manage the display, on a plurality ofexternal display devices 202, of output data associated with a plurality of resources. For example, output data associated with aresource 204 a displays on amobile computing device 102, output data associated with aresource 204 b displays on oneexternal display device 202 a, and output data associated with aresource 204 c display on another external display device 202 b. In another of these examples, thewindow management device 206 identifies one of theexternal display devices 202 for displaying output data generated by aresource 204 a based upon a type of theresource 204 a. For example, thewindow management service 206 may determine that a type of resource rendering a video may display on a television screen, while a type of resource rendering a word processing application may render on a display of a laptop computer. - Referring now to
FIG. 3B , a block diagram illustrates an examplemobile computing device 102 providing avirtual screen 210 withvirtual screen spaces 310, 312 a-n of varying resolutions. In this example, thevirtual screen 210 includes a nativedisplay screen space 310 corresponding to thenative display 201 of themobile computing device 102 with a resolution of 320 pixels×240 pixels. Thevirtual screen 210 also includes anextended screen 312 a corresponding to the display of anexternal display device 202 with a resolution of 1024 pixels×768 pixels, anextended screen 312 n-1 corresponding to the display of anexternal display device 202 with a resolution of 800 pixels×600 pixels, and anextended screen 312 n corresponding to the display of anexternal display device 202 with a resolution of 640 pixels×480 pixels. In many examples, thevirtual screen 210 may include a nativedisplay screen space 310 and any number of extended screens 312 of any resolution. The entirevirtual screen space 210 may be mapped into a single virtual screen frame buffer, although examples that map into multiple buffers may be used. - Referring now to
FIG. 3C , the block diagram illustrates an example logical representation of a plurality of virtual screen spaces managed by a virtual graphics driver. In this example, thevirtual graphics driver 208 manages multiple virtual screen spaces with different resolutions in a virtual screen frame buffer. In this example, thenative display 201 of the mobile computing device is the primary display and theexternal display device 202, corresponding to theextended screen 312 a, is a secondary display. In various examples, output data associated with resources 204 on the nativedisplay screen space 310 will be displayed on thenative display 201 and output data associated with resources 204 on theextended screen space 312 a will be displayed on theexternal display device 202 associated with theextended screen spaces 312 a. In other examples, all output data associated with resources 204 may be displayed on thenative display 201. In still other arrangements, all output data associated with resources 204 may be displayed on an external display device such asexternal display device 202. -
FIG. 5 illustrates an example system in which applications may be remotely executed at a server while providing display and user interaction through a client device such asmobile device 102. Accordingly, a user may view and interact with an application remoted from (e.g., remotely executed by the server (e.g., remote computer 501)) at a client device (e.g., mobile device 102) without requiring the application and/or data to be installed or stored on the client device. The system ofFIG. 5 may include a mobile computing device 102 (e.g., similar to that shown inFIG. 2B ) configured to execute and/or store one or more resources 204 such as applications executing on themobile computing device 102. In some examples, at least one resource may be an operating system executing on themobile device 102 and one or more other resources may execute within the context of the operating system. Themobile device 102 may further execute awindows management service 206 that may communicate with other applications executing on themobile device 102 and, in some examples, may communicate with avirtual graphics driver 208 and the resources or applications 204 executing on themobile device 102. Themobile computing device 102 may further execute avirtual graphics driver 208, and may store a buffer for generating avirtual screen space 210 that may include a buffer for a nativedisplay screen space 210 a. Themobile device 102 may communicate with anative display 201 of themobile device 102 on which application output generated by a resource 204 may be displayed. Themobile device 102 may further communicate with aremote computer 501 that may execute one ormore resources 505A-505N (generally referred to as remote resources 505) which may in some examples be remote applications 505. Theremote computer 501 may further execute aremoting client 510. The remote applications 505 may be executed by the remote computer for use at themobile device 102 in one or more arrangements as discussed in further detail below. When one or more remote applications 505 are executed by the remote computer for use at themobile device 102, those applications may be considered remoted applications. - According to one or more arrangements, resources 204 executing on the
mobile device 102 and/orresources 505A-505N may be applications configured to accept data inputted by a touch screen. For example, the applications may be modified versions of applications that typically do not receive data from a touch screen. These modified applications 204 may receive data inputted by a user via a touch screen of themobile device 102, and may be modified to accommodate typical touch screen input functions like a virtual keyboard and a virtual scroll menu. For example, a modified version of SAFARI, a web browser published by APPLE, may be modified to pan up when a user selects or gives focus to an edit control such as a text box. SAFARI pans the application upwards to accommodate the virtual keyboard displayed over the browser and so that a user may view the text box while typing on the virtual keyboard. - Touch input may include a variety of hand or finger movements and actions including touching a point on the screen, swiping movements, flicking movements, multi-finger or multi-touch actions (e.g., pinching, expansion) and gestures and the like. The touch input may trigger different functionalities depending on the type of application receiving the input. For example, some applications may receive a flicking action as closing the application while other application may interpret a flicking input as moving the application window in a certain direction. Each applications may define its own interpretation of different types of touch input or may use a standard definition provided by an underlying operating system.
- According to one or more configurations, the
mobile device 102 may communicate withremote computer 501 in a variety of ways. Theremote computer 501 may be a server, a client or any other computing machine. In some examples theremote computer 501 may be a remote application server that executes one or more applications formobile device 102. Accordingly, in some examples, themobile device 102 may communicate with theremote computer 501 over one or more virtual channels. In some examples, virtual channels may be established over a network and may be referred to as a control virtual channel. In other examples, the virtual channel may be a seamless virtual channel. A control virtual channel may be used to remote control commands and other miscellaneous commands (e.g., for an application) while the seamless virtual channel may be used to remote application windows, a taskbar, a systray, etc. Thus, in some examples, each virtual channel may remote different functions and content. In still other examples, the virtual channel may be established by awindow management service 206 executing on themobile device 102, or the virtual channel may be established by both thewindow management service 206 and aremoting client 510 executing onremote computer 501. The virtual channel may in some examples facilitate communication sent using the ICA protocol. - In some examples, the
remote computer 501 may execute aremoting client 510. Theremoting client 510 may be a control virtual channel or a seamless virtual channel and may be a remoting application that corresponds to the virtual channel used by theclient 510 to transmit data to and receive data from themobile device 102. In some examples, theremoting client 510 may collaborate with the window management service 206 (of the mobile computing device 102) to modify remote applications 505 for remote display on themobile device 102. Thewindow management service 206 may be, for example, a CITRIX RECEIVER published by CITRIX SYSTEMS. In some examples, theremoting client 510 may communicate with the remote applications 505 to intercept event providing notifications and data generated by the remote applications 505. In particular, theremoting client 510 may perform event-based detection of one or more controls using application programming interfaces provided by one or more of the remote applications 505. For example, at least one remote application 505 may be an operating system executing on theremote computer 501. Anotherremote application 505B may be executing on theremote computer 501 and within the context of the operating system 505. In these examples, theremoting client 510 may intercept events generated by controls within theremote application 505B using application program interfaces made available by the operating system 505. The controls may be selected by theremoting client 510 based on whether the control has focus or is on a predetermined list of controls of interest. A control, in some examples, may be an object within an application that a user interacts with, e.g., a text box, drop down menu, radio button, button, check box, edit box, combo box etc. The control may further be referred to as a field or interactive element. In some examples, theremoting client 510 may intercept the control-generated events by registering with a communication interface associated with theapplication 505B to receive notifications when a focus-change event occurs within theapplication 505B. For example, theremoting client 510 may receive a notification when an object or control receives focus, e.g., indicating that a user selected a text box within the application. - In some examples, the
remoting client 510 may communicate with thewindow management service 206 over the virtual channel. In a particular example, theremoting client 510 may send intercepted event notifications over the virtual channel to thewindow management service 206. In other examples, theremoting client 510 may transmit location coordinates for a control, the type of control, the contents of a control, the window handle of a control, and/or parent window information for a control. Window handles or other identification information for elements within an interface or application may include a globally unique identifier (GUID). Thewindow management service 206 may receive information from theremoting client 510 and adjust or modify display of an application 204 on thenative display 201 using the received information. Adjusting the display may include panning, zooming (in or out), scrolling or otherwise modifying the display of the application 204. - In some examples, the
remote computer 501 may execute one or more resources 505. These resources, in some examples, may be applications. In other examples, at least one resource may be an operating system executing on theremote computer 501. In those examples, the other applications 505 may execute within the context of the operating system. In some instances, the applications 505 may correspond to remote applications 505 from the perspective ofdevice 102 and one or more of applications 505 may be remoted tomobile device 102 for use by the user atmobile device 102. -
FIG. 6 illustrates ageneral method 600 for altering or modifying the display of a remoted application on a mobile device. For example, instep 602, an application executing on a computer remote from the mobile device may intercept a focus event notification generated in response to a control within a remoted application receiving focus. The remote computer may forward the received or intercepted focus event notification to a client device through which a user is viewing and interacting with the remoted application (e.g.,mobile computing device 102 ofFIG. 5 ) in step 605. Upon receiving the focus event notification, an application executing on the mobile computing device may modify a display of the remoted application in response to receiving the notification and according to information associated with the event (Step 610). -
FIG. 7 is a flowchart illustrating a method that may be performed by a computer executing an application for a remote client device (e.g., within the general process of method 600). Instep 700, for example, a remoting management application executing on the computer may intercept or otherwise receive a focus event notification. The remoting management application may, in some examples, be aremoting client 510 executing on aremote computer 501. In other examples, the application may be any application executing on aremote computer 501 and able to hook into function calls issued by an application executing on theremote computer 501. The remoting management application executing on theremote computer 501, in a particular example, may be a component inside WFSHELL.EXE (ICACTLS.DLL) or another service of Citrix XenApp or XenDesktop server. In other examples, the remoting management application may be an application managing a control or seamless virtual channel established between theremote computer 501 and themobile device 102. The focus event notification may be a call issued by a remote application executing on theremote computer 501 when a control or field within the remote application receives focus, e.g., a user selects or otherwise highlights the control or field throughmobile device 102. In some examples, the control may be any object or control or user interactive or input element, such as: an edit box, a textbox, a memo field, a combo box, a drop-down menu, a slider, a list box, or any other similar object or control. The function may, for example, be a function such as ‘focus combo box” or any other function that generates a notification indicating a control receiving focus. In some examples, the remoting management application may use event-based detection facilitated by WINDOWS 7 UI AUTOMATION application program interfaces. In other examples, the remoting management application may use event-based detection facilitated by any API. In these examples, the application may register for notifications provided by these APIs using a COM interface associated with a particular control and/or a particular function. - The remoting management application may send additional information to the
mobile device 102, e.g., separately or as part of the notification message, including: the type of control or interface element that received focus (e.g., combo box, textbox); the name or an identifier of the control that received focus; the location or coordinates of the control within the application (e.g., where in the application output the control is located); the content of the control (e.g., values assigned to the control, strings associated with the control, etc.); a window handle of the control; a window identifier of the window displaying the control; the location or coordinates of the parent window displaying the control; or any additional information that could be used to determine the location of the control within the application output displayed by themobile device 102. - In
step 705, theremote computer 501 may determine the type of object or control element receiving focus and/or a position thereof. For example, the computer may determine whether the control element is a text entry field, a menu of selectable values, a radio button or the like. The computer may also determine position information based on position data specified in the notification message, including the coordinates and size of the control element on which focus is placed. Instep 710, theremote computer 501 may then determine a general type of user control element to activate on the mobile device based on the determined type of control element of the remoted application receiving focus. For example, if the control element of the remoted application corresponds to a text entry field, the computer may determine that a software keyboard of type standard alphanumeric would be appropriate for interacting with the text entry field in a touch-input environment. In another example, if the control element of the remoted application corresponds to a number entry field (e.g., only numbers are permitted in the field), the computer may determine that a software keyboard of a number pad type may be most appropriate for interacting with the number entry field. - In
step 715, the computer may transmit a command to the mobile device to activate a native control element corresponding to the determined type of control element locally on the mobile device. The command may be transmitted through a virtual channel between the remoting application and a remoting client application executing on the mobile device. Additionally, in one or more examples, if the type of control element to be shown or activated on the client device is a software keyboard, the computer may transmit the command identifying the type of the control element and identification of an editable area or field for which the control element is to be activated. The identification of the editable area or field may include coordinates, size information and the like. In another example, if the type of control element to be shown or activated locally on the client device is a picker control (e.g., a menu or scroll selection bar), the computer may transmit the command along with an identifier of the picker control, an area or field definition (e.g., coordinates, size information), contents of the picker control (e.g., the selectable items), current selection index, a title of the picker control and/or window and parent window handles. - In some arrangements, the
remote computer 501 might not determine a mobile device-native control element or type thereof to activate or use. Instead, the user of the mobile device may manually select or activate a native control element at the mobile device. - In
step 720 the remote computer may receive an indication (e.g., a notification or confirmation) from the mobile device of the activation of a user control or input element native to the mobile device. The indication may include, for example, a confirmation of the state of the control element (e.g., show/hide), success or failure from execution of the command and/or an identifier of the local control element, e.g., mirroring the control identifier from the original command from the remote computer. - In some arrangements, the
remote computer 501 may receive a return operation status message from the client device (e.g., mobile device 102) instep 725, indicating an operation status of an activated local control element such as a picker control. For example, the return operation status message may include the identifier of the control, the operation status of the control and a selection index if the operation status indicates a selection made by the user. The selection index may identify the selected item. The operation status may include success (indicating that a selection has been made by the user) or cancel/abort (e.g., if the user has deactivated the control element without selection). - Additionally or alternatively, in
step 730, theremote computer 501 may further update the application display to reflect a selection made by a user (e.g., in a picker control element) in response to a notification from the client of the operation status and the selection. For example, the selection field may be updated with the selected value and the updated application display may be transmitted to the client device (e.g., mobile device 102) for display. The application display update may then be transmitted to the client device instep 735. - In
step 740, theremote computer 501 may receive another notification indicating that focus has left the control element and/or the remoted application. In response, theremote computer 501 may issue an instruction to the mobile device 102 (or a client application executing thereon) to return the application display to an original state and/or to deactivate the native control element in step 745 (e.g., if the application display was previously modified upon the remoted application receiving focus). In some arrangements, theremote computer 501 may issue the instruction only upon receiving notification that the native control element is no longer active (e.g., being displayed) or in use. -
FIG. 8 is a flowchart illustrating a process by which a device such as mobile device 102 (FIG. 5 ) may receive output data and instructions from a remote computing device executing a remoted application. Instep 800, the device (e.g., a client device) may execute a management application to provide interactivity with and receive data output from an application remotely executing on a server or other remote computing device. For example, the management application executing on themobile device 102 may modify the display of the remoted application on themobile device 102 in response to receiving instructions from the remote computing device (e.g., as described with respect toFIG. 7 ). In some examples, the management application may be awindow management service 206 executing on themobile device 102. In other examples, the management application may be any application able to receive event and control information from aremote computer 501 and use it to modify output displayed on thenative display 201 ofmobile device 102. Instep 805, the device may receive, from the remote computer executing the application, an instruction to activate a type of control element. In one or more examples, the instruction may be received from the remote computer in response to the remote computer detecting an editable or control field receiving focus (e.g., a user selecting, through the client device, a text entry field in a remoted application display). - In response to the instruction, the device may determine a native control (e.g., user input) element to activate based on the type of control element specified in the instruction in
step 810. For example, if the general type of control element corresponds to a standard alphanumeric keyboard, the device may identify a native control element matching the specified type such as an iOS standard alphanumeric keyboard (if it is an iOS mobile device), or an Android standard alphanumeric keyboard (if it is an Android mobile device). In some arrangements, the general types of controls supported by or available on the client device may be reported to the remote computer executing the application so that the remote computer may select an appropriate general type. Alternatively or additionally, the remote computer may process the focus event differently if a corresponding general type of control element is not supported by the client device, as is described in further detail herein. In some examples, the instructions may specify activation of multiple types of controls. Operation of the multiple native controls may be based on the use and specification of the GUID assigned to each native control. - Upon determining an appropriate control element corresponding to the specified type of control element, the device may activate or execute the determined native control in
step 815. In one example, activation and/or execution of the native control may include modifying the native display interface to display the control. Activation and/or execution of the native control might also include extracting control element values (e.g., values within a drop-down menu) and incorporating those values into the native control element (e.g., the virtual picker in the above example). For example, in some arrangements, the remote computer executing the remoted application may transmit control values included in the application control element receiving focus to the device for display in the native control element. Accordingly, upon receiving the event notification and control information, thewindow management service 206 at the mobile or client device may determine whether the control is a type of control having associated values. Upon determining that the control has associated values, thewindow management service 206 may modify the display of the application to include the received values. In one example, if one or more strings are associated with a combo box having received focus, thewindow management service 206 may modify the display to display a combo box that includes those strings. In other aspects, a client executing on the device, or awindow management service 206, may use local controls native to the device to display a remote control or to display graphics representative of a remote control. For example, when a remote Windows Combo Box is presented in an application, the device may display and/or use a local iPad picker control. Thus, the remote application displayed on the device may adopt the appearance of a device native application and may include at least some native functionality. - The device may further determine, in
step 820, whether the native control element will (or does) obscure the element of the remoted application receiving focus. For example, the device may determine an amount of the remoted application element being obscured (e.g., a percentage, absolute amount in pixels, inches or other metric), a particular area being obscured (e.g., obscuring a left part of a text entry field may be less important than obscuring a right part of the text entry field for language environments that read left to right), and the like. In some arrangements, the remoted application element might not be considered obscured if the amount obscured or portion obscured does not meet a predefined rule or threshold. For example, a remoted application element might only be considered obscured if at least 25%, at least 10%, at least 50%, etc. of the element is not visible. In another example, a remoted application might only be considered obscured if any portion of the left side of a text entry field (e.g., any part of the left 50% of the text entry field) is hidden from view. Other rules for determining when a control element is obscured may be defined and used. - If the device determines that the application control element will be or is obscured by the native control element, the device may determine, in
step 825, a manner in which to modify the local display such that at least the application control element is no longer obscured. For example, if a mobile device native keyboard is to be displayed at the bottom of the mobile device display and will obscure the application control element, the device may determine that the one or more portions of the local display (desktop interface or application specific window(s) or portions thereof) should be panned upward (e.g., auto-scroll). Additionally or alternatively, the device may modify the display (entire desktop, portions of the desktop and/or specific windows) to include a scrolling feature so that a remainder of the application display may be viewed once the display is panned upward if there is insufficient display real estate. In another example, the device may modify the application window to zoom out (e.g., shrink the application display) so that the application display fully fits within the mobile device display. - In
step 830, the device may subsequently modify the display in accordance with the determined manner to avoid obscuring the editable/control field. Alternatively, if the editable/control field receiving focus will not be obscured, the device might not modify the display. In some examples, steps 820-830 may be performed in conjunction with (e.g., before, during or after) activation of the native control element instep 815. - In some examples, the device may transmit a confirmation message to the remote computing device confirming activation of the native control element as shown in
step 835. Alternatively or additionally, the device may further transmit a return operation status message to the remote computing device instep 840, as described above with respect tosteps FIG. 7 . For example, status messages may be provided when the device detects a user selection of an item in the picker control. In another example, the device may transmit a status message upon detecting a user cancelling or aborting the picker control. These events may be reported to the remote computing device in the update message for appropriate handling by the remote computing device. If the update message includes an event that causes changes to the application display, for instance, those changes to the application display may be received from the remote computing device for updating the local application display. - The client device may further receive, in
step 845, a message indicating that focus has left the editable/control field and providing instructions to deactivate the corresponding native control element. In response, the client device may deactivate the native control element and return the application display to a previous state (e.g., prior to activation of the native control element) instep 850 if modification was performed. - The processes of
FIGS. 7 and 8 may further include transmitting actions performed on or through a native control element of the client device back to the remote computing device and remote application. For example, Application A executing on the remote computing device may generate application output which may be displayed in an application output window on the remote computing device. The application output generated by Application A is subsequently transmitted to the client device where it is displayed within an application output window thereon (e.g., a local application window). In some examples, when an editable combo box field within the remotely generated application output receives focus, a client or awindow management service 206 may intercept the on-focus event and may display or invoke a native, virtual picker control, e.g., an iPad picker control. Within the local application window and amongst the remotely generated application output, the native, virtual picker control is displayed. A user may interact with the native control to generate input. Once the control is used to generate input, the client orwindow management service 206 may transmit the input to the remote computing device for processing within the context of the remoted application. For example, upon receiving the input, theremote computer 501 may inject the control input into an actual control displayed within the actual application output generated by Application A. In a particular example, injection of the received control input may be accomplished using a Windows UI Automation API. Changes in the remoted application based on the received input may then be outputted back to the client device. This process may repeat until the remoted application is closed and/or the remoting relationship between the client device and the remote computing device is ended. - Additionally or alternatively, the remote computing device (executing the remoted application) may further be configured to determine whether the client device supports or has local control elements such as a native virtual keyboard or a native picker corresponding to a particular type or category of control element. For example, the client device's capabilities may be specified to the remote computing device and as such, the remote computing device may make the above noted determination. If the client device does not support or does not have a local control element corresponding to a determined type of control element corresponding to the editable/control field, the remote computing device may generate a virtual keyboard or other control element as part of the remoted application or separately from the remoted application and output the display of the virtual keyboard to the client device (similar to the remoting of the application). In one particular example, the virtual control element generated by the remote computing device may be executed and remoted as a separate application. In another example, the virtual control element may be included within and as part of the existing remoted application receiving focus.
- According to another aspect, the activation of a native control element may be switched on or off by the user or by a service providing the application remoting capability. Accordingly, if the user or the service does not wish to use native controls, the device might not automatically trigger a native control element upon receiving a focus event notification for an editable field. Additionally or alternatively, the remote computing device executing the application may be configured to determine whether an editable/control field will be obscured and to determine and provide instructions for modifying a local display of the client device in response. Accordingly, the remote computing device may transmit instructions for modifying the local display and/or the application display of the client device.
-
FIGS. 9A and 9B illustrate example interfaces displaying the use of native controls including a virtual keyboard. InFIG. 9A , for example, a native control of thedevice 900 such asvirtual keyboard 901 is activated and displayed for entry of input into a text entry field without modifying aremoted application display 903. As a result, the text entry field and other potentially important or relevant portions of theapplication display 903 become obscured by thevirtual keyboard 901. - In
FIG. 9B , to avoid obscuring thetext entry field 905, the application display may be panned upward. In some arrangements including the arrangement ofFIG. 9B , the display of an entire desktop including theremoted application display 903 may be modified. Panning may include moving theapplication display 903 in a particular direction (in this example, towards the top of the display) with or without modifying the proportions or orientation of theapplication display 903 or desktop. In some examples, panning may result in elements that were previously visible being moved off of the device display. For example, recyclebin icon 907 inFIG. 9A is moved off of the displayable area after theapplication display 903 is panned upward as shown inFIG. 9B . The amount or degree to which theapplication display 903 is modified (e.g., panned) may be determined based on various rules and thresholds. In a particular example, theapplication display 903 may be panned until at least 75% of the text entry field is visible. In another example, theapplication display 903 may be panned until an entirety of the text entry field is visible. -
FIGS. 10A and 10B illustrate other example interfaces displaying the use of native controls including a scrolling picker. InFIG. 10A , aweb browser application 1001 is displayed with content having various control elements including a drop downmenu 1003. The drop downmenu 1003 may include multiple selectable values that, in some arrangements, may be difficult to see because of font size, the number of values, size of the drop down portion of the menu and the like. - Accordingly, in
FIG. 10B , the interface may display a nativepicker control element 1005. Thenative picker element 1005 may be populated with the values included in drop downmenu 1003. The nativepicker control element 1005 may be displayed for easier viewing and accessibility (e.g., larger font size, easier selection mechanism, etc.). In the illustrated example ofFIG. 10B , the application display might not be modified. Because the values listed in the drop downmenu 1003 are also displayed in nativepicker control element 1005, obscuring part or all of the drop downmenu 1003 might not have as significant an impact in usability. In some examples, the application display may be modified to insure that the non-expanded drop down menu portion remains visible so that a user may confirm his or her selection once populated into the drop downmenu 1003. In yet other examples, the application display may zoom in to the drop downmenu 1003 or a portion thereof to increase viewability. - The methods and features recited herein may further be implemented through any number of computer readable media that are able to store computer readable instructions. Examples of computer readable media that may be used include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical disk storage, magnetic cassettes, magnetic tape, magnetic storage and the like.
- Moreover, while many of the aspects and features described herein include the use of a mobile device, stationary or fixed location devices may also include the same or similar aspects and features. For example, a personal computer or workstation (e.g., plugged into a wall socket) may include touch input controls and provide access to a remoted application from a remote computing device. Accordingly, similar native control elements may be generated and used from the personal computer or workstation in similar fashion to a mobile device such as a tablet computer.
- Additionally or alternatively, in at least some examples, the methods and features recited herein may be implemented through one or more integrated circuits (ICs). An integrated circuit may, for example, be a microprocessor that accesses programming instructions or other data stored in a read only memory (ROM). In some such examples, the ROM stores programming instructions that cause the IC to perform operations according to one or more of the methods described herein. In at least some other examples, one or more of the methods described herein are hardwired into an IC. In other words, the IC is in such cases an application specific integrated circuit (ASIC) having gates and other logic dedicated to the calculations and other operations described herein. In still other examples, the IC may perform some operations based on execution of programming instructions read from ROM or RAM, with other operations hardwired into gates and other logic of IC. Further, the IC may output image data to a display buffer.
- Although specific examples of carrying out various features have been described, those skilled in the art will appreciate that there are numerous variations and permutations of the above-described systems and methods that are contained within the spirit and scope of the disclosure as set forth in the appended claims. Additionally, numerous other examples, modifications and variations within the scope and spirit of the appended claims will occur to persons of ordinary skill in the art from a review of this disclosure.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/678,679 US20220179531A1 (en) | 2010-10-05 | 2022-02-23 | Display Management for Native User Experiences |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US38997510P | 2010-10-05 | 2010-10-05 | |
US13/253,414 US9400585B2 (en) | 2010-10-05 | 2011-10-05 | Display management for native user experiences |
US15/215,175 US10761692B2 (en) | 2010-10-05 | 2016-07-20 | Display management for native user experiences |
US16/929,339 US11281360B2 (en) | 2010-10-05 | 2020-07-15 | Display management for native user experiences |
US17/678,679 US20220179531A1 (en) | 2010-10-05 | 2022-02-23 | Display Management for Native User Experiences |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/929,339 Continuation US11281360B2 (en) | 2010-10-05 | 2020-07-15 | Display management for native user experiences |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220179531A1 true US20220179531A1 (en) | 2022-06-09 |
Family
ID=45890898
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/253,414 Active US9400585B2 (en) | 2010-10-05 | 2011-10-05 | Display management for native user experiences |
US15/215,175 Active 2032-11-19 US10761692B2 (en) | 2010-10-05 | 2016-07-20 | Display management for native user experiences |
US16/929,339 Active US11281360B2 (en) | 2010-10-05 | 2020-07-15 | Display management for native user experiences |
US17/678,679 Abandoned US20220179531A1 (en) | 2010-10-05 | 2022-02-23 | Display Management for Native User Experiences |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/253,414 Active US9400585B2 (en) | 2010-10-05 | 2011-10-05 | Display management for native user experiences |
US15/215,175 Active 2032-11-19 US10761692B2 (en) | 2010-10-05 | 2016-07-20 | Display management for native user experiences |
US16/929,339 Active US11281360B2 (en) | 2010-10-05 | 2020-07-15 | Display management for native user experiences |
Country Status (4)
Country | Link |
---|---|
US (4) | US9400585B2 (en) |
EP (1) | EP2625685B1 (en) |
CN (1) | CN103168325B (en) |
WO (1) | WO2012047997A1 (en) |
Families Citing this family (64)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130104062A1 (en) | 2011-09-27 | 2013-04-25 | Z124 | Unified desktop input segregation in an application manager |
GB2492789B (en) * | 2011-07-12 | 2018-01-03 | Denso Corp | Displays |
US9582187B2 (en) * | 2011-07-14 | 2017-02-28 | Microsoft Technology Licensing, Llc | Dynamic context based menus |
US10564791B2 (en) * | 2011-07-21 | 2020-02-18 | Nokia Technologies Oy | Method and apparatus for triggering a remote data entry interface |
US20130104051A1 (en) * | 2011-09-27 | 2013-04-25 | Z124 | Unified desktop big brother application pools |
US11416131B2 (en) | 2011-09-27 | 2022-08-16 | Z124 | Unified desktop input segregation in an application manager |
US20130080899A1 (en) | 2011-09-27 | 2013-03-28 | Paul E. Reeves | Unified desktop big brother applications |
US20130111391A1 (en) * | 2011-11-01 | 2013-05-02 | Microsoft Corporation | Adjusting content to avoid occlusion by a virtual input panel |
US9612724B2 (en) | 2011-11-29 | 2017-04-04 | Citrix Systems, Inc. | Integrating native user interface components on a mobile device |
US20130160095A1 (en) * | 2011-12-14 | 2013-06-20 | Nokia Corporation | Method and apparatus for presenting a challenge response input mechanism |
KR101921276B1 (en) * | 2012-02-03 | 2018-11-22 | 삼성전자주식회사 | Device and method for performing multi_tasking in wireless terminal |
US9794735B2 (en) | 2012-02-15 | 2017-10-17 | Dropbox Inc. | Context determination for mobile devices when accessing remote resources |
US9307009B2 (en) * | 2012-02-15 | 2016-04-05 | Mobilespan Inc. | Presenting execution of a remote application in a mobile device native format |
US9552130B2 (en) * | 2012-05-07 | 2017-01-24 | Citrix Systems, Inc. | Speech recognition support for remote applications and desktops |
US9075499B2 (en) * | 2012-08-24 | 2015-07-07 | Cellco Partnership | Method and apparatus for providing reference content |
US9201563B2 (en) * | 2012-10-10 | 2015-12-01 | Google Inc. | Mobile device friendly window management for remote desktop |
WO2014062872A1 (en) * | 2012-10-17 | 2014-04-24 | Avocent Huntsville Corp. | System and method for controlling display of virtual keyboard to avoid obscuring data entry fields |
US10095663B2 (en) | 2012-11-14 | 2018-10-09 | Amazon Technologies, Inc. | Delivery and display of page previews during page retrieval events |
KR101717179B1 (en) | 2013-01-07 | 2017-03-16 | 삼성전자주식회사 | Display apparatus, method for ui display thereof and computer-readable recording medium |
CN104823150B (en) * | 2013-02-27 | 2018-11-30 | 夏普株式会社 | information terminal and storage medium |
CN104102418B (en) * | 2013-04-03 | 2015-08-26 | 腾讯科技(深圳)有限公司 | Input frame target location localization method and device in a kind of browser of mobile terminal |
GB2512626B (en) * | 2013-04-04 | 2015-05-20 | Nds Ltd | Interface mechanism for massive resolution displays |
JP6115293B2 (en) * | 2013-05-02 | 2017-04-19 | 株式会社リコー | Apparatus, information processing terminal, information processing system, display control method, and program |
US10180728B2 (en) | 2013-05-17 | 2019-01-15 | Citrix Systems, Inc. | Remoting or localizing touch gestures at a virtualization client agent |
US10572124B2 (en) * | 2013-05-20 | 2020-02-25 | Citrix Systems, Inc. | Bound based contextual zoom |
TWI493433B (en) * | 2013-08-28 | 2015-07-21 | Acer Inc | Covered image projecting method and portable electronic apparatus using the same |
CN104461272A (en) * | 2013-09-18 | 2015-03-25 | 华为技术有限公司 | Virtual keyboard processing method and device |
US10621276B2 (en) * | 2013-10-24 | 2020-04-14 | Wmware, Inc. | User interface virtualization for web applications |
US10740540B2 (en) * | 2013-12-13 | 2020-08-11 | Freedom Scientific, Inc. | Techniques for programmatic magnification of visible content elements of markup language documents |
KR101619922B1 (en) * | 2013-12-24 | 2016-05-12 | 전자부품연구원 | Apparatus and Method for Providing Collaborative Service based on Virtual Resources |
US9922007B1 (en) | 2014-05-22 | 2018-03-20 | Amazon Technologies, Inc. | Split browser architecture capable of determining whether to combine or split content layers based on the encoding of content within each layer |
US11169666B1 (en) | 2014-05-22 | 2021-11-09 | Amazon Technologies, Inc. | Distributed content browsing system using transferred hardware-independent graphics commands |
US10042521B1 (en) | 2014-05-22 | 2018-08-07 | Amazon Technologies, Inc. | Emulation of control resources for use with converted content pages |
US9454515B1 (en) | 2014-06-17 | 2016-09-27 | Amazon Technologies, Inc. | Content browser system using graphics commands and native text intelligence |
CN104375825B (en) * | 2014-09-25 | 2018-06-08 | 福建升腾资讯有限公司 | Desktop window element tracks and identifies method and system under VDI environment |
CN104391707B (en) * | 2014-12-05 | 2018-01-30 | 上海斐讯数据通信技术有限公司 | The startup method and starter of a kind of application program |
KR102367184B1 (en) * | 2014-12-11 | 2022-02-25 | 삼성메디슨 주식회사 | Method and apparatus for inputting information by using a screen keyboard |
US20160179368A1 (en) * | 2014-12-19 | 2016-06-23 | Hand Held Products, Inc. | Intelligent small screen layout and pop-up keypads for screen-only devices |
US9804750B2 (en) * | 2015-01-21 | 2017-10-31 | Unisys Corporation | Textbox associated with device native features |
US9251372B1 (en) * | 2015-03-20 | 2016-02-02 | Yahoo! Inc. | Secure service for receiving sensitive information through nested iFrames |
US20160313958A1 (en) * | 2015-04-27 | 2016-10-27 | Microsoft Technology Licensing, Llc | Cross-platform command extensibility |
US10097902B2 (en) | 2015-05-06 | 2018-10-09 | Blackfire Research Corporation | System and method for using multiple audio input devices for synchronized and position-based audio |
US9848247B2 (en) * | 2016-04-27 | 2017-12-19 | Blackfire Research Corporation | Media computing device with multiple virtual screens and personal area network capability |
CN105872059B (en) * | 2016-03-31 | 2019-08-09 | 北京奇艺世纪科技有限公司 | A kind of remote execution method and device |
US10055187B2 (en) | 2016-04-27 | 2018-08-21 | Blackfire Research Corporation | System and method for multiuser interaction with a media computing device with multiple virtual screens and personal area network capability |
US10402210B2 (en) * | 2016-08-23 | 2019-09-03 | Sap Se | Optimizing user interface requests for backend processing |
CN106716357B (en) * | 2016-12-29 | 2019-11-01 | 深圳前海达闼云端智能科技有限公司 | Control method, control device and the electronic equipment of multisystem mobile terminal |
US11863622B2 (en) | 2017-02-07 | 2024-01-02 | Drumwave Inc. | Cross-device data distribution with modular architecture |
US10880272B2 (en) * | 2017-04-20 | 2020-12-29 | Wyse Technology L.L.C. | Secure software client |
TWI658401B (en) * | 2017-11-14 | 2019-05-01 | 財團法人工業技術研究院 | Mobile communication device based on virtual mobile infrastructure and related input method switching methods thereof |
US11379252B1 (en) * | 2018-01-31 | 2022-07-05 | Parallels International Gmbh | System and method for providing layouts for a remote desktop session |
US11243650B2 (en) | 2018-03-07 | 2022-02-08 | International Business Machines Corporation | Accessing window of remote desktop application |
CN109033772B (en) * | 2018-08-09 | 2020-04-21 | 北京云测信息技术有限公司 | Verification information input method and device |
US20200099760A1 (en) * | 2018-09-24 | 2020-03-26 | Salesforce.Com, Inc. | Interactive customized push notifications with customized actions |
US11113021B2 (en) * | 2018-11-06 | 2021-09-07 | Citrix Systems, Inc. | Systems and methods for SAAS application presentation mode on multiple displays |
US10860178B1 (en) * | 2019-09-05 | 2020-12-08 | Shabu Ans Kandamkulathy | Task management through soft keyboard applications |
US11625155B2 (en) * | 2020-03-23 | 2023-04-11 | Ricoh Company, Ltd. | Information processing system, user terminal, method of processing information |
CN116391170A (en) * | 2020-10-28 | 2023-07-04 | 三星电子株式会社 | Electronic device, display device and control method thereof |
CN112491974A (en) * | 2020-11-11 | 2021-03-12 | 恒安嘉新(北京)科技股份公司 | Remote browsing method and device, storage medium, terminal equipment and server |
CN113434103B (en) * | 2021-06-24 | 2022-04-05 | 华东师范大学 | Multi-screen interaction method and system based on screen virtualization and application thereof |
US11606456B1 (en) | 2021-10-19 | 2023-03-14 | Motorola Mobility Llc | Electronic devices and corresponding methods utilizing ultra-wideband communication signals for user interface enhancement |
US11503358B1 (en) | 2021-10-19 | 2022-11-15 | Motorola Mobility Llc | Electronic devices and corresponding methods utilizing ultra-wideband communication signals for user interface enhancement |
US11907495B2 (en) * | 2021-10-19 | 2024-02-20 | Motorola Mobility Llc | Electronic devices and corresponding methods utilizing ultra-wideband communication signals for user interface enhancement |
WO2023108475A1 (en) | 2021-12-15 | 2023-06-22 | Citrix Systems, Inc. | Application hotspot on endpoint device |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5872567A (en) * | 1996-03-29 | 1999-02-16 | International Business Machines Corporation | Method, memory and apparatus for automatically resizing a window in response to a loss or gain in focus |
US6677933B1 (en) * | 1999-11-15 | 2004-01-13 | Espial Group Inc. | Method and apparatus for operating a virtual keyboard |
US20060265662A1 (en) * | 2005-05-19 | 2006-11-23 | Custom Credit Systems, L.P. | System and method for generating and updating user interfaces of web-based applications |
US20100269152A1 (en) * | 2009-04-15 | 2010-10-21 | Wyse Technology Inc. | Method and system for rendering composite view of an application |
Family Cites Families (83)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE69523543T2 (en) * | 1994-12-13 | 2002-04-04 | Microsoft Corp | Taskbar with start menu |
US5940077A (en) * | 1996-03-29 | 1999-08-17 | International Business Machines Corporation | Method, memory and apparatus for automatically resizing a window while continuing to display information therein |
US6133914A (en) * | 1998-01-07 | 2000-10-17 | Rogers; David W. | Interactive graphical user interface |
US20060033724A1 (en) | 2004-07-30 | 2006-02-16 | Apple Computer, Inc. | Virtual input device placement on a touch screen user interface |
US6211856B1 (en) * | 1998-04-17 | 2001-04-03 | Sung M. Choi | Graphical user interface touch screen with an auto zoom feature |
US6260160B1 (en) * | 1998-06-15 | 2001-07-10 | Siemens Information And Communication Networks, Inc. | Remote troubleshooting of a computing device |
US6407749B1 (en) * | 1999-08-04 | 2002-06-18 | John H. Duke | Combined scroll and zoom method and apparatus |
JP3409752B2 (en) | 1999-10-26 | 2003-05-26 | トヨタ自動車株式会社 | Moving object detection device, abnormality alarm device and loading room opening control device |
US7210099B2 (en) * | 2000-06-12 | 2007-04-24 | Softview Llc | Resolution independent vector display of internet content |
KR20040041082A (en) * | 2000-07-24 | 2004-05-13 | 비브콤 인코포레이티드 | System and method for indexing, searching, identifying, and editing portions of electronic multimedia files |
US6563913B1 (en) * | 2000-08-21 | 2003-05-13 | Koninklijke Philips Electronics N.V. | Selective sending of portions of electronic content |
US6961945B2 (en) * | 2000-10-30 | 2005-11-01 | Microsoft Corporation | Method and apparatus for adapting and hosting legacy user interface controls |
US7036086B2 (en) | 2001-01-04 | 2006-04-25 | Intel Corporation | Displaying software keyboard images |
US20020129096A1 (en) * | 2001-02-14 | 2002-09-12 | Mansour Peter M. | Platform-independent distributed user interface client architecture |
US20020111995A1 (en) * | 2001-02-14 | 2002-08-15 | Mansour Peter M. | Platform-independent distributed user interface system architecture |
US20020130900A1 (en) | 2001-03-16 | 2002-09-19 | Tomorrowsoft Corporation | System for generating an interface for software applications in a client-server environment |
JP2004532430A (en) * | 2001-05-02 | 2004-10-21 | ビットストリーム インコーポレーティッド | Method, system, and program for generating and displaying sub-pixel optimized images and digital content including such images |
US8230359B2 (en) * | 2003-02-25 | 2012-07-24 | Microsoft Corporation | System and method that facilitates computer desktop use via scaling of displayed objects with shifts to the periphery |
US8225224B1 (en) * | 2003-02-25 | 2012-07-17 | Microsoft Corporation | Computer desktop use via scaling of displayed objects with shifts to the periphery |
US8555165B2 (en) | 2003-05-08 | 2013-10-08 | Hillcrest Laboratories, Inc. | Methods and systems for generating a zoomable graphical user interface |
US20040230915A1 (en) * | 2003-05-13 | 2004-11-18 | International Business Machines Corporation | System and method for improved distributed menu performance |
US7366777B2 (en) * | 2003-05-15 | 2008-04-29 | Sap Aktiengesellschaft | Web application router |
WO2005041029A2 (en) * | 2003-10-24 | 2005-05-06 | Matsushita Electric Industrial Co., Ltd. | Remote operation system, communication apparatus remote control system and document inspection apparatus |
NZ529640A (en) | 2003-11-19 | 2006-09-29 | Mattersmiths Holdings Ltd | Method of removing moisture from a lignocellulosic substrate such as wood |
US20060188096A1 (en) * | 2004-02-27 | 2006-08-24 | Aguilar Joseph G | Systems and methods for remotely controlling computer applications |
US20060092177A1 (en) * | 2004-10-30 | 2006-05-04 | Gabor Blasko | Input method and apparatus using tactile guidance and bi-directional segmented stroke |
US7536410B2 (en) * | 2005-04-22 | 2009-05-19 | Microsoft Corporation | Dynamic multi-dimensional scrolling |
US20070024646A1 (en) * | 2005-05-23 | 2007-02-01 | Kalle Saarinen | Portable electronic apparatus and associated method |
US20060288015A1 (en) | 2005-06-15 | 2006-12-21 | Schirripa Steven R | Electronic content classification |
CN101228570B (en) * | 2005-07-22 | 2010-05-19 | 马特·帕尔拉科夫 | System and method for a thumb-optimized touch-screen user interface |
US20070067738A1 (en) * | 2005-09-16 | 2007-03-22 | Microsoft Corporation | Extensible, filtered lists for mobile device user interface |
US7962552B2 (en) * | 2005-11-14 | 2011-06-14 | Red Hat, Inc. | Borrow and give back of windows |
KR100722372B1 (en) * | 2005-11-16 | 2007-05-28 | 주식회사 로직플랜트 | Internet mail client system for mobile devices using remote computer systems and method thereof |
WO2007075846A2 (en) * | 2005-12-19 | 2007-07-05 | Propero Ltd. | Method and system for providing virtualized application workspaces |
US7786975B2 (en) * | 2005-12-23 | 2010-08-31 | Apple Inc. | Continuous scrolling list with acceleration |
US7676761B2 (en) * | 2006-06-30 | 2010-03-09 | Microsoft Corporation | Window grouping |
US7761807B2 (en) * | 2007-01-31 | 2010-07-20 | Research In Motion Limited | Portable electronic device and method for displaying large format data files |
US7877701B2 (en) * | 2007-03-08 | 2011-01-25 | International Business Machines Corporation | In-context total document views for manipulating data |
US8381096B2 (en) * | 2007-03-15 | 2013-02-19 | Yahoo! Inc. | Managing list tailoring for a mobile device |
US8286086B2 (en) * | 2007-03-30 | 2012-10-09 | Yahoo! Inc. | On-widget data control |
KR20080097059A (en) * | 2007-04-30 | 2008-11-04 | 삼성전자주식회사 | Methods for managing user contents of communication terminal |
TWI397899B (en) * | 2007-04-30 | 2013-06-01 | Mstar Semiconductor Inc | Display controller for displaying multiple windows and method for the same |
US8407605B2 (en) * | 2009-04-03 | 2013-03-26 | Social Communications Company | Application sharing |
US20090172557A1 (en) * | 2008-01-02 | 2009-07-02 | International Business Machines Corporation | Gui screen sharing between real pcs in the real world and virtual pcs in the virtual world |
US9100716B2 (en) * | 2008-01-07 | 2015-08-04 | Hillcrest Laboratories, Inc. | Augmenting client-server architectures and methods with personal computers to support media applications |
CN101482788A (en) * | 2008-01-08 | 2009-07-15 | 宏达国际电子股份有限公司 | Method for editing files by touch control keyboard, hand-hold electronic device and storage media |
US8473276B2 (en) * | 2008-02-19 | 2013-06-25 | Google Inc. | Universal language input |
JP5368547B2 (en) * | 2008-04-05 | 2013-12-18 | ソーシャル・コミュニケーションズ・カンパニー | Shared virtual area communication environment based apparatus and method |
CN102027450B (en) * | 2008-05-20 | 2015-05-13 | 思杰系统有限公司 | Methods and systems for using external display devices with a mobile computing device |
US9191238B2 (en) * | 2008-07-23 | 2015-11-17 | Yahoo! Inc. | Virtual notes in a reality overlay |
US8543935B2 (en) * | 2008-08-20 | 2013-09-24 | Red Hat, Inc. | Full-screen heterogeneous desktop display and control |
WO2010032079A2 (en) * | 2008-09-17 | 2010-03-25 | Nokia Corp. | User interface for augmented reality |
KR101609162B1 (en) * | 2008-11-13 | 2016-04-05 | 엘지전자 주식회사 | Mobile Terminal With Touch Screen And Method Of Processing Data Using Same |
US20100153866A1 (en) | 2008-12-11 | 2010-06-17 | Accenture Global Services Gmbh | Method and system for modifying the execution of a native application running on a portable electronic device |
US8159465B2 (en) * | 2008-12-19 | 2012-04-17 | Verizon Patent And Licensing Inc. | Zooming techniques for touch screens |
EP2199818B1 (en) * | 2008-12-19 | 2019-02-20 | Koninklijke KPN N.V. | Method of determining a location of a mobile device |
US9917702B2 (en) * | 2009-04-08 | 2018-03-13 | Blackberry Limited | System and method for managing items in a list shared by a group of mobile devices |
US8254890B2 (en) * | 2009-04-08 | 2012-08-28 | Research In Motion Limited | System and method for managing items in a list shared by a group of mobile devices |
US8370762B2 (en) * | 2009-04-10 | 2013-02-05 | Cellco Partnership | Mobile functional icon use in operational area in touch panel devices |
US9448815B2 (en) * | 2009-04-15 | 2016-09-20 | Wyse Technology L.L.C. | Server-side computing from a remote client device |
US20100268762A1 (en) * | 2009-04-15 | 2010-10-21 | Wyse Technology Inc. | System and method for scrolling a remote application |
US9241062B2 (en) * | 2009-05-20 | 2016-01-19 | Citrix Systems, Inc. | Methods and systems for using external display devices with a mobile computing device |
US9298336B2 (en) * | 2009-05-28 | 2016-03-29 | Apple Inc. | Rotation smoothing of a user interface |
JP2011048538A (en) * | 2009-08-26 | 2011-03-10 | Access Co Ltd | Context arrangement method, context arrangement program, and information display device |
KR20110037298A (en) * | 2009-10-06 | 2011-04-13 | 삼성전자주식회사 | Edit method of list and portable device using the same |
US9268466B2 (en) * | 2009-12-09 | 2016-02-23 | Citrix Systems, Inc. | Methods and systems for updating a dock with a user interface element representative of a remote application |
US8736561B2 (en) * | 2010-01-06 | 2014-05-27 | Apple Inc. | Device, method, and graphical user interface with content display modes and display rotation heuristics |
US8694902B2 (en) * | 2010-01-06 | 2014-04-08 | Apple Inc. | Device, method, and graphical user interface for modifying a multi-column application |
US9052894B2 (en) * | 2010-01-15 | 2015-06-09 | Apple Inc. | API to replace a keyboard with custom controls |
US20110175826A1 (en) * | 2010-01-15 | 2011-07-21 | Bradford Allen Moore | Automatically Displaying and Hiding an On-screen Keyboard |
KR101690786B1 (en) * | 2010-02-12 | 2016-12-28 | 삼성전자주식회사 | Device and method for performing multi-tasking |
US9417787B2 (en) * | 2010-02-12 | 2016-08-16 | Microsoft Technology Licensing, Llc | Distortion effects to indicate location in a movable data collection |
US8595645B2 (en) * | 2010-03-11 | 2013-11-26 | Apple Inc. | Device, method, and graphical user interface for marquee scrolling within a display area |
WO2011123840A2 (en) * | 2010-04-01 | 2011-10-06 | Citrix Systems, Inc. | Interacting with remote applications displayed within a virtual desktop of a tablet computing device |
US20110252364A1 (en) * | 2010-04-07 | 2011-10-13 | Freddy Allen Anzures | Device, Method, and Graphical User Interface for Navigation of Multiple Applications |
US8683377B2 (en) * | 2010-05-12 | 2014-03-25 | Adobe Systems Incorporated | Method for dynamically modifying zoom level to facilitate navigation on a graphical user interface |
US9542091B2 (en) * | 2010-06-04 | 2017-01-10 | Apple Inc. | Device, method, and graphical user interface for navigating through a user interface using a dynamic object selection indicator |
US20110319056A1 (en) * | 2010-06-29 | 2011-12-29 | Enterproid Hk Ltd | Remote access to a mobile device |
US9483175B2 (en) * | 2010-07-26 | 2016-11-01 | Apple Inc. | Device, method, and graphical user interface for navigating through a hierarchy |
US8700545B2 (en) * | 2010-08-27 | 2014-04-15 | Google Inc. | Sorted inbox with important message identification based on global and user models |
US8589800B2 (en) * | 2010-12-10 | 2013-11-19 | Wyse Technology Inc. | Methods and systems for accessing and controlling a remote desktop of a remote machine in real time by a web browser at a client device via HTTP API utilizing a transcoding server |
US20130111391A1 (en) * | 2011-11-01 | 2013-05-02 | Microsoft Corporation | Adjusting content to avoid occlusion by a virtual input panel |
US20140245205A1 (en) * | 2013-02-27 | 2014-08-28 | Microsoft Corporation | Keyboard navigation of user interface |
-
2011
- 2011-10-05 US US13/253,414 patent/US9400585B2/en active Active
- 2011-10-05 CN CN201180048406.1A patent/CN103168325B/en active Active
- 2011-10-05 WO PCT/US2011/054922 patent/WO2012047997A1/en active Application Filing
- 2011-10-05 EP EP11831515.9A patent/EP2625685B1/en active Active
-
2016
- 2016-07-20 US US15/215,175 patent/US10761692B2/en active Active
-
2020
- 2020-07-15 US US16/929,339 patent/US11281360B2/en active Active
-
2022
- 2022-02-23 US US17/678,679 patent/US20220179531A1/en not_active Abandoned
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5872567A (en) * | 1996-03-29 | 1999-02-16 | International Business Machines Corporation | Method, memory and apparatus for automatically resizing a window in response to a loss or gain in focus |
US6677933B1 (en) * | 1999-11-15 | 2004-01-13 | Espial Group Inc. | Method and apparatus for operating a virtual keyboard |
US20060265662A1 (en) * | 2005-05-19 | 2006-11-23 | Custom Credit Systems, L.P. | System and method for generating and updating user interfaces of web-based applications |
US20100269152A1 (en) * | 2009-04-15 | 2010-10-21 | Wyse Technology Inc. | Method and system for rendering composite view of an application |
Also Published As
Publication number | Publication date |
---|---|
US20120084663A1 (en) | 2012-04-05 |
WO2012047997A1 (en) | 2012-04-12 |
US9400585B2 (en) | 2016-07-26 |
EP2625685A4 (en) | 2016-10-05 |
US20160328107A1 (en) | 2016-11-10 |
US10761692B2 (en) | 2020-09-01 |
EP2625685B1 (en) | 2020-04-22 |
EP2625685A1 (en) | 2013-08-14 |
US20200348810A1 (en) | 2020-11-05 |
US11281360B2 (en) | 2022-03-22 |
CN103168325B (en) | 2017-06-30 |
CN103168325A (en) | 2013-06-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11281360B2 (en) | Display management for native user experiences | |
US11494010B2 (en) | Touch support for remoted applications | |
US10244039B2 (en) | Shell integration on a mobile device for an application executing remotely on a server | |
US9740507B2 (en) | Interacting with remote applications displayed within a virtual desktop of a tablet computing device | |
US9467346B2 (en) | Interpreting a gesture-based instruction to selectively display a frame of an application user interface on a mobile computing device | |
US10200453B2 (en) | Reverse seamless integration between local and remote computing environments | |
US9588637B2 (en) | Transparent user interface integration between local and remote computing environments | |
US9270678B2 (en) | Mediating resource access based on a physical location of a mobile device | |
US9241062B2 (en) | Methods and systems for using external display devices with a mobile computing device | |
US20100138780A1 (en) | Methods and systems for using external display devices with a mobile computing device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: CITRIX SYSTEMS, INC., FLORIDA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MOMCHILOV, GEORGY;VALICHERLA, CHAKRAVARTHI;INGALE, MUKUND;AND OTHERS;REEL/FRAME:059080/0249 Effective date: 20111005 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, DELAWARE Free format text: SECURITY INTEREST;ASSIGNOR:CITRIX SYSTEMS, INC.;REEL/FRAME:062079/0001 Effective date: 20220930 |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT, DELAWARE Free format text: PATENT SECURITY AGREEMENT;ASSIGNORS:TIBCO SOFTWARE INC.;CITRIX SYSTEMS, INC.;REEL/FRAME:062113/0470 Effective date: 20220930 Owner name: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT, NEW YORK Free format text: SECOND LIEN PATENT SECURITY AGREEMENT;ASSIGNORS:TIBCO SOFTWARE INC.;CITRIX SYSTEMS, INC.;REEL/FRAME:062113/0001 Effective date: 20220930 Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH CAROLINA Free format text: PATENT SECURITY AGREEMENT;ASSIGNORS:TIBCO SOFTWARE INC.;CITRIX SYSTEMS, INC.;REEL/FRAME:062112/0262 Effective date: 20220930 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
AS | Assignment |
Owner name: CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.), FLORIDA Free format text: RELEASE AND REASSIGNMENT OF SECURITY INTEREST IN PATENT (REEL/FRAME 062113/0001);ASSIGNOR:GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT;REEL/FRAME:063339/0525 Effective date: 20230410 Owner name: CITRIX SYSTEMS, INC., FLORIDA Free format text: RELEASE AND REASSIGNMENT OF SECURITY INTEREST IN PATENT (REEL/FRAME 062113/0001);ASSIGNOR:GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT;REEL/FRAME:063339/0525 Effective date: 20230410 Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT, DELAWARE Free format text: PATENT SECURITY AGREEMENT;ASSIGNORS:CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.);CITRIX SYSTEMS, INC.;REEL/FRAME:063340/0164 Effective date: 20230410 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |