WO2015058100A1 - Accélération graphique destinée à des applications s'exécutant sur des dispositifs mobiles avec un environnement à systèmes d'exploitation multiples - Google Patents

Accélération graphique destinée à des applications s'exécutant sur des dispositifs mobiles avec un environnement à systèmes d'exploitation multiples Download PDF

Info

Publication number
WO2015058100A1
WO2015058100A1 PCT/US2014/061166 US2014061166W WO2015058100A1 WO 2015058100 A1 WO2015058100 A1 WO 2015058100A1 US 2014061166 W US2014061166 W US 2014061166W WO 2015058100 A1 WO2015058100 A1 WO 2015058100A1
Authority
WO
WIPO (PCT)
Prior art keywords
hosted
native
graphics
application
operating system
Prior art date
Application number
PCT/US2014/061166
Other languages
English (en)
Inventor
Jaap VERMEULEN
Original Assignee
OpenMobile World Wide, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/061,288 external-priority patent/US20140115606A1/en
Application filed by OpenMobile World Wide, Inc. filed Critical OpenMobile World Wide, Inc.
Publication of WO2015058100A1 publication Critical patent/WO2015058100A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines

Definitions

  • Acer's Aspire One supported dual boot modes: one for Windows OS and one for Android. However, the device could not run apps for both operating systems in a single mode.
  • an object of the invention is to provide improved systems and methods for digital data processing.
  • Another, more particular, object is to provide such systems and methods as support executing on a single hardware/software platform applications ("apps") made for execution on multiple different hardware/software platforms.
  • the foregoing are among the objects attained by the invention, which provides in some aspects a computing device that includes a central processing unit, a graphics processing unit, and a display, all coupled (directly or indirectly for communications).
  • the central processing unit executes a native operating system including one or more native runtime environments within which native software applications are executing, where each such native software application has instructions for execution under the native operating system.
  • a first native software application (“ACL") executing within one or more of the native runtime environments defines one or more hosted runtime environments within which hosted software applications are executing.
  • the first native software application executes code comprising those hosted runtime environment or portions thereof (such as virtual machines); whereas, in other practices, the first native software application can, instead or in addition, effect the installation, instantiation and/or invocation of services/processes that make up those environments or portions thereof (but, indeed, may not in some practices execute code comprising that environment).
  • the hybrid application that is executed in the single application address space additionally includes instructions from at least one of a runtime library and another resource of the hosted operating system.
  • Yet still further aspects of the invention provide a computing device, e.g., as described above, in which the hybrid application that is executed in the single application address space additionally includes instructions adapted from at least one of a runtime library and another resource of the hosted and/or native operating systems.
  • FIGS. 1A-1C depict a computing device of the type embodying the invention
  • Figure 3 depicts one or more hosted runtime environments defined by a native software application for execution of hosted software applications in the device of Figure 1;
  • Figure 8 depicts interaction of the notification subsystems of the hosted runtime environments and native runtime environments in a system according to the invention
  • the native operating system 14 defines one or more native runtime environments 16 of the type known in the art (as adapted in accord with the teachings hereof) within which native software applications of the type known in the art (as adapted in accord with the teachings hereof)— i.e., applications having instructions for execution under the native operating system— are executing.
  • Such applications are labeled 15, 18 and 46-52 in the drawing.
  • the terms "application” and “app” are used interchangeably.
  • Kernel 24 serves inter alia as an interface, in the conventional manner known in the art has adapted in accord with the teachings hereof, between CPU 12 (and, more typically, the native applications executing within the native runtime environment 16 executing thereon) and hardware devices 24-30 integral or attached to device 10.
  • the display/touch screen 24, the frame buffer 26, and other integral/peripheral devices supporting interactions between the device 10 and its user are referred to as a "hardware interface,” regardless of whether they comprise hardware, software or (as is more typically the case) a combination thereof.
  • Each such hosted software application has instructions for execution under a hosted operating system that differs from the native operating system.
  • FIG. 3 depicts one or more hosted runtime environments 32 defined by the native software application 18 (or ACL) for execution of hosted software applications 34, 36 in the device 10 according to some practices of the invention.
  • the illustrated hosted runtime environment 32 is of the type known in the art (as adapted in accord with the teachings hereof) within which software applications having instructions for execution under the hosted operating system (i.e., hosted software applications) are built and intended to be executed.
  • the hosted runtime environment(s) 32 may comprise one or more virtual machines or otherwise, as is conventional in the art (as adapted in accord with the teachings hereof), depending on the type of the hosted operating system and the specifics of its implementation within the runtime environments 32.
  • Illustrated hosted runtime environment 32 is intended for executing Android-based software applications 34, 36 (though, other embodiments may be intended for executing applications designed and built for other operating systems) and includes, by way of non-limiting example, a resource framework 38, virtual machines (VMs) 40, event handler 42 and run-time libraries 44, all by way of non-limiting example and all of the type known in the art, as adapted in accord with the teachings hereof.
  • VMs virtual machines
  • the illustrated runtime environment 32 does not include a kernel per se (as might normally be included, for example, in the runtime environment of a Linux-/Android-based operating system) in the sense of running operations in a protected, kernel space of the type known in the art. Instead, some such operations (e.g., operations that might normally be included, for example, in the kernel of a Linux-/Android-based operating system) are executed in user space.
  • a kernel per se as might normally be included, for example, in the runtime environment of a Linux-/Android-based operating system
  • some such operations are executed in user space.
  • kernel space operations relied upon by the resource framework 38, virtual machines (VMs) 36, event handler 42, run-time libraries 44, and/or other components of the runtime environment 32 to load graphics to a frame buffer for presentation on a display.
  • VMs virtual machines
  • run-time libraries 44 run-time libraries 44
  • other components of the runtime environment 32 to load graphics to a frame buffer for presentation on a display.
  • those operations are elevated to user space and are employed to load such graphics to a "virtual" frame buffer 54, which (as discussed below) is shared with the native runtime environment 16 and the applications executing there — particularly, the I/O proxy applications 50, 52.
  • kernel-space operations can be avoided by passing-off to native operating system 14 and its runtime environment 16 operations and, more broadly, functions required for execution of hosted software applications 34, 36 that would otherwise be performed within the runtime environment 32 and, specifically, for example by a kernel thereof.
  • Such passing-off is effected, for example, by the resource framework 38, virtual machines (VMs) 36, event handler 42, run-time libraries 44, and/or other components of the runtime environment 32, which communicate with and/or otherwise rely on the native software application proxies 46-52 (executing in runtime environment 16) of hosted software applications 34, 36 to perform such functions or alternates thereof.
  • VMs virtual machines
  • event handler 42 event handler 42
  • run-time libraries 44 run-time libraries 44
  • other components of the runtime environment 32 which communicate with and/or otherwise rely on the native software application proxies 46-52 (executing in runtime environment 16) of hosted software applications 34, 36 to perform such functions or alternates thereof.
  • Native software applications e.g., 15 and 18, are installed (upon direction of the user or otherwise) on device 10 and, more particularly, for execution within native runtime environments 16, in the conventional manner of the art for installations of apps within operating systems of the type of operating system 14.
  • Such installation typically involves cooperative action of native operating system 14 and the runtime environments 16 executing an "installer" app (not shown) of the type conventional to OS 14 and typically includes unpacking, from an application package file (e.g., downloaded from a developer site or otherwise), the to-be-installed application's executable file, icon file, other support files, etc., and storing those to designated locations in static storage (MEM) on device 10, again, in the conventional manner known in the art, as adapted in accord with the teachings hereof.
  • application package files are referred to herein as "native" application package files.
  • Hosted software applications 34, 36 are installed (upon direction of the user or otherwise) under control of ACL 18 for execution under hosted runtime environments 32.
  • the ACL 18 can utilize an installer app the type conventional to the hosted operating system, albeit, modified as discussed herein, e.g., to unpack from the application package files, or otherwise, the to-be-installed application's executable file, icon file, other support files, etc., to suitable locations in static storage (MEM) on device 10, e.g., locations dictated by native operating system 14, yet, consistent with the hosted operating system, or otherwise.
  • Such application package files are referred to herein as "hosted" application package files.
  • the native software applications 46-52 if any, that are proxies of a hosted software application 34, 36 are installed, by request from ACL 18 to native operating system 14, in connection with the installation by ACL 18 of each respective hosted software application.
  • Each such proxy 46-52 is installed by the native operating system 14 in the conventional manner, albeit, from application package files (or otherwise) generated by ACL's 18 proxy installer interface 62, which triggers installation of those proxies.
  • Those package files can include, in lieu of the respective hosted software application 34, 36 executable, a "stub" executable suitable for
  • Those package files can also include icon files that are identical to or variants of those originally supplied with the application package files (or otherwise) for the respective hosted software applications 34, 36.
  • icon files that are identical to or variants of those originally supplied with the application package files (or otherwise) for the respective hosted software applications 34, 36.
  • two proxies may be associated with each hosted software application, only a single icon is associated with both proxies as displayed on the graphical desktop, e.g., of Figure 1A.
  • the analogy can also be said to break down in embodiments where the instruction set utilized by the hosted application 34 is suitable for execution on the CPU of device 10 (or, put another way, where the native and hosted operating systems are both targeted to the same CPU, i.e., that provided by device 10).
  • execution of the the hosted software application 34 instructions can be carried out directly by the CPU of device (and not, for example, merely emulated by native software application 18)— though, the handling of interrupts generated by and/or calls made in the course of such execution may be handled by the hosted runtime environments 32 (whether, themselves, executed by application 18 or otherwise).
  • native software application 18 when native software application 18 is installed on device 10 under native operating system 14, the application 18 itself, its installation package, or other functionality (e.g., the native operating system 14) concurrently installs, instantiates and invokes daemons 33 on device 10, e.g., for execution as persistent background processes that auto-load with each reboot of device 10 and/or operating system 14.
  • native software application 18 effects installation, instantiation and invocation of such persistent/auto-loading daemons 33 when the application 18 is executed for a first time by the user of device 10.
  • application 18 installs, instantiates and/or invokes the daemons 33 on a onetime or short-term basis, persisting those daemons for only so long as application 18 is itself executing on device 10. Still other embodiments utilize other mechanisms for installing, instantiating and/or invoking daemons 33, e.g., under control of application 18 or otherwise.
  • daemons 33 are shown in the drawing, in some embodiments other numbers of daemons (for example, just one) may be utilized. And, although, the daemons may be allocated on a per service basis in the illustrated embodiment, in other embodiments they may be allocated on a per hosted application-basis, a per proxy basis, or otherwise.
  • application 18 takes a mix of the approaches discussed above, e.g., executing code that makes up some portions of the environments 32 (e.g., like shown in Figures 2-3, while installing, instantiating and/or invoking services/ pro cess to provide services making up other portions of those environments (e.g., like shown in Figures 13-14 of aforementioned incorporated-by-reference application United States Patent Application Serial No. 61/983,698, filed April 24, 2014, entitled "HOSTED APP INTEGRATION SERVICES IN MULTI -OPERATING SYSTEM MOBILE AND OTHER COMPUTING DEVICES").
  • the native operating system 14 drives the computing device to display, on display/touch screen 24, a graphical desktop with icons 58 representing applications that can be selected for launch or other activation by the user of the device 10.
  • these can be native software applications, e.g., 15, and hosted software applications, e.g., 34, 36.
  • That desktop display includes a status bar 56 of the type conventional in the art— and, particularly, conventional to native operating system 14 (although, some embodiments may vary in this regard).
  • that status bar 56 indicates the current date/time, carrier conductivity signal strength (e.g., Wi-Fi, cellular, etc.), active apps, and so forth, though, in other embodiments, it may indicate other things.
  • carrier conductivity signal strength e.g., Wi-Fi, cellular, etc.
  • active apps e.g., it may indicate other things.
  • FIG. IB when a native software application, e.g.
  • the operating system 14 and/or runtime environments 16 do not overwrite the status bar 56 with the applications window 60. (Of course, it will be appreciated that this is the default mode of operation of the operating system 14 and/or runtime environments 16, and that in other modes, e.g., so called "full screen” modes, the application window 60 may occupy the entirety of the screen).
  • the application window generated for it (reflecting execution in the hosted runtime environments 32) is presented identically on the screen 24 as that of a native software application—that is, it is presented without overwriting the status bar 56 (e.g., at least when displaying in default mode).
  • Another example of the illustrated computing device's 10 merging the user experience so that applications executed in the hosted runtime environment appear, to the user, as if they are executing within the native operating system 14 is the use of a common notification mechanism, e.g., that of the native operating system 14 and/or runtime environments 16, as discussed in the incorporated-by-reference applications identified below.
  • Still another example is the consistent activation of running software applications in response to user replies to notifications (and otherwise), whether they are native applications, e.g., 15, or hosted software applications 34, 36, Again, as identified by the incorporated-by-reference applications identified below.
  • Yet still another example is the use of consistent theming as between the hosted software applications and native software applications, as discussed above. Still other examples will be evident to those skilled in the art from the discussion that follows and otherwise.
  • FIG 4 depicts the interaction of the components discussed above in launching an exemplary hosted software application 34 (here, labelled "App 1”) in hosted runtime environments 32 based on user interaction with that app's launch proxy 46 (here, labelled "App #1 Launch Stub”) executing in native runtime environments 16, displaying an application window representing operation of hosted software application 34 via that app's 10 proxy 50 (here, labelled "App #1 10 Stub”), and transmitting user input from that proxy 50 back to the app 34.
  • App 1 exemplary hosted software application 34
  • App #1 Launch Stub executing in native runtime environments 16
  • an application window representing operation of hosted software application 34 via that app's 10 proxy 50 here, labelled "App #1 10 Stub”
  • the native software application that is launch proxy 46 is launched by native runtime environments 16 and/or native operating system 14 upon its selection for activation by the user. See, step 64.
  • Proxy 50 can be simultaneously launched by native runtime environments 16 and/or native operating system 14; alternatively, proxy 50 can be launched by proxy 46 upon its launch. Id.
  • proxy 46 Upon launch (or other notification of activation from native runtime environments 16 and/or native operating system 14), proxy 46 effects activation of corresponding hosted software application 34. See, step 66. I n the illustrated embodiment, proxy 46 does this by transmitting a launch message to the event handler 42 that forms part of the hosted runtime environments 32 and that is common to the one or more hosted software applications 34, 36 (e.g., in that it is the common, shared recipient of system level-events, such as user input to the hardware interface, which events it distributes to appropriate hosted applications or other software executing in the hosted runtime environments 32 or provided as part of the hosted operating system).
  • the event handler 42 that forms part of the hosted runtime environments 32 and that is common to the one or more hosted software applications 34, 36 (e.g., in that it is the common, shared recipient of system level-events, such as user input to the hardware interface, which events it distributes to appropriate hosted applications or other software executing in the hosted runtime environments 32 or provided as part of the hosted operating system).
  • the launch message which can be delivered to event handler 42 by proxy 46 using any convention mechanism for inter process communication (I PC), e.g., APIs, mailboxes, etc., includes an identifier of the proxy 46 and/or its corresponding hosted software application 34, as well as any other information required by the hosted operating system and/or hosted runtime environments 32 to effect launch of a hosted software application.
  • I PC inter process communication
  • the event handler 42 launches the hosted software application 34 in the conventional manner required of hosted operating system and/or the hosted runtime environments 32. Put more simply, that app 34 is launched as if it had been selected by the user of device 10 directly.
  • event handler 42 uses IPC, e.g., as described above, to signal that hosted software application 34 has begun execution and, more aptly, to insure launch (if not already effected) and activation of proxy application 50 with the native runtime environments 16. See, step 70.
  • hosted software application 34 runs in the conventional manner within hosted runtime environments 32— e.g., generating interrupts and making such calls to the hosted resource framework 38, hosted event handler 42 and run-time libraries 44, all by way of non-limiting example— as it would otherwise make if it were installed on a device executing a single operating system of the type of the hosted operating system.
  • This is advantageous in that it does not require special recoding (i.e., "porting") of the hosted software application 34 by the developer or publisher thereof in order to make it possible to run in the multi-operating system environment of device 10.
  • Hosted resource framework 38, hosted event handler 42, run-time libraries 44, and the other components of hosted runtime environments 32 respond to such interrupts and calls in the conventional manner known in the art of operating systems of the type of hosted operating system, all as adapted in accord with the teachings hereof.
  • some such operations e.g., those for loading frame buffers
  • hosted runtime environments 32 are, instead, executed in user space.
  • other such operations are passed-off to native operating system 14 and its runtime environment 16, e.g., via the proxies 46-52.
  • the hosted runtime environment 32 loads the virtual frame buffer 54 with such graphics. See, step 72.
  • the hosted runtime environment 32 effects this through use of windowing and graphics subsystems that form part of the hosted runtime environment 32 and that is common to the one or more hosted software applications 34, 36 (e.g., in that it is the common, shared system used by the hosted software applications for generating applications windows for display to the user of device 10.)
  • the 10 proxy 50 of hosted software application 34 effects presentation on screen 24 of the applications windows generated for application 34 by hosted runtime environments 32, e.g., in the manner shown in Figure 1C and discussed in connection therewith above. See, step 74. 10 proxy 50 does this by transferring the graphics defining that applications window from virtual frame buffer 54 to the native frame buffer 26, e.g., using an API provided by native runtime environments 16 for such purpose or otherwise.
  • the hosted runtime environments 32 utilizes messaging to alert 10 proxy 50 of the need for effecting such a transfer, e.g., when the window subsystem of hosted runtime environments 32 has generated an updated applications window for hosted software application 34, when hosted software application 34 becomes the active (or foreground) app in hosted runtime environments 32, or otherwise, in other embodiments 10 proxy 50 effects such transfers on its own accord on a periodic basis or otherwise.
  • 10 proxy 50 effects such transfers on its own accord on a periodic basis or otherwise.
  • 10 proxy 50 utilizes a mechanism paralleling that discussed above in connection with steps 64-68 in order to transmit taps and other input made by the user to device 10 and specifically, for example, to display/touch screen 24, a keyboard, trackball, touch stick, other user input devices.
  • a common event handler (not shown) or other functionality of native runtime environments 16 notifies applications executing within them, including the 10 proxies 50, 52, of user input made with respect to them via the touch screen 24 or those other input devices.
  • Such notifications are made in the conventional manner known in the art of operating systems of the type of native operating system 14, as adapted in accord with the teachings hereof.
  • 10 proxy 50 When 10 proxy 50 receives such a notification, it transmits information with respect thereto to its corresponding hosted software application 34 via event handler 42, e.g., in a manner similar to that discussed above in connection with step 66. See, step 76.
  • That information which can be delivered to event handler 42 by 10 proxy 50 using any conventional I PC mechanism, can include an identifier of the 10 proxy 50 and/or its corresponding hosted software application 34, an identifier of the device to which input was made, the type of input, and relevant information with respect thereto (e.g., location, time, duration and type of touch, key tapped, pressure on pointer, etc. ).
  • That information is received by event handler 42 and applied to the corresponding hosted software application 34 in the conventional manner required of hosted operating system and/or the hosted runtime environments 32, e.g., as if the touch or other user input had been made directly to hosted software application 34. See, step 78.
  • the respective hosted software applications utilize their corresponding proxies (e.g., 46) to perform the following, by way of nonlimiting example: • present (via operation of native operating system 14) icons on the native operating system 14 graphical desktop of display/touch screen 24 for selection by the user;
  • the hosted software applications can similarly use proxies executing in the native runtime environments 16—e.g., proxies 46-52 or otherwise— for access to other resources of the native operating system 14 and native runtime environments 16, as well as of the hardware resources of the device 10
  • hosted software applications e.g., 34
  • that utilize a still, video or other camera provided with device 10 can access and/or alter pictures, movies of other image(s) and/or related data generated by that camera and/or by associated application resources 19 and/or runtime libraries 20 (and, more generally, by native runtime environments 16) through use of the 10 proxy 50 or another proxy, e.g., associated with that same hosted software application.
  • Step 72 paralleling the actions discussed in connection with Step 72, when a camera subsystem that forms part of the hosted runtime environment 32 (e.g., and that is common to the one or more hosted software applications) is invoked by a hosted software application, that subsystem loads a buffer and/or messages the natively-executing proxy corresponding to that hosted software application in order to identify primitives to be executed within the native runtime environments 16.
  • the proxy can utilize a camera subsystem of the native runtime environments 16 (or other functionality) to execute those primitives.
  • the proxy can, them, reload that or another buffer or otherwise generate a message with results of such execution and can pass that back to the hosted runtime environments 32 via its event handler 42, e.g., paralleling the actions discussed above in connection with Step 76.
  • the camera subsystem of the hosted runtime environments 32 responds to notification from that event handler 42 by returning to the requisite image(s) and/or other information to the hosted software application that invoked that subsystem.
  • natively-executing proxies can be utilized by hosted software applications to accesses a telephony-related services and/or related data provided by device 10 and/or its native runtime environments 16. This includes not only use of the so-called telephone function (i.e., to make and receive calls), but also telephone logs, address books and other contact information.
  • Native runtime environments 16 responds to activation of an executing native application, e.g., via user selection of the corresponding applications window or icon on the desktop of display 24, or otherwise, by bringing that applications window to the foreground and making it the active task with which the user interacts (and to which user input is directed). Similar functionality is provided by the event handler 42 of hosted runtime environments 32, albeit with respect to executing hosted software applications, with respect to a virtual desktop residing on virtual frame buffer 54, and with respect to virtual user input devices.
  • interactive task is used in reference to an application for which an applications window is generated as part of the graphical user interface of the respective operating system and/or runtime environment reflecting execution that application;
  • the teachings below provide for managing tasks (i.e., applications) where the designation of a foreground task in the hosted application runtime environment 32 is independent of the designation of a foreground task in the native application runtime environment 16, and where tasks in the hosted application runtime environment 32 may (or may not) span multiple processes.
  • native application tasks in operating systems with simple task models are each associated with a single process.
  • Interactive native application tasks 230, 231 are further differentiated from non-interactive tasks (not shown) by their utilization of the graphics stack 255 of the native application runtime environment 110.
  • the graphics stack 255 comprised of drawing module 245 and compositing module 250, updates the contents of the native frame buffer 260 with the visual portions of the foreground task for display to a user via display/touch screen 24.
  • proxy tasks 235, 236 monitor the state (foreground or background) of the hosted application tasks 205, 206, and enable the hosted application tasks 205, 206 to be fully functional within the device 100, despite the differences between the application runtime environments 110 and 120.
  • proxy tasks are created when the hosted tasks are created, but this is not a limitation of the invention.
  • hosted application runtime environment 120 further comprises a task 405 operating in accord with the complex task model and having two processes 411, 412, and a task 406 operating in accord with the simple task model and having one process 413).
  • each of the tasks 405, 406 is associated with one proxy (or client) task 235, 236 respectively, and also associated with one hosted application 205, 206 respectively.
  • step 310 the user selects an interactive task from the task list in the native system.
  • the task switch does not need to be propagated, and the method proceeds to step 330.
  • the hosted application task's view of the virtual frame buffer 220 is updated to the native frame buffer 260.
  • the hosted application task is in the foreground, and the user will be able to view and make use of the user- selected task.
  • the seamless transition allows the user to view the hosted application task 205, 206 as if viewing a native application task.
  • FIG. 10 Another example of the illustrated computing device's 10 merging the user experience so that applications executed in the hosted runtime environment appear, to the user, as if they are executing within the native operating system 14 is the use of a common notification mechanism, e.g., that of the native operating system 14 and/or runtime environments 16.
  • the native and hosted operating systems are assumed to have diverse implementations of notification systems: Each might have a different set of standard prompts, visual indicators, and interprocess messages, on different interprocess message systems, used to notify applications of user interactions with notifications;
  • Described below is a mechanism for enabling hosted applications to use and interact with native system notification subsystems.
  • native operating system 14 has a notification subsystem 1102 that provides a visual display of notifications 1101.
  • Applications 1103 post notifications, using an API of subsystem,1102, and, optionally, can interact with notifications by specifying that they be notified of touches and other user actions through that API, which may use inter-process communication to convey the information about interactions to the application.
  • the notification is not simple, then it is determined if the application is posting a notification with standard, predetermined prompt text, or with a prompt that is application- specific 1303. If the notification being posted uses a standard prompt with a counterpart in the host system, the reference to that prompt is mapped to a reference to the counterpart in the host system 1304. If the prompt is application-specific, or if there is no counterpart to a standard prompt in the host system, the prompt text is passed to the host system to be used in the call to post the notification 1305. If there are graphical assets such as a notification icon in the notification and the asset to be used is from the hosted system 1306 any necessary format conversion is performed 1307. If a graphical asset from the host system is to be used in the notification, the specification or reference to the graphical asset is translated into one used in the host system 1308.
  • the user interacts with the notification 1501, and if the notification return message is not addressed to the proxy 1502, it is a notification for host system applications, and is processed as usual in the host system 1503. If the return message is addressed to the proxy for return messages, it is delivered to the proxy using the host system's inter-process communications mechanism 1504. The proxy uses the reference contained in the return message to find a return message registered with the proxy when the notification was posted, and this message is delivered to the hosted application, using the hosted system's IPC mechanism, as if it were sent by the hosted system's notification system 1505.
  • the illustrated computing device 10 more fully merges the user experience by executing, within a single application address space, instructions comprising a hosted software application (e.g., hosted software application 34) along with instructions from the native runtime libraries 20 and/or other resources of the native runtime environments 16. Also included within that application address space can be instructions from the hosted run-time libraries 44 and/or other resources of the hosted runtime environments 32.
  • the device 10 accomplishes this, inter alia, by linking and loading that hybrid collection of instructions into CPU (and RAM) for execution by using two linker- loaders: one for the hosted instructions and one for the native instructions, yet, both executing in the native runtime environments 16. This assumes that, although the hosted and native operating systems differ (e.g., as discussed elsewhere herein), the instructions of executables of both are suitable for execution on a like CPU— particularly, that of device 10.
  • references to one or more libraries containing at least selected classes and/or functions (collectively, “functions” for sake of simplicity and without loss of generality) of hosted run-time libraries 44 and/or other resources of the hosted runtime environments 32 (collectively, “hosted run-time libraries 44" for sake of simplicity and without loss of generality) called and/or potentially called by the hosted software application 34 executable.
  • the stub can include inline versions of (1) - (4), or a subset thereof, consistent with the teachings hereof.
  • code corresponding to item (3) and, potentially, items (2) and (3) may be absent from any particular stub.
  • a proxy 46 comprising such code can by request from ACL 18 to native operating system 14, in connection with the installation by ACL 18 of respective hosted software application 34, e.g., consistent with the discussion above in the section entitled "Native and Hosted Software Application Installation.”
  • the libraries referred to in (2), above, of the illustrated embodiment are adapted from conventional run-time libraries 44 of the type available in the marketplace for use under the hosted operating system and, particularly, in which at least the selected functions are modified to interface with and to utilize corresponding and/or other functions provided in native runtime libraries 20 and/or native runtime environments 16 resources.
  • some or all of those "adapted" libraries can be adapted from conventional runtime libraries 20 of the type available in the marketplace for use under the native operating system 14 and, particularly, in which at least selected functions are modified to intercept calls from the hosted software application 34 as if part of the hosted run-time libraries 44.
  • selected functions can include any or all functions referenced within hosted software application 34— and, indeed, can include any or all functions (regardless of whether referenced by hosted software application 34) provided within hosted run-time libraries 44— in the illustrated embodiment, the selected functions are those functions of hosted run-time libraries 44 whose execution can be more efficiently and/or beneficially executed, at least in whole or part, using from the native runtime libraries 20 and/or other resources of the native runtime environments 16. This includes, by way of nonlimiting example,
  • the adapted libraries preferably include code that is adapted from the native runtime libraries 20 so as to (i) to intercept calls from the hosted software application 34, and (ii) in the case of memory allocation functions, particularly, malloc, for example, to utilize the malloc function of the native runtime libraries 20 in lieu of that of the hosted runtime libraries 44, (iii) in the case of application threading functions, particularly, pthreads, for example, to emulate the hosted runtime library functions albeit in a manner expressible in context of native runtime library thread management, and (iv) in the case of thread local storage functions, particularly, for example, TLS, parlaying information maintained in individual entries of the vector maintained by TLS of the native runtime libraries 20 (for purposes of managing threads of individual native applications) to manage multiple threads of the hosted software application 34, all by way of nonlimiting example.
  • memory allocation functions particularly, malloc, for example, to utilize the malloc function of the native runtime libraries 20 in lieu of that of the hosted runtime libraries 44
  • application threading functions particularly, p
  • the other functions of the hosted run-time libraries 44 referred to in (3), above, are those functions of conventional hosted run-time libraries 44 (i.e., conventional run-time libraries 44 of the type available in the marketplace for use under the hosted operating system) whose execution is not necessarily more efficiently and/or beneficially effected using from the native runtime libraries 20 and/or other resources of the native runtime environments 16. Examples include mathematical and other computationally-based functions.
  • the native linking/loader 2002 can be a link/loader of the type conventionally available in the marketplace (as adapted in accord with the teachings hereof) for linking and loading native software applications for execution on device 10 under hosted operating system 14.
  • Hosted linking/loader can be of the type conventionally available in the marketplace for linking and loading hosted software applications for execution under the hosted operating system, albeit as adapted in accord with the teachings hereof for execution within native runtime environments 16.
  • Figure 14 is a flow chart depicting operation of device 10 in creating and executing a hybrid application 2000 in native runtime environments 16.
  • native linker/loader 2002 loads general functions necessary for application execution under native operating system 14, e.g., functions of the native runtime libraries 20 and/or other resources of the native runtime environments 16 necessary to allocate allocate and manage memory, threads and so forth, by way of nonlimiting example.
  • step 2012 native linker/loader 2002 accesses the hosted linker/loader 2004, links and loads it for execution.
  • code references functions of the hosted run-time libraries 44 this includes linking the adapted runtime libraries 2008, and, then, the native runtime libraries 20, so as to insure that the adapted libraries 2008 are used in preference to the conventional hosted run-time libraries 44 and to insure that any still unresolved references are satisfied by the native runtime libraries 20.
  • step 2014 once the hosted linker/loader is executed, the native linker/loader 2002 relinquishes control to native operating system 14 and/or native runtime environments 16 to commence execution of the hybrid application 2000 in native runtime environments 16, beginning with the instruction to link and load the hosted software application 34 executable using the hosted linker/loader 2004.
  • This causes the hosted linker/loader 2004 to access the hosted software application 34 executable, and to link and load it for execution.
  • this includes resolving references made in that code by linking it, first, to the code of the adapted adapted libraries 2008, then, to the code of the hosted run-time libraries 44.
  • the hosted linker/loader 2004 can also link the native runtime libraries 20 to resolve any final unresolved references.
  • the executing hybrid application 2000 next executes instructions causing the linked/loaded hosted software application 34 to execute within the native hardware environment of device 10 under the native operating system 14, using functions both from the native runtime libraries 20, the adapted libraries 2008 and the hosted run-time libraries 44.
  • hosted software application 34 runs in the conventional manner within hosted runtime environments 32— e.g., making such calls to the hosted resource framework 38, hosted event handler 42 and run-time libraries 44, all by way of non-limiting example— as it would otherwise make if it were installed on a device executing a single operating system of the type of the hosted operating system.
  • Hosted resource framework 38, hosted event handler 42, run-time libraries 44, and the other components of hosted runtime environments 32 respond to such calls in the conventional manner known in the art of operating systems of the type of hosted operating system, all as adapted in accord with the teachings hereof.
  • the hosted runtime environment 32 loads the virtual frame buffer 54 with such graphics. See, step 72.
  • the hosted runtime environment 32 effects this through use of the windowing and graphics subsystems that form part of the hosted runtime environment 32 and that is common to the one or more hosted software applications 34, 36 (e.g., in that it is the common, shared system used by the hosted software applications for generating applications windows for display to the user of device 10.)
  • the 10 proxy 50 of hosted software application 34 effects presentation on screen 24 of the applications windows generated for application 34 by hosted runtime environments 32, e.g., in the manner shown in Figure 1C and discussed in connection therewith above. See, step 74. 10 proxy 50 does this by transferring the graphics defining that applications window from virtual frame buffer 54 to the native frame buffer 26, e.g., using an API provided by native runtime environments 16 for such purpose or otherwise.
  • resources native to device 10 can include, for example, use of a graphics processing unit (not shown) native (or otherwise coupled) to device 10, e.g., as a companion to the central processing unit (CPU) depicted in Figures 1A.
  • Those resources can also include, for example, use of the native runtime environment(s) 16 (including, kernel 24, application resources 19 and/or runtime libraries 20) and the native graphics subsystem and/or native windowing subsystem forming part of those environments 16, or otherwise, as discussed further below to accelerate execution of at least certain instructions and/or operations of the host software application 34.
  • hosted software application 34 executes instructions that invoke, directly or indirectly, a hosted graphics subsystem and a hosted windowing subsystem that are common to the hosted software applications 34, 36. Together, these are referred to occasionally herein as a "hosted graphics framework.”
  • the hosted graphics subsystem of the illustrated embodiment comprises a subset of the runtime libraries 44 and/or framework 38 that present interfaces (APIs) and provide functionality in accord with the OpenGL ES standard of Khronos Group and its EGL windowing system interface, all as adapted in accord with the teachings hereof.
  • the hosted graphics subsystem may present interfaces and/or functionality in accord with other standards, in accord with proprietary protocols or otherwise.
  • the hosted windowing subsystem of the illustrated embodiment likewise comprises a subset of the runtime libraries 44 and/or framework 38 that creates, manages and presents on display 24 (e.g., via frame buffer 26) windows or frames (as well, optionally, as menus, icons, etc.) that form a graphical user interface in which graphics generated by the application 34 (or applications 34, 36) via the graphics subsystem are presented.
  • the native software applications likewise execute instructions that invoke, directly or indirectly, a native graphics subsystem and a native windowing subsystem that are common to them (i.e., the native software applications), e.g., as shown in Figure 2. Together, these are referred to occasionally herein as a "native graphics framework.”
  • the native graphics subsystem of the illustrated embodiment comprises a subset of the native application resources 19, runtime libraries 20, and/or kernel 22. As above, that graphics subsystem present APIs and provide functionality in accord with the OpenGL ES or other protocol (industry-standard or otherwise) and a complimentary interface to the windowing native to the operating system 14, again, all as adapted in accord with the teachings hereof.
  • the native windowing subsystem of the illustrated embodiment likewise comprises a subset of native application resources 19, runtime libraries 20, and/or kernel 22 that creates, manages and presents on display 24 (e.g., via frame buffer 26) windows or frames (as well, optionally, as menus, icons, etc.) that form a graphical user interface in which graphics generated by the application 34 (or applications 34, 36) via the graphics subsystem are presented.
  • the host and native graphics subsystem can be based on like protocols (e.g., OpenGL and EGL), they need not be. This is likewise true of the host and native windowing subsystems.
  • the native graphics subsystem and the native windowing subsystem are typically adapted to exploit the GPU and other hardware and/or software resources of device 10 (and more directly coupled to them), e.g., in order to rapidly display graphics generated by the native software applications, the hosted graphics subsystem and the native windowing subsystem are not typically so adapted.
  • Some embodiments of the invention remedy that by placing the "hosted graphics framework” and the “native graphics framework” in cooperation as discussed below and elsewhere herein, e.g., to accelerate execution of at least certain instructions in the hosted software applications using the GPU (and/or other graphics-related resources) of device 10.
  • the native graphics framework responds to certain instructions executed by the native software applications by populating respective off-screen pixmap or other buffers allocated by the EGL (or other native graphics interface) component of that framework with graphics. And, a native compositor that also forms part of the native graphics framework generates a composite graphic from such buffers and transmits that graphic (e.g., to frame buffer 26) for presentation on the display 24.
  • the EGL (or other native graphics interface) component of the native graphics interface creates an off-screen buffer and passes at least an identifier of that buffer to the hosted graphics framework to be populated with graphics by it in response to instructions executed by the hosted software applications.
  • a compositor that forms part of the hosted graphics framework generates a composite graphic from one or more off-screen buffers assigned to it by the native graphics framework in connection with graphics generated in response to instructions executed by one or more of the hosted software applications, e.g., 34.
  • the hosted graphics framework and the native graphics framework cooperate to transfer the composite graphic to at least one of the graphics processing unit and the frame buffer 26 for presentation on display 24. That composite graphic, itself, can be transferred by way of an off-screen buffer allocated by the EGL (or other native graphics interface) component of the native graphics framework and passed to the hosted graphics framework.
  • parallel mechanisms can be employed by the native and hosted graphics frameworks to transfer to display 24 two-dimensional graphics generated by hosted graphics framework in response to instructions executed by the hosted applications.
  • the Android runtime environment 32 to run unmodified Android applications 34, 36 in a manner that they both integrate seamlessly with, and take advantage of, the functionality exposed by, the host operating system 14.
  • One area of particular interest in this regard is 3D graphics and, particularly, enabling the Android applications 34, 36 to take advantage of the hardware acceleration provided by a GPU provided on device 10 through the host OS 14.
  • a buffer is essentially an image: a collection of pixels. Surfaces sit one level above buffers, containing them. The most common use of surfaces is as an application window. The window and surface have many external properties (e.g. input handling), but are also associated with one or more buffers. Typically, there are two buffers: a back buffer, not shown on screen, which the application is actively rendering new content to, and a front buffer, containing a set of completed rendering, which is being shown on screen as the window's current content.
  • Compositing is the process of combining multiple buffers and painting them together to produce a coherent image.
  • early window systems involved multiple clients drawing to the same buffer, which was displayed directly on screen (the framebuffer)
  • modern window systems have clients paint directly to their own full-size buffers for each window, with the display server (e.g. the X server or SurfaceFlinger) combining these buffers to render the final display.
  • the display server e.g. the X server or SurfaceFlinger
  • hosted linking loader 2004 runs in the address space of, and is linked with libraries from, the standard host OS.
  • the loader is able to interpret Android binaries, and either direct the calls they make to the Android libraries installed by the OpenMobile system, pass them to the host OS libraries, or intercept them and execute its own code in place of these OS routines.
  • Embodiments without host hardware acceleration have functional support for Android applications using OpenGL ES and EGL to render their user interface, of which Angry Birds is the most prominent example.
  • These implementations uses the reference Android implementations of OpenGL ES and EGL, which are purely software-based.
  • ACL intercepts application requests to display the back buffer (the eglSwapBuffers routine), and in response to this request, copies the contents of the current back buffer into an Xll window inside the host OS, through XPutlmage.
  • This allows Android applications using 3D rendering APIs to run unmodified under ACL, however it does not make use of any 3D acceleration provided by the hardware, and also incurs a number of copies along the way to being displayed on screen.
  • OpenGL ES is a 3D rendering API from Khronos, which is standardised across all manner of devices. Whilst OpenGL ES 1.x provided a deeply inflexible model in which applications were only able to render exactly what had already been specified and implemented, OpenGL ES 2.x provides a fully programmable model. Vertex and fragment programs (collectively 'shaders'), provide a powerful C-like language which allows the application to programmatically direct all rendering.
  • OpenGL ES 2.x has been standard in consumer devices since approximately 2009, when the iPhone 3GS shipped with the PowerVR SGX, along with a number of higher-end Android, Symbian and generic Linux smartphones in the same price range. Since then, OpenGL ES 1.x has rapidly disappeared; even the most low-margin featurephones have shipped with OpenGL ES 2.x for the past couple of years.
  • OpenGL ES 2.x introduced a much stronger reliance on large textures, and flexible sampling from those textures. Whilst this was initially quite painful (an experience borne out by Collabora's work on many early OpenGL ES 2.x devices using 3D compositing, including years of work on the Nokia N900 and N9), modern GPUs have adapted and now place a much stronger emphasis on flexible and fast sourcing from textures. However, OpenGL ES does not and cannot exist in a vacuum.
  • EGL is the window system integration for rendering APIs such as OpenGL ES, whose role is essentially to provide context. Rendering APIs do not have any interaction with the outside world in and of themselves: they have a limited number of buffers provided to them, and their sole function is rendering within those buffers.
  • EGL By providing integration with the outside window system (such as Xll), EGL is able to both initially provide OpenGL ES with buffers to render into, and finally transfer the results to the final display. EGL does not provide complete integration with the window system: in particular, it does not provide any events, so applications are responsible for ensuring that input is handled separately, as well as window resizing.
  • EGL provides a Context, containing the current rendering API in use (e.g. GL ES 1.1, or 2.0), and the surface which is currently being used for rendering.
  • Each thread may have exactly one context bound at any time, thus it is always possible to determine without any additional information or function arguments, all current rendering-related state.
  • EGL provides threetypes of greater interest: Surfaces, Images, and Configs.
  • Surfaces as defined in Surfaces and buffers
  • Images are essentially individual buffers.
  • EGL provides the rendering API with buffers for the window system's surfaces. This is, however, purely an implementation detail: users of OpenGL ES and EGL cannot discover any details about the individual buffers being used in rendering to a surface.
  • Configs fully describe the pixel format used by a particular surface: the depth of the colour channels and resultant bits per pixel value, the conditions under which the config is usable (such as whether it is usable with windows and/or pixmaps), and information about ancillary buffers such as depth and stencil buffers.
  • EGL requires that applications create their own connections to the window system, and from that, create their own windows.
  • the eglGetDisplay and eglCreateWindowSurface entrypoints both take 'native' 1 types, which are both opaque and platform-dependent. On Xll systems, the types are assumed to be a pointer to an open Xll display, and the ID of a client-created Xll window, respectively.
  • the display type is largely ignored, but the window type is a pointer to an abstract class 2 named ANativeWindow. Implementations of this class provides a number of methods, allowing the EGL implementation to be a lightweight wrapper around ANativeWindow, and for users to essentially provide their own window system support without in-depth modification to the entire graphics stack.
  • Skia is Android's 2D graphics library, similar to both canvas-based APIs such as Cairo, and software compositing libraries such as Pixman. Gingerbread's Skia implementation does not attempt to take advantage of any hardware acceleration, and is thus relatively simple to handle: it renders directly to CPU-accessible memory.
  • SurfaceFlinger is mostly restricted to pure compositing (rather than rendering), which in the Gingerbread implementation is provided through OpenGL ES. Anything requiring policy or user interaction - such as window stacking/layering and positioning - is provided through auxiliary services such as WindowManager and Activity Manager.
  • GPU vendors have provided their own complete OpenGL ES and EGL implementations, which are dropped into the system root and used directly by applications and toolkits. This results in unfortunate variations between vendors, where differing implementations produce differing results for seemingly valid input.
  • the Android graphics environment is also rather unique in that it can dynamically switch between the hardware-accelerated backend and its own software-based implementation (AGL), at runtime, as well as switching between GL ES versions. Depending on the EGL context in which they were called, function calls can sometimes end up in both implementations.
  • AGL hardware-accelerated backend
  • AGL software-based implementation
  • XPutlmage is an Xll request which takes the pixels provided by the caller (in this case, the mapped view of the back buffer), and copies them over a UNIX socket to the X server, inline with all Xll requests.
  • the socket is optimised for small and frequent transfers; copying full buffer contents often causes expensive stalls of the client and server as they struggle to complete the copy with a fully blocked socket.
  • the scoring algorithm primarily prefers Xll configs which have the exact same number of bits per pixel as the Android config, and secondarily prefers configs that do not have ancillary buffers wasting memory. More scoring rules would be required if the host EGL offered a larger variety of configs, e.g. GLES2-renderable configs without depth or stencil buffers. This custom scoring is required as the host's eglChooseConfig violates the EGL specification with regards to config ordering.
  • FramebufferNativeWindow is an abstraction of a direct-to-screen rendering path without a real window system behind it. It can be described as rendering to the hardware framebuffer, except it usually involves handling several graphic buffers: at least a front and a back buffer, and sometimes more (a flip queue).
  • map_and_copy is called to create a new CPU-accessible mapping of the buffer contents, usually for software rendering through Skia.
  • Memory is allocated to hold the CPU accessible copy (acl_buffer->cpu_copy).
  • the code to actually do the copy from the Pixmap into cpu_copy is disabled, as it seems to be unnecessary. This is an important performance optimisation, as reading the surface contents back from GL ES into the CPU copy 18 is a particularly expensive operation.
  • the lock function is given a rectangle specified by the application, of the area it will modify.
  • the invention pertains to digital data processing and, more particularly, to apps for both operating systems in a single mode.
  • an object of the invention is to provide improved The invention has application in supporting cross-platform compatibility among systems and methods for digital data processing.
  • apps for smart mobile devices e.g., smart phones, tablet computers
  • set-top object is to provide such systems and methods as support executing on a single boxes, connected televisions, in-vehicle infotainment systems, or in-flight hardware/software platform applications ("apps") made for execution on entertainment systems, and the like, all by way of non-limiting example. multiple different hardware/software platforms.
  • Still another object is to provide such systems and methods as support cross-platform compatibility among apps
  • the smart mobile device market has grown nearly 40% in the past year, for smart mobile devices, e.g., smart phones, tablet computers, set-top boxes, according to analysts. This has been fueled, to a large degree, by the sale of connected televisions, in-vehicle infotainment systems, or in-flight entertainment devices running variants of the open-source Linux and Android operating systems and the like, all by way of non-limiting example.
  • a computing device thot includes a central processing unit that is coupled to a hardware interface (including at least a display and an associated video frame buffer) and that executes a native operating system including one or more native runtime environments within which native software applications are executing, where each such native software application has instructions for execution under the native operating system.
  • a first native software application (“ACL") executing within one or more of the native runtime environments defines one or more hosted runtime environments within which hosted software applications are executing.
  • the first native software application executes code comprising those hosted runtime environment or portions thereof (such as virtual machines); whereas, in other practices, the first native software application can, instead or in addition, effect the installation, instantiation and/or invocation of services/processes executing outside the context of that application that make up those environments or portions thereof.
  • Each such hosted software application has instructions for execution under a hosted operating system that differs from the native operating system.
  • One or more of the hosted software applications executing within the runtime environments each executes instructions to interact with a user of the computing device via graphics generated (as part of a graphical user interface) by the respective hosted software application, using a hosted windowing subsystem that is common to the one or more hosted runtime environments. That windowing subsystem is coupled to, and loads, one or more buffers with those graphics.
  • launch proxies Each application corresponding to IO proxy that received that notification of that corresponding to a respective one of the hosted software applications and user input.
  • a hardware interface for selection by the user of the computing device responds Yet still other aspects of the invention provide a computing device, e.g., to notification of such selection by activating the respective hosted software as described above, in which a first native software application installs an IO application. proxy and launch proxy for execution under the one or more native runtime environments in connection with installation of a respective hosted software
  • IO proxies One or more further native software applications
  • each application for execution under the one or more hosted runtime environments.
  • IO proxies Each application for execution under the one or more hosted runtime environments.
  • a computing device receives the graphics generated by the respective hosted described above, that is a mobile computing device, such as, by way of software application and effects writing of those graphics to the video frame nonlimiting example, a smart phone, tablet computer, set-top box, connected buffer for presentation on the display of the computing device.
  • a mobile computing device such as, by way of software application and effects writing of those graphics to the video frame nonlimiting example, a smart phone, tablet computer, set-top box, connected buffer for presentation on the display of the computing device.
  • television in-vehicle infotainment system, or in-flight entertainment system.
  • the invention provides in other aspects a computing device, e.g., as Further related aspects of the invention provide a computing device, e.g., described above, in which as described above, in which the hosted operating system is a Linux-based operating system, such as, by way of nonlimiting example, an Android-based
  • the hosted and executing within them, including the IO proxies, of user input made native operating systems are differing variants of Linux-based operating systems. with respect to those applications, and And, in yet still further related aspects of the invention, the hosted and native operating systems are differing variants of Android-based operating systems.
  • runtime environments receive notifications of events from a hosted Hosted Application Display in Multi-Operating System Mobile and Other event handler subsystem that forms part of the one or more hosted Computing Devices
  • o computing device thot includes g central processing unit thot is coupled to a hardware interface (including at
  • Each IO proxy responds to notification of user input by transmitting least a display and an associated video frame buffer) and that executes a information with respect thereto received from the one or more native runtime native operating system including one or more native runtime environments environments to the hosted event handler, which notifies the hosted software within which native software applications are executing.
  • Each such native runtime native operating system including one or more native runtime environments environments to the hosted event handler, which notifies the hosted software within which native software applications are executing.
  • a computing device e.g., as system. described above, that is a mobile computing device, such as, by way of nonlimiting example, a smart phone, tablet computer, set-top box, connected
  • a first native software application (“ACL") executing within the one or television, in-vehicle infotainment system, or in-flight entertainment system.
  • ACL A first native software application executing within the one or television, in-vehicle infotainment system, or in-flight entertainment system.
  • ACL A first native software application executing within the one or television, in-vehicle infotainment system, or in-flight entertainment system.
  • ACL A first native software application executing within the one or television, in-vehicle infotainment system, or in-flight entertainment system.
  • more native runtime environments defines one or more hosted runtime
  • first native software application executes code comprising those environments as described above, in which the hosted operating system is a Linux-based (or portions thereof) and/or it effects the installation, instantiation and/or operating system, such as, by way of nonlimiting example, an Android-based invocation of services/processes that make them (or portions thereof) up.
  • the hosted and such hosted software application has instructions for execution under a hosted native operating systems are differing variants of Linux-based operating systems. operating system that differs from the native operating system.
  • the hosted and native operating systems are differing variants of Android-based operating systems.
  • hosted runtime environments each executes instructions to interact with a user
  • Still further related aspects of the invention provide a computing device, of the computing device via graphics generated, as part of a graphical user e.g., as described above, in which each of the native software applications interface, by the respective hosted software application using a hosted executes instructions to interact with the user of the computing device via windowing subsystem that is common to the one or more hosted runtime graphics generated as part of a graphical user interface by the respective environments.
  • Those graphics can be, for example, a graphical window native software application using a native windowing subsystem that is common representing execution of the respective hosted software application. That to the one or more native runtime environments. That windowing subsystem windowing subsystem is coupled to and loads one or more buffers with those effects loading of the native frame buffer with those graphics for presentation graphics. on the display of the computing device.
  • One or more native software applications (“IO proxies"), each executing Yet still further related aspects of the invention provide a computing within the one or more native runtime environments and each corresponding to device, e.g., as described above, in which the one or more buffers loaded by a respective one of the one or more hosted software applications, receives the the hosted windowing subsystem is a virtual frame buffer.
  • Still yet further related aspects of the invention provide a computing display of the computing device. device, e.g., as described above, in which the graphics generated by the hosted software applications using the hosted windowing subsystem are applications windows.
  • a launch proxy effects activation of as described above, in which any of the native operating system and the one or the respective hosted software application by transmitting a launch message to more native runtime environments effects loading of the native frame buffer the hosted event handler, which activates that hosted software application with graphics representing a status bar for presentation on the display of the within one of more of the hosted runtime environments.
  • hosted software applications i.e., the IO proxies
  • a computing device e.g., frame buffer of the graphics received from those hosted software applications as described above, in which the first native software application installs an IO so as to preserve presentation of the status bar on the display. proxy and launch proxy for execution under the one or more native runtime environments in connection with installation of a respective hosted software
  • the invention provides in other aspects a computing device, e.g., as application for execution under the one or more hosted runtime environments. described above, in which (i) the one or more native runtime environments
  • o computing device thot includes notifications of events from a hosted event handler subsystem that forms part of
  • g central processing unit thot is coupled to a hardware interface and that the one or more hosted runtime environments and that is common to the one or
  • Each IO proxy responds to notification of
  • each such native software application has instructions for execution under the one or more native runtime environments to the hosted event handler, which
  • a computing device According to other related aspects of the invention, a computing device,
  • first native software application executes code comprising those environments applications ("launch proxies"), each corresponding to a respective one of the environments applications ("launch proxies"), each corresponding to a respective one of the environments applications ("launch proxies"), each corresponding to a respective one of the environments applications ("launch proxies"), each corresponding to a respective one of the environments applications ("launch proxies"), each corresponding to a respective one of the environments applications ("launch proxies"), each corresponding to a respective one of the environment applications "launch proxies")
  • Such hosted software application has instructions for execution under a hosted the computing device.
  • Each launch proxy responds to notification of such hosted software application
  • runtime environments receive notifications of events from a hosted event operating system, such as, by way of nonlimiting example, an Android-based handler subsystem that forms part of the one or more hosted runtime operating system.
  • a hosted event operating system such as, by way of nonlimiting example, an Android-based handler subsystem that forms part of the one or more hosted runtime operating system.
  • the hosted and environments and that is common to the one or more hosted software native operating systems are differing variants of Linux-based operating systems. applications.
  • the hosted and native operating systems are differing variants of Android-based operating systems.
  • IO proxies One or more native software applications (IO proxies), each executing
  • a computing device a respective one of the one or more hosted software applications, receive e.g., as described above, includes one or more further native software notification of user input made with respect to them from the one or more applications ("launch proxies"), each corresponding to a respective one of the native runtime environments.
  • Each IO proxy responds to notification of user input hosted software applications and each associated with an icon or other by transmitting information with respect thereto received from the one or more identifier that is presented on the hardware interface for selection by the user of native runtime environments to the hosted event handler, which notifies the the computing device.
  • Each launch proxy responds to notification of such hosted software application corresponding to the IO proxy that received that selection by activating the respective hosted software application.
  • the native operating system and/or the one or more native runtime buffer) and that executes a native operating system including one or more environments responds to user selection of an executing one of the native native runtime environments within which native software applications are software applications by bringing a graphical window representing execution of executing.
  • Each such native software application has instructions for execution that application to a foreground of the display and making it "active" within the under the native operating system.
  • one or more native runtime environments are provided.
  • the first native software application e.g., upon being brought to the
  • a first native software application (“ACL") executing within the one or foreground and/or being made active, effects making the the first hosted more native runtime environments defines one or more hosted runtime software application active within the one or more hosted runtime environments environments within which hosted software applications are executing, e.g., the as if it had been brought to the foreground in them.
  • Each executing interface by the respective hosted software application can be a graphical within the one or more native runtime environments and each corresponding to window representing execution of the respective hosted software application.
  • IO proxies Each executing interface by the respective hosted software application can be a graphical within the one or more native runtime environments and each corresponding to window representing execution of the respective hosted software application.
  • a respective one of the one or more hosted software applications receives the
  • the windowing subsystem is coupled to and display of the computing device. loads one or more buffers with those graphics.
  • the first native software application determines whether the corresponding hosted software application
  • a smart phone is active in the one or more hosted application runtime environments using those nonlimiting example, a smart phone, tablet computer, set-top box, connected one or more buffers. television, in-vehicle infotainment system, or in-flight entertainment system.
  • Yet still other aspects of the invention provide a computing device, e.g., display and an associated video frame buffer) and that executes a native as described above, in which a first native software application installs a said IO operating system including one or more native runtime environments within proxy for execution under the one or more native runtime environments in which native software applications are executing, where each such native connection with installation of a respective hosted software application for software application has instructions for execution under the native operating execution under the one or more hosted runtime environments.
  • a first native software application (“ACL") executing within the one or more native runtime environments defines one or more hosted runtime
  • a computing device e.g., as environments within which hosted software applications are executing, e.g., the described above, that is a mobile computing device, such as, by way of first native software application executes code comprising those environments
  • the first native software application e.g., upon being brought to the Still other aspects of the invention provide a computing device, e.g., as foreground and/or being made active, effects making the first hosted software described above, in which the translation layer adapts notifications received application active within the one or more hosted runtime environments as if it from the one or more hosted software applications that include messages that had been brought to the foreground in them. are to be delivered based on the user's interaction with the notification by registering the message with the first native software application and posting to
  • Still other aspects of the invention provide a computing device, e.g., as execution under the native operating system.
  • ACL A first native software application
  • the native runtime environments effects installation, instantiation and/or environments. invocation of services/processes that run outside the context of the first native software application and that make up one or more hosted runtime environments within which hosted software applications are executing (or
  • the invention execute the hosted runtime environments and/or portions thereof.
  • hardware interface including at least a display and an associated video frame
  • One or more of the hosted software applications executing within the buffer) and that executes a native operating system including one or more runtime environments each executes instructions to interact with a user of the native runtime environments within which native software applications are computing device via graphics generated (as part of a graphical user executing.
  • Each such native software application has instructions for execution interface) by the respective hosted software application, using a hosted under the native operating system.
  • windowing subsystem that is common to the one or more hosted runtime
  • a first native software application (“ACL") executing within one or more of environments. That windowing subsystem is coupled to, and loads, one or more
  • the native runtime environments effects installation, instantiation and/or buffers with those graphics.
  • One or more native software applications (“launch proxies"), each software application and that make up one or more hosted runtime corresponding to a respective one of the hosted software applications and environments within which hosted software applications are executing (or each associated with an icon or other identifier, that is presented on the portions of those runtime environments) and may, according to some aspects of hardware interface for selection by the user of the computing device, responds the invention, execute the hosted runtime environments and/or portions thereof. to notification of such selection by activating the respective hosted software
  • Each such hosted software application has instructions for execution under a application. hosted operating system that differs from the native operating system.
  • each One or more of the hosted software applications executing within the executing within the one or more native runtime environments and each hosted runtime environments each executes instructions to interact with a user corresponding to a respective one of the one or more hosted software of the computing device via graphics generated, as part of a graphical user applications, receives the graphics generated by the respective hosted interface, by the respective hosted software application using a hosted software application and effects writing of those graphics to the video frame windowing subsystem that is common to the one or more hosted runtime buffer for presentation on the display of the computing device.
  • Those graphics can be, for example, a graphical window representing execution of the respective hosted software application. That windowin
  • One or more native software applications (“IO proxies”), each executing One or more native software applications (“IO proxies”), each executing within the one or more native runtime environments and each corresponding to within the one or more native runtime environments and each corresponding to a respective one of the one or more hosted software applications, receives the a respective one of the one or more hosted software applications, receive graphics generated by the respective hosted software application and effects notification of user input made with respect to them from the one or more writing of those graphics to the video frame buffer for presentation on the native runtime environments.
  • Each IO proxy responds to notification of user input display of the computing device. by transmitting information with respect thereto received from the one or more native runtime environments to the hosted event handler, which notifies the hosted software application corresponding to the IO proxy that received that notification of that user input.
  • each such native software application has instructions for computing device, e.g., as described above, that includes a central processing execution under the native operating system. unit that is coupled to a hardware interface (including at least a display and an associated video frame buffer) and that executes a native operating system
  • a first native software application (“ACL") executing within one or more of including one or more native runtime environments within which native software the native runtime environments effects installation, instantiation and/or applications are executing.
  • ACL A first native software application
  • Each such native software application has invocation of services/processes that run outside the context of the first native instructions for execution under the native operating system.
  • a first native software application (“ACL") executing within one or more of portions of those runtime environments) and may, according to some aspects of the native runtime environments effects installation, instantiation and/or the invention, execute the hosted runtime environments and/or portions thereof. invocation of services/processes that run outside the context of the first native
  • Each such hosted software application has instructions for execution under a software application and that make up one or more hosted runtime hosted operating system that differs from the native operating system.
  • One or environments within which hosted software applications are executing (or more of the hosted software applications executing within the one or more portions of those runtime environments) and may, according to some aspects of hosted runtime environments receive notifications of events from a hosted event the invention, execute the hosted runtime environments and/or portions thereof. handler subsystem that forms part of the one or more hosted runtime
  • Each such hosted software application has instructions for executioi environments and that is common to the one or more hosted software hosted operating system that differs from the native operating system.
  • One or more of the hosted software applications executing within the one operating systems and that adapts user notifications and replies for applications or more hosted runtime environments each executes instructions to interact with executing on non-native ones of those operating systems.
  • a windowing subsystem that is common to the one or more hosted runtime computing device, e.g., of the type described above, that includes a central environments. That windowing subsystem is coupled to and loads one or more processing unit that is coupled to a hardware interface (including at least a buffers with those graphics. display and an associated video frame buffer) and that executes a native operating system including one or more native runtime environments within a hardware interface (including at least a buffers with those graphics. display and an associated video frame buffer) and that executes a native operating system including one or more native runtime environments within
  • One or more native software applications (“IO proxies”), each executing which native software applications are executing, where each such native within the one or more native runtime environments and each corresponding to software application has instructions for execution under the native operating a respective one of the one or more hosted software applications, receives the system.
  • a first native software application (“ACL”) executing within one or more graphics generated by the respective hosted software application and effects of the native runtime environments effects installation, instantiation and/or writing of those graphics to the video frame buffer for presentation on the invocation of services/processes that run outside the context of the first native display of the computing device.
  • software application and that make up one or more hosted runtime environments within which hosted software applications are executing or
  • the native operating system and/or the one or more native runtime portions of those runtime environments may, according to some aspects of environments responds to user selection of an executing one of the native the invention, execute the hosted runtime environments and/or portions thereof.
  • software applications by bringing a graphical window representing execution of Each such hosted software application has instructions for execution under a that application to a foreground of the display and making it "active" within the hosted operating system that differs from the native operating system.
  • the first native software application e.g., upon being brought to the
  • the one or more native runtime environments include a common native foreground and/or being made active, effects making the the first hosted notification subsystem that is in communications coupling with the native software application active within the one or more hosted runtime environments software applications and that marshals notifications generated by them for as if it had been brought to the foreground in them. presentation to the user via the hardware interface.
  • the one or more hosted runtime environments include a common hosted notification subsystem that is in communications coupling with the hosted
  • the hosted notification subsystem comprises instructions for execution under the hosted operating
  • the native notification subsystem comprises
  • FIGS. 1 A-1 C depict a computing device of the type embodying the
  • a plurality of hosted software applications that each comprise instructions invention
  • Figure 2 depicts a native operating system of the type executing in the processing unit within one of more of the hosted runtime environments.
  • the device and transmit those notifications to the hosted notification subsystem
  • Figure 3 depicts one or more hosted runtime environments defined by a which is in communications coupling with an adaptation layer that adapts
  • Figure 4 depicts the interaction of components in launching an exemplary software applications.
  • Figure 5 is a block diagram illustrating task operations in both the hosted application runtime environment and the native application runtime environment, and a one-to-one correspondence between hosted application tasks and proxy tasks, in accordance with an embodiment of the invention
  • Figure 6 is a block diagram illustrating the relationships between proxy tasks in the native application runtime environment and the complex task
  • Figure 7 is a flow chart illustrating a task switching method occurring in
  • FIG. 1 A depicts a computing device 10 of the type embodying the runtime environment of the device of Figure 5, in accordance with an invention.
  • the illustrated device 10 includes a central processing unit (CPU), embodiment of the invention; input/output (I/O), memory (RAM) and nonvolatile storage (MEM) subsections, of the type commonly provided computing devices of the type commercially
  • CPU central processing unit
  • I/O input/output
  • RAM memory
  • MEM nonvolatile storage
  • Figure 8 depicts interaction of the notification subsystems of the hosted available in the marketplace, all as adapted in accord with the teachings runtime environments and native runtime environments in a system according to hereof.
  • the device 10 comprises a mobile the invention computing device, such as a smart phone or tablet computer, though, in other embodiments it may comprise other computing devices, mobile or otherwise,
  • Figure 9 depicts a notification translation function in a system according e.g., a set-top box, connected television, in-vehicle infotainment system, or into the invention; flight entertainment system, just to name a few.
  • a notification translation function in a system according e.g., a set-top box, connected television, in-vehicle infotainment system, or into the invention; flight entertainment system, just to name a few.
  • Figures 10-12 are flowcharts depicting notification translation in a system
  • the device 10 may be connected permanently, intermittently or otherwise according to the invention; and to one or more other computing devices, servers, or other apparatus capable of digital communications (not shown) by a network, here, depicted by "cloud"
  • FIG. 13-14 parallel Figures 2-3 and depict implementations of hosted
  • the CPU of device 10 executes a native operating system 14 of the type commercially available in the marketplace, as adapted in accord with the teachings hereof.
  • a native operating system 14 of the type commercially available in the marketplace, as adapted in accord with the teachings hereof. Examples of such operating systems include the Meego, Tizen, Android, WebOS, and Linux operating systems, to name just a few. More generally and/or in addition, the native operating system 14 can be a Linux-based operating
  • Native Runtime Environment(s) display/touch screen 24 and the frame buffer 26 that drive displays thereon in the conventional manner known in the art, as adapted in accord with the
  • Figure 2 depicts a native operating system 14 of the type executing on
  • This can also include, by way of non-limiting example, a illustrated device 10 of Figure 1 .
  • keyboard trackball
  • touch stick other user input devices, and/or other integral or peripheral devices of the type known in the art.
  • the native operating system 14 defines one or
  • one or more native runtime environments 1 6, defines one or more hosted runtime environments within which hosted software applications are executing.
  • the native runtime environment(s) 1 6 may comprise one or more virtual
  • the application 18 can execute code comprising those hosted machines or otherwise, as is conventional in the art (as adapted in accord with
  • runtime environment(s) 32 or portions thereof e.g., the virtual machines that the teachings hereof, depending on the native operating system 14 and the
  • hosted software applications may have The hosted runtime environment(s) 32 may comprise one or more virtual corresponding proxies of only one type (e.g., IO or launch) or otherwise.
  • proxies may be provided for each the teachings hereof), depending on the type of the hosted operating system hosted application, and, yet, in still other embodiments, the functions of multiple and the specifics of its implementation within the runtime environments 32.
  • proxies may be combined into a single proxy— all without deviating from Illustrated hosted runtime environment 32 is intended for executing Android- the spirit hereof.
  • based software applications 34, 36 (though, other embodiments may be intended for executing applications designed and built for other operating
  • the native operating system 14 can likewise be, for example,
  • Hosted software applications 34, 36 are installed (upon direction of the passing-off to native operating system 14 and its runtime environment 1 6 user or otherwise) under control of ACL 18 for execution under hosted runtime operations and, more broadly, functions required for execution of hosted environments 32.
  • the ACL 18 can utilize an installer app the type software applications 34, 36 that would otherwise be performed within the conventional to the hosted operating system, albeit, modified to unpack from runtime environment 32 and, specifically, for example by a kernel thereof. the application package files, or otherwise, the to-be-installed application's executable file, icon file, other support files, etc., to suitable locations in static
  • Such passing-off is effected, for example, storage (MEM) on device 10, e.g., locations dictated by native operating system by the resource framework 38, virtual machines (VMs) 36, event handler 42, run14, yet, consistent with the hosted operating system, or otherwise.
  • MEM storage
  • VMs virtual machines
  • event handler 42 run14
  • the native proxies 46-52 executing in runtime environment 16
  • hosted software software applications 46-52 that are proxies of a hosted software application 34, applications 34, 36 to perform such functions or alternates thereof. 36 are installed, by request from ACL 18 to native operating system 14, in connection with the installation by ACL 18 of each respective hosted software
  • Each such proxy 46-52 is installed by the native operating system 14 discussion that follows and elsewhere herein. in the conventional manner, albeit, from application package files (or otherwise) generated by ACL's 18 proxy installer interface 62, which triggers
  • Those package files can include, in lieu of the respective hosted software of the user or otherwise) on device 10 and, more particularly, for execution
  • execution of the the hosted software application 34 instructions attributable to the launch proxies and the IO proxies, respectively. can be carried out directly by the CPU of device (and not, for example, merely emulated by native software application 18)— though, the handling of interrupts
  • proxies may be associated with each
  • application 18 can, instead, effect the displayed on the graphical desktop, e.g., of Figure 1 A— and, more particularly, installation, instantiation and/or invocation of processes— and, more typically, an icon may be associated with only a single one of the multiple proxies that for example, daemons— that execute outside the context of application 18 and are associated with a given hosted software application. that provide services making up those environments 32 without itself executing the code that makes them up. This is illustrated in Figures 13-14, which parallel
  • the native application As illustrated in Figures 2-3, in some embodiments, the native application
  • the code comprising the substituent resource frameworks 38, virtual machines
  • native software application 18 when native software application 18 is 40, event handlers 42, run-time libraries 44, and/or other components of the installed on device 10 under native operating system 14, the application 18 environments 32. Execution of that code can spawn threads or processes but, itself, its installation package, or other functionality (e.g., the native operating typically, they execute within the context of the application 18 itself. system 14) concurrently installs, instantiates and invokes daemons 33 on device
  • application 18 can be likened to an emulator
  • native software application 18 effects installation, instantiation and invocation applications that serve as proxies 50, 52 are taken into account as discussed
  • application 18 installs, instantiates and/or invokes the daemons 33 on a one ⁇
  • m meecchhaanniissmmss ffoorr iinnssttaalllliinngg,, iinnssttaannttiiaattiinngg and/or invoking daemons 33, e.g., under are both targeted to the same CPU, i.e., that provided by device 10).
  • daemons 33 are Referring to Figure 1 A, the native operating system 14 drives the shown in the drawing, in some embodiments other numbers of daemons (for computing device to display, on display/touch screen 24, a graphical desktop example, just one) may be utilized.
  • the daemons may be with icons 58 representing applications that can be selected for launch or other allocated on a per service basis in the illustrated embodiment, in other activation by the user of the device 10. In the illustrated embodiment, these can embodiments they may be allocated on a per hosted application-basis, a per be native software applications, e.g., 15, and hosted software applications, e.g., proxy basis, or otherwise. 34, 36.
  • application 18 takes a mix of the approaches
  • That desktop display includes a status bar 56 of the type conventional in discussed above, e.g., executing code that makes up some portions of the the art — and, particularly, conventional to native operating system 14 environments 32 (e.g., like shown in Figures 2-3, while installing, instantiating (although, some embodiments may vary in this regard) .
  • that status bar 56 and/or invoking services/process to provide services making up other portions of indicates the current date/time, carrier conductivity signal strength (e.g., Wi-Fi, those environments (e.g., like shown in Figures 13-14). cellular, etc.), active apps, and so forth., though, in other embodiments, it may indicate other things.
  • the computing device 10 supports the seomless execution of opplicgtions activated by the operating system 14 and/or runtime environments 16 in of multiple operating systems— or, put onother woy, it "merges" the user response to user selection, the application window 60 generated for it by the experience so thot gpplicgtions executed in the hosted runtime environment native runtime environment 16 (reflecting execution of the application) for oppegr, to the user, os if they gre executing within the ngtive operating system presentation on the screen 24 occupies that screen along with the status bar 14. 56— here, particularly, with the status bar 56 on the top fraction of the screen and the application window 60 on the remainder. Put another way, the
  • operating system 14 and/or runtime environments 1 6 do not overwrite the status hosted soft worn applications are presented to the user without interfering with
  • the application window 60 may occupy the entirety of the screen).
  • Figures 1 A-1 C Referring to Figure 1 C, likewise, in the illustrated embodiment, when a hosted software application 34, 36 is activated, the application window generated for it (reflecting execution in the hosted runtime environments 32) is
  • FIG. 10 Another example of the illustrated computing device's 10 merging the the components discussed obove in launching an exemplary hosted software user experience so that applications executed in the hosted runtime application 34 (here, labelled "App 1 ”) in hosted runtime environments 32 based environment appear, to the user, as if they are executing within the native on user interaction with that app's launch proxy 46 (here, labelled "App #1 operating system 14 is the use of a common notification mechanism, e.g., that Launch Stub") executing in native runtime environments 1 6, displaying an of the native operating system 14 and/or runtime environments 16, e.g., as application window representing operation of hosted software application 34 shown in Figures 8-12 and discussed below in connection therewith. via that app's IO proxy 50 (here, labelled "App # 1 IO Stub”), and transmitting user input from that proxy 50 back to the app 34.
  • App 1 applications executed in the hosted runtime application 34
  • FIG. 10 Another example of the illustrated computing device's 10 merging the the components discussed
  • Still another example is the consistent activation of running software
  • native runtime environments 1 6 and/or native they are native applications, e.g., 15, or hosted software applications 34, 36, operating system 14
  • icons 58 representing native and hosted software applications that can be selected for launch or other activation by the user of the device 10.
  • the native software application that is launch proxy 46 is launched by
  • Proxy 50 can be simultaneously drawings, and elsewhere herein. launched by native runtime environments 16 and/or native operating system 14;
  • proxy 50 can be launched by proxy 46 upon its launch, id.
  • proxy 46 effects activation manner within hosted runtime environments 32, generating such interrupts and of corresponding hosted software application 34. See, step 66. makes such calls to the hosted resource framework 38, hosted event handler 42 and run-time libraries 44, all by way of non-limiting example, as it would
  • proxy 46 does this by transmitting a launch otherwise make if it were installed on a device executing a single operating message to the event handler 42 that forms part of the hosted runtime system of the type of the hosted operating system.
  • This is advantageous in that it environments 32 and that is common to the one or more hosted software does not require special recoding (i.e., "porting") of the hosted software applications 34, 36 (e.g., in that it is the common, shared recipient of system application 34 by the developer or publisher thereof in order to make it possible level-events, such as user input to the hardware interface, which events it to run in the multi-operating system environment of device 10.
  • the launch message which can be delivered to event handler 42 by libraries 44, and the other components of hosted runtime environments 32 proxy 46 using any convention mechanism for inter process communication respond to such interrupts and calls in the conventional manner known of (IPC), e.g., APIs, mailboxes, etc., includes an identifier of the proxy 46 and/or its operating systems of the type of hosted operating system, except insofar as oe corresponding hosted software application 34, as well as any other information evident from the teachings herein.
  • IPC e.g., APIs, mailboxes, etc.
  • step 68 the event handler 42 launches the hosted software application privileged kernel space by hosted runtime environments 32 are, instead, 34 in the conventional manner required of hosted operating system and/or the executed in user space. And, other such operations or, more broadly, functions hosted runtime environments 32. Put more simply, that app 34 is launched as if it are passed-off to native operating system 14 and its runtime environment 1 6, had been selected by the user of device 10 directly. e.g., via the proxies 46-52.
  • event handler 42 uses By way of example, in lieu of loading an actual frame buffer with graphics IPC, e.g., as described above, to signal that hosted software application 34 has defining an applications window representing execution of the hosted software begun execution and, more aptly, to insure launch (if not already effected) and application 34, the hosted runtime environment 32 loads the virtual frame buffer activation of proxy application 50 with the native runtime environments 16. See, 54 with such graphics. See, step 72. The hosted runtime environment 32 effects step 70. this through use of the windowing subsystem that forms part of the hosted runtime environment 32 and that is common to the one or more hosted software applications 34, 36 (e.g., in that it is the common, shared system used by the
  • the IO proxy 50 of hosted software application 34 effects presentation on When IO proxy 50 receives such a notification, it transmits information with screen 24 of the applications windows generated for application 34 by hosted respect thereto to its corresponding hosted software application 34 via event runtime environments 32, e.g., in the manner shown in Figure 1 C and discussed handler 42, e.g., in a manner similar to that discussed above in connection with in connection therewith above. See, step 74. IO proxy 50 does this by transferring step 66. See, step 76.
  • That information which can be delivered to event handler the graphics defining that applications window from virtual frame buffer 54 to 42 by IO proxy 50 using any conventional IPC mechanism, can include and the native frame buffer 26, e.g., using an API provided by native runtime identifier of the IO proxy 50 and/or its corresponding hosted software environments 1 6 for such purpose or otherwise.
  • application 34 an identifier of the device to which input was made
  • the type of the hosted runtime environments 32 utilizes messaging to alert IO proxy 50 of the input, and relevant information with respect thereto (e.g., location, time, need for effecting such a transfer, e.g., when the window subsystem of hosted duration and type of touch, key tapped, pressure on pointer, etc. ) .
  • That runtime environments 32 has generated an updated applications window for information is received by event handler 42 and applied to the corresponding hosted software application 34, when hosted software application 34 becomes hosted software application 34 in the conventional manner required of hosted oe the active (or foreground) app in hosted runtime environments 32, or otherwise, operating system and/or the hosted runtime environments 32, e.g., as if the in other embodiments IO proxy 50 effects such transfers on its own accord on a touch or other user input had been made directly to hosted software periodic basis or otherwise. application 34. See, step 78.
  • IO proxy 50 utilizes o mechanism paralleling that discussed above in As discussed gbove ond elsewhere herein, the respective hosted softwear connection with steps 64-68 in order to transmit taps and other input made by gpplicgtions (e.g., 34) utilize their corresponding proxies (e.g., 46) to perform the the user to device 10 and specifically, for example, to display/touch screen 24, following, by woy of nonlimiting exomple:
  • common event handler (not shown) or other functionality of native runtime present (via operation of native operating system 14) icons on the environments 1 6 notifies applications executing within them, including the IO ngtive operating system 1 4 graphicgl desktop of disploy/touch screen proxies 50, 52, of user input made with respect to them via the touch screen 24 24 for selection by the user;
  • the proxy and native operating system 14 can utilize a camera subsystem of the native runtime environments 1 6 (or other functionality) to execute those primitives.
  • the proxy can, them, reload that or
  • handler 42 e.g., paralleling the actions discussed above in connection with Step input devices
  • the camera subsystem of the hosted runtime environments 32 responds to notification from that event handler 42 by returning to the requisite image(s)
  • the hosted software applications can similarly use proxies executing in the
  • natively- native runtime environments 16 e.g., proxies 46-52 or otherwise— for access to
  • executing proxies can be utilized by hosted software applications to accesses a other resources of the native operating system 14 and native runtime
  • hosted software applications e.g., 34
  • hosted software applications e.g., 34
  • runtime libraries 20 and/or runtime libraries 20 (and, more generally, by native runtime environments
  • Ngtive runtime environments 16 responds to octivgtion of on executing same hosted software application.
  • concise interactive task refers to an application running in respect to executing hosted software applications, with respect to a virtual one process
  • complex interactive task refers to an application running in more than one process
  • application runtime environment 32 may (or may not) span multiple processes. environments 32, as if it had been brought to the foreground in them.
  • the term "interactive task” is used in reference to an application for 255 of the native application runtime environment 1 10.
  • the graphics stack 255 which an applications window is generated as part of the graphical comprised of drawing module 245 and compositing module 250, updates the user interface of the respective operating system and/or runtime contents of the native frame buffer 260 with the visual portions of the foreground environment reflecting execution that application; task for display to a user via display/touch screen 24.
  • Foreground task is used in reference to an application with Hosted (or non-native) application tasks 205, 206 reside within the hosted which the user of device 10 is currently interacting; application runtime environment 120. If the hosted application runtime environment 120 employs a different task model than the native operating system 105, each hosted application task 205, 206 is associated with a proxy (or
  • the proxy tasks 235, 236 reside within the the native application runtime environment 1 10; (ii) enabling the hosted native application runtime environment 1 10 along with the native application application runtime environment's 120 foreground status to be abstracted from tasks 230, 231 , and are managed by the same native task management system the operation and semantics of the task management system in the native in the native application runtime environment 1 10 as the native application application runtime environment 1 10; and (iii) integrating and coordinating the tasks 230, 231 . operation of the hosted application runtime environment 120 and the native application runtime environment 1 10 such that the user cannot discern any
  • the proxy tasks 235, 236 monitor the state (foreground or background) of differences between the functioning of the native application tasks 230, 231 and the hosted application tasks 205, 206, and enable the hosted application tasks the hosted application tasks 205, 206.
  • Figure 7 illustrates the method of switching between interactive tasks and, embodiment, proxy tasks are created when the hosted tasks are created, but more particularly, of coordinating foreground/active tasks, as between the this is not a limitation of the invention.
  • native and posted runtime environments in accordance with a preferred embodiment of the invention.
  • Figure 7 illustrates how the task
  • Hosted application runtime environment 120 comprises a drawing module displayed in the virtual frame buffer 220 of the hosted application interface 210, a windowing module 212, and a compositing module 215, that together environment 120 is coordinated with its corresponding proxy task and the provide the visual portions of the hosted application tasks 230, 231 to the virtual foreground task of the native application runtime environment 1 10.
  • step 310 the user selects an interactive task from the task list in the
  • hosted application runtime environment 120 further native system.
  • proxy tasks 235, 236 are tasks within the native embodiment, each of the tasks 405, 406 is associated with one proxy (or client) application runtime environment 230 that act as proxies for hosted application task 235, 236 respectively, and also associated with one hosted application 205, tasks 205, 206 respectively), are available in the task list for selection by the user. 206 respectively.
  • the method determines whether the user has selected a proxy task or a native application task. Proxy tasks are distinguished from native
  • Any property where a value or a string can be hosted system of drawing 210, windowing 212, and compositing 215 modules, modified can be used, by convention, to identify a proxy task.
  • the virtual frame (or screen) buffer 220 provide the following functions: (i) embodiment, task names are used to distinguish between proxy tasks and enabling the hosted application tasks 205, 206 to run as background tasks within native application tasks, although this is not a limitation of the invention.
  • step 322 the native application seamless transition allows the user to view the hosted application task 205, 206 runtime environment 1 10 switches to the process associated with the selected as if viewing a native application task.
  • step 325 if the hosted application task is not in the virtual foreground of the hosted application runtime environment 120, the task
  • step switch if the user selects a proxy task (i.e., one of 235, 236) at step switch needs to be propagated, and the method proceeds to step 340.
  • the method proceeds to step 320.
  • the native application 340 the hosted application runtime environment 120 switches to the hosted runtime environment 1 10 switches to the process associated with the selected application task 205, 206 associated with the proxy task 235, 236 as described in proxy task (e.g., as discussed elsewhere herein) and brings the selected proxy step 320.
  • the method determines whether the hosted application task
  • the task switch has occurred in the native application 205, 206 is now in the virtual foreground of the hosted application runtime runtime environment 1 10, and may need to be propagated to the hosted environment 120. If the hosted application task is not in virtual foreground of the application runtime environment 120.
  • the method determines hosted application runtime environment 120, the method waits until the hosted whether or not the task switch needs to be propagated to the hosted application task moves to the virtual foreground of the hosted application application runtime environment. runtime environment 120. At this point, the method proceeds to step 330, as described above.
  • the method determines whether the hosted application task
  • the proxy tasks monitor the state (foreground or As noted obove, onother exomple of the illustrated computing device's background) of the hosted application tasks. 10 merging the user experience so thot gpplicgtions executed in the hosted runtime environment oppegr, to the user, os if they gre executing within the
  • the hosted application task is in the virtual foreground of the hosted ngtive operating system 14 is the use of o common notificotion mechanism, e.g., application runtime environment 120, the task switch does not need to be that of the native operating system 14 and/or runtime environments 1 6.
  • step 330 the hosted resource
  • adaptation layer 1 1 04 can be used to translate notifications between the two e.g., some from the notification subsystem of the native operating systems.
  • the adaptation layer 1 104 provides the following functionality to facilitate adaptation:
  • the adaptation layer decides if the hosted communications mechanism (IPC) than the hosted non-native application is posting a simple notification 1301 , without graphical assets, environment, the adaptation layer uses the native inter-process standard prompts that need to be mapped, or a return message. If that is the communications system and is a proxy for non-native applications to case, the parameters of the hosted system's (i.e., the hosted operating system's) the native environment, and uses the non-native IPC mechanism to method are translated to the corresponding parameters in the host system (i.e., communicate with the non-native applications 1 106. the native operating system), and the notification is posted 1302.
  • IPC hosted communications mechanism
  • the prompt is application-specific, or if there is no counterpart to a environment visual display of notifications 1 101 .
  • the translation layer standard prompt in the host system, the prompt text is passed to the host system 1202 can be implemented in the native component and/or the non- to be used in the call to post the notification 1305.
  • graphical assets native component of the adaptation layer 1 104 as needed. such as a notification icon in the notification and the asset to be used is from the hosted system 1306 any necessary format conversion is performed 1307. If a
  • adaptation layer 1 104 has a non-native graphical asset from the host system is to be used in the notification, the component and a native component which provide the aforementioned specification or reference to the graphical asset is translated into one used in functionality.
  • the non-native component has instructions for execution under the the host system 1308.
  • the notification return message is not addressed to the proxy 1502, it is a
  • the proxy uses the reference contained in the return message
  • HOSTED APP INTEGRATION SERVICES IN MULTI-OPERATING SYSTEM MOBILE corresponding to a respective one of the one or more hosted software AND OTHER COMPUTING DEVICES applications, that affords those one or more respective hosted software applications access to resources of the native operating system, native
  • a computing device comprising
  • CPU central processing unit
  • a computing device comprising
  • the CPU executes a native operating system including one or more native
  • each such native software application has instructions for execution under the native operating system
  • runtime environments effects at least one of installation, instantiation and
  • a first native software application executes within the one or more native invocation of any of services and processes (collectively, “processes") that
  • processes make up one or more hosted runtime environments within which hosted
  • hosted software applications are executing, where each such hosted system that differs from the native operating system
  • one or more of the hosted software applications executing within the one utilize a subsystem that is common to the one or more hosted runtime
  • D. one or more native software applications (“IO proxies”), each executing D. one or more native software applications (“IO proxies”), each executing within the one or more native runtime environments and each within the one or more native runtime environments and each corresponding to a respective one of the one or more hosted software corresponding to a respective one of the one or more hosted software applications, receive the graphics generated by the respective hosted applications, receive notification of user input made with respect to them software application and effect writing of those graphics to the video from the one or more native runtime environments, and
  • each IO proxy responds to notification of user input by transmitting
  • a computing device comprising: information with respect thereto received from the one or more native runtime environments to the hosted event handler, which notifies the
  • a computing device comprising
  • a first native software application executing within the one or more native including at least a display and an associated video frame buffer and runtime environments effects at least one of installation, instantiation and that executes a native operating system including one or more native invocation of any of services and processes (collectively, "processes") that runtime environments within which native software applications are run outside the context of the first native software application and that executing, where each such native software application has instructions make up one or more hosted runtime environments within which hosted for execution under the native operating system,
  • a first native software application executes within the one or more native system that differs from the native operating system, runtime environments and effects at least one of installation, instantiation and invocation of any of services and processes (collectively, “processes”)
  • one or more of the hosted software applications executing within the one that run outside the context of the first native software application and or more hosted runtime environments receive notifications of events from that make up one or more hosted runtime environments within which a hosted event handler subsystem that forms part of the one or more hosted software applications are executing, where each such hosted
  • a software application has instructions for execution under a hosted A computing device, comprising:
  • a central processing unit that is coupled to a hardware interface and that
  • one or more of the hosted software applications executing within the one executes a native operating system including one or more native runtime or more hosted runtime environments each execute instructions to environments within which native software applications are executing, interact with a user of the computing device via graphics generated, as where each such native software application has instructions for part of a graphical user interface, by the respective hosted software execution under the native operating system,
  • runtime environments effects at least one of installation, instantiation and invocation of any of services and processes (collectively, "processes") that one or more native software applications (“IO proxies”), each executing run outside the context of the first native software application and that within the one or more native runtime environments and each make up one or more hosted runtime environments within which hosted corresponding to a respective one of the one or more hosted software software applications are executing, where each such hosted software applications, receive the graphics generated by the respective hosted application has instructions for execution under a hosted operating software application and effect writing of those graphics to the video system that differs from the native operating system,
  • IO proxies native software applications
  • the one or more native runtime environments include a common native
  • At least one of the native operating system and the one or more native notification subsystem that is in communications coupling with native runtime environments responds to user selection of an executing one of software applications and that marshals notifications generated by them the native software applications by bringing a graphical window for presentation to the use via the hardware interface
  • the one or more hosted runtime environments include a common hosted and notification subsystem that is in communications coupling with hosted software applications and that marshals notifications generated by them
  • the hosted notification subsystem comprises instructions for execution environments as if it had been brought to the foreground in them. under the hosted operating system and executes on the central
  • processing unit within one of more of the hosted runtime environments.
  • the native notification subsystem comprises instructions for execution ABSTRACT
  • the invention provides, in some aspects, a computing device that includes a central processing unit that is coupled to a hardware interface and that a plurality of hosted software applications that each comprise instructions executes a native operating system including one or more native runtime for execution under the hosted operating system execute on the central environments within which native software applications are executing.
  • a first processing unit within one of more of the hosted runtime environments, native software application executing within the one or more native runtime and environments defines one or more hosted runtime environments within which hosted software applications are executing.
  • One or more further native software one or more of the hosted software applications generate notifications applications ("IO proxies"), each executing within the one or more native for presentation to a user of the device and transmit those notifications to runtime environments and each corresponding to a respective one of the one the hosted notification subsystem, which is in communications coupling or more hosted software applications, receives the graphics generated by the with an adaptation layer that adapts notifications received from the one respective hosted software application and effects writing of those graphics to or more hosted software applications for, and transmits them to, the the video frame buffer for presentation on the display of the computing device.
  • IO proxies notifications applications

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Stored Programmes (AREA)

Abstract

La présente invention concerne, selon certains aspects, un dispositif informatique qui comprend une unité de traitement centrale, une unité de traitement graphique, et un dispositif d'affichage, tous couplés (directement ou indirectement pour des communications). L'unité de traitement centrale exécute un système d'exploitation natif comprenant un ou plusieurs environnements d'exécution natifs dans lesquels des applications logicielles natives s'exécutent, chacune de ces applications logicielles natives possédant des instructions pour une exécution sous le système d'exploitation natif. Une première application logicielle native ("ACL") s'exécutant dans un ou plusieurs des environnements d'exécution natifs définit un ou plusieurs environnements d'exécution hébergés dans lesquels des applications logicielles hébergées s'exécutent. Ainsi, par exemple, selon certaines pratiques de l'invention, la première application logicielle native exécute un code comprenant cet environnement d'exécution hébergé ou des parties de celui-ci (telles que des machines virtuelles) ; tandis que, dans d'autres pratiques, la première application logicielle native peut, à la place ou en plus, effectuer l'installation, l'instanciation et/ou l'invocation de services/processus qui composent ces environnements ou des parties de celui-ci (mais, en effet, peut dans certaines pratiques ne pas exécuter un code comprenant cet environnement). Chaque application logicielle hébergée possède des instructions pour une exécution sous un système d'exploitation hébergé qui est différent du système d'exploitation natif. Des applications logicielles hébergées s'exécutant dans le ou les environnements d'exécution hébergés exécutent des instructions pour générer des graphiques d'affichage tridimensionnels à l'aide d'un sous-système graphique hébergé et d'un sous-système de fenêtrage hébergé (collectivement,"un cadre graphique hébergé"), tous deux communs audit ou auxdits environnements d'exécution hébergés. Des applications logicielles natives s'exécutant dans le ou les environnements d'exécution natifs génèrent également des graphiques d'affichage tridimensionnels à l'aide d'un sous-système graphique natif et d'un sous-système de fenêtrage natif (collectivement, "un cadre graphique natif"), également, tous deux communs audit ou auxdits environnements d'exécution natifs. Les graphiques tridimensionnels peuvent être, par exemple, des fenêtres graphiques (ou des parties de celles-ci) représentant des séquences visuelles de jeux ou d'autres applications graphiques. Le cadre graphique natif est couplé à l'unité de traitement graphique pour accélérer l'exécution d'au moins certaines instructions dans les applications logicielles natives. Le cadre graphique hébergé et le cadre graphique natif coopèrent pour accélérer l'exécution d'au moins certaines instructions dans les applications logicielles hébergées à l'aide de l'unité de traitement graphique.
PCT/US2014/061166 2013-10-18 2014-10-17 Accélération graphique destinée à des applications s'exécutant sur des dispositifs mobiles avec un environnement à systèmes d'exploitation multiples WO2015058100A1 (fr)

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US201361892896P 2013-10-18 2013-10-18
US61/892,896 2013-10-18
US14/061,288 US20140115606A1 (en) 2012-10-24 2013-10-23 Multi-platform mobile and other computing devices and methods
US14/061,288 2013-10-23
US201461983698P 2014-04-24 2014-04-24
US61/983,698 2014-04-24
US201461984549P 2014-04-25 2014-04-25
US61/984,549 2014-04-25

Publications (1)

Publication Number Publication Date
WO2015058100A1 true WO2015058100A1 (fr) 2015-04-23

Family

ID=52828747

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/061166 WO2015058100A1 (fr) 2013-10-18 2014-10-17 Accélération graphique destinée à des applications s'exécutant sur des dispositifs mobiles avec un environnement à systèmes d'exploitation multiples

Country Status (1)

Country Link
WO (1) WO2015058100A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060253548A1 (en) * 2005-04-18 2006-11-09 Research In Motion Limited Method and system for hosting and executing a component application
US20120324338A1 (en) * 2010-06-18 2012-12-20 Sweetlabs, Inc. System and Methods for Integration of an Application Runtime Environment Into a User Computing Environment
US20130100242A1 (en) * 2011-10-24 2013-04-25 Microsoft Corporation Rendering Hardware Accelerated Graphics in a Web Application

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060253548A1 (en) * 2005-04-18 2006-11-09 Research In Motion Limited Method and system for hosting and executing a component application
US20120324338A1 (en) * 2010-06-18 2012-12-20 Sweetlabs, Inc. System and Methods for Integration of an Application Runtime Environment Into a User Computing Environment
US20130100242A1 (en) * 2011-10-24 2013-04-25 Microsoft Corporation Rendering Hardware Accelerated Graphics in a Web Application

Similar Documents

Publication Publication Date Title
US20150193904A1 (en) Graphics acceleration for applications executing on mobile devices with multi-operating system environment
US10127628B2 (en) Method and system to virtualize graphic processing services
EP2622463B1 (fr) Rendu distant instantané
US10074206B1 (en) Network-optimized graphics library for virtualized graphics processing
US8963939B2 (en) Extended graphics context with divided compositing
US8819705B2 (en) User interaction support across cross-environment applications
US8933949B2 (en) User interaction across cross-environment applications through an extended graphics context
US8463980B2 (en) Shared memory between child and parent partitions
US20140115606A1 (en) Multi-platform mobile and other computing devices and methods
CN107292807B (zh) 一种图形合成方法、窗口设置方法及系统
US20150193241A1 (en) Multi-operating system mobile and other computing devices with proxy applications running under a browser
US20150193285A1 (en) Hosted app integration services in multi-operating system mobile and other computing devices
US20150193284A1 (en) Host/hosted hybrid apps in multi-operating system mobile and other computing devices
WO2015058100A1 (fr) Accélération graphique destinée à des applications s'exécutant sur des dispositifs mobiles avec un environnement à systèmes d'exploitation multiples
CN111813404B (zh) 基于混合图形显示的应用方法、介质及客户端
WO2015088646A1 (fr) Services d'intégration d'applications hébergées dans plusieurs systèmes d'exploitation
WO2015058099A1 (fr) Applications hybrides d'hôte/hébergées dans des dispositifs mobiles et d'autres dispositifs informatiques à systèmes d'exploitation multiples
KR20100038027A (ko) 메모리 가상화를 구현한 그래픽 처리 방법 및 그 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14854023

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14854023

Country of ref document: EP

Kind code of ref document: A1