US10402187B2 - Efficient workload deployment using containers and unikernels - Google Patents

Efficient workload deployment using containers and unikernels Download PDF

Info

Publication number
US10402187B2
US10402187B2 US15/662,649 US201715662649A US10402187B2 US 10402187 B2 US10402187 B2 US 10402187B2 US 201715662649 A US201715662649 A US 201715662649A US 10402187 B2 US10402187 B2 US 10402187B2
Authority
US
United States
Prior art keywords
services
applications
container
operating system
packages
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US15/662,649
Other versions
US20180046446A1 (en
Inventor
Andrey TUROVSKY
Aleksandr BIBERMAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Trilio Data Inc
Original Assignee
Trilio Data Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Trilio Data Inc filed Critical Trilio Data Inc
Priority to US15/662,649 priority Critical patent/US10402187B2/en
Assigned to Jisto Inc. reassignment Jisto Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BIBERMAN, ALEKSANDR, TUROVSKY, ANDREY
Publication of US20180046446A1 publication Critical patent/US20180046446A1/en
Assigned to Trilio Data Inc. reassignment Trilio Data Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Jisto Inc.
Application granted granted Critical
Publication of US10402187B2 publication Critical patent/US10402187B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/60Software deployment
    • G06F8/65Updates
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/70Software maintenance or management
    • G06F8/71Version control; Configuration management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/60Software deployment
    • G06F8/61Installation
    • G06F8/63Image based installation; Cloning; Build to order
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/445Program loading or initiating
    • G06F9/44521Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading

Definitions

  • This disclosure is directed generally to containerization and unikernel generation, and, more particularly, to a bottom-up approach for forming condensed containers and unikernels that minimize overhead and enhance performance and resource utilization.
  • Containers enable application/service deployment environments with relatively-low overhead compared with other virtualization techniques, while offering portability, compatibility, and isolation. They leverage server virtualization methods such as operating system-level virtualization, where the kernel of an operating system allows for multiple isolated user space instances, instead of just one.
  • Unikernels consist of single address space machine images, and may be constructed with the use of library operating systems (OS). Traditionally, to deploy applications/services as unikernels, the developers would manually identify the minimum set of libraries corresponding to the operation system (OS) constructs for specific types of unikernels. These libraries would then be compiled with the applications/services and any corresponding configuration code. Many times, device drivers would also be required for specific hardware on which the unikernel is to be deployed, as well as protocol libraries.
  • OS operation system
  • unikernels Compared with deploying applications/services using traditional operating systems, deploying applications/services using unikernels offers several benefits, which include small footprint, short boot times, and increased security. However, because of their high degree of specialization, combined with their inability to allow developers to debug or modify them after compilation, unikernels are unsuitable for general-purpose, multi-user computing, an area where traditional operating systems excel.
  • One implementation of condensed containerization, a bottom-up approach includes a system that first automatically determines the minimum set of requirements and dependencies, in forms of libraries and packages, for deployed applications/services and any corresponding user-specified content. This system determines which versions of libraries and packages are most fitting by comparing lists corresponding to compatible operating systems, merges them with the applications/services, and performs the containerization of the merged results using a minimized version of a container base image. This minimized version of a container base image, which sometimes may be referred to as a “scratch” container, acts as a minimal starting point for building containers.
  • the system produces optimized (in terms of size, speed, resource utilization, and performance) container images from applications/services by leveraging traditional containerization technologies.
  • One implementation of specialized unikernel generation includes a system that first automatically determines the minimum set of requirements and dependencies, in forms of libraries and packages, for the deployed applications/services and any corresponding user-specified content. This system then matches the best available library operating system, and performs the compilation to generate the unikernel. Thus, this system produces optimized (in terms of most fitting to applications/services, for any combination of size, speed, resource utilization, and performance) unikernel images from applications/services by leveraging traditional unikernel technologies.
  • part of this disclosure describes a method that automatically selects the best deployment packages for which to deploy the applications/services at any given time. This selection may be based on application/service type, size, granularity, duration, estimated execution time, coupling, modularity, technical fit, user-specified content, resource availability, resource utilization metrics, historical data, including the use of machine learning, specifications for any licensed software, security boundaries in the network and/or organization, as well as device driver and protocol library availabilities. Based on these decision points, there is also a path in the system for bypassing the deployment packages, which may also be used for debugging purposes.
  • a method for creating a condensed container includes receiving an instruction to create a container that includes one or more applications and services, and identifying one or more shared libraries and corresponding packages required for executing applications and services.
  • An optimal operating system for the container is determined, and versions of the shared libraries and the corresponding packages that correspond to the optimal operating system are selected.
  • a container is then created based on a merge formed of the selected versions of the shared libraries and corresponding packages with the applications and services.
  • Other aspects of the foregoing method include corresponding systems and non-transitory computer-readable media.
  • one or more user-selected libraries and corresponding packages are identified based on functionality specified by a user for inclusion in the container. Versions of these user-selected libraries and corresponding packages that correspond to the optimal operating system are selected and included in the container.
  • a user interface that allows the user to select functionality for inclusion in the container can be provided.
  • the optimal operating system for the container can be determined by identifying one or more operating systems that support each of the applications and services.
  • the container can be created by containerizing the merge using a minimized container based image.
  • the container can be deployed to one or more servers.
  • a deployment package is selected from a plurality of deployment packages to deploy one or more applications and services, based on at least one factor associated with the applications and services, specified user content, deployment resources, historical data, licensed software specifications, organization or network security boundaries, or driver or protocol library availability.
  • the applications and services are then deployed using the selected deployment package.
  • Factors associated with the applications and services can include type, size, granularity, duration, estimated execution time, coupling, modularity, and technical fit between each application or service and available deployment packages.
  • Factors associated with the deployment resources can include server cluster resource availability and resource utilization metrics.
  • a method for creating a unikernel includes identifying one or more requirements or dependencies required for executing one or more deployed applications and services, and identifying a library operating system to use for the deployed applications and services.
  • the deployed applications and services and the library operating system are compiled into a unikernel.
  • a container is created based on the unikernel.
  • Other aspects of the foregoing method include corresponding systems and non-transitory computer-readable media.
  • FIG. 1 is an exemplary Architecture for Efficient Workload Deployment Using Containers and Unikernels.
  • FIG. 2 is an exemplary Architecture for Condensed Containerization.
  • FIG. 3 is an exemplary Library and Package Repository, depicting the Example Repository Lists of Libraries and Packages for Various Example Operating Systems.
  • FIG. 1 an exemplary embodiment of the Architecture for Efficient Workload Deployment Using Containers and Unikernels concept is illustrated and include the following elements.
  • User submission Interface [ 100 ] is an interface that a user may use to interface with the system. For example, the user may submit Applications/Services [ 101 ] to the Deployment Package Selector [ 106 ] through the User submission Interface [ 100 ]. This may be accomplished, for example, manually or automatically, through a command-line interface (CLI), graphical user interface (GUI), application program interface (API), web browser interface, or any combination thereof.
  • CLI command-line interface
  • GUI graphical user interface
  • API application program interface
  • web browser interface or any combination thereof.
  • Applications/Services [ 101 ] are software functionalities that can be reused for different purposes, and may also be coupled with policies that control their usage.
  • Applications/Services [ 101 ] may be defined through enterprise architecture (EA) practices for organizing business processes and information technology (IT) infrastructure.
  • EA enterprise architecture
  • IT information technology
  • Different enterprises may have different types of Applications/Services [ 101 ] with various application granularities, which are often also correlated with their size and duration.
  • Applications/Services [ 101 ] that may be deployed in any state, including (but not limited to) as any combination of source code, binaries, functional elements, build instructions, container images, virtual machine (VM) images, or unikernel images.
  • Applications/Services [ 101 ] may run in architectures that include, for example, any combination of:
  • User Specification Interface [ 102 ] is an interface that the user may use to interface with the system. For example, the user may select which packages they want to be included with the Applications/Services [ 101 ] through the User Specification Interface [ 102 ]. This may be accomplished, for example, manually or automatically, through a command-line interface (CLI), graphical user interface (GUI), application program interface (API), web browser interface, or any combination thereof.
  • CLI command-line interface
  • GUI graphical user interface
  • API application program interface
  • web browser interface or any combination thereof.
  • Example User Specification Interface (e.g. Web Browser Interface) [ 103 ], is one example instantiation of the User Specification Interface [ 102 ].
  • This example includes the Package Selection [ 104 ], which has an example list of packages for the user, with corresponding checkboxes the user can check, selecting, in this example, which packages are to be included with the Application/Service [ 101 ] deployments.
  • Add SSH (Secure Shell) Support [ 105 ] is an example checkbox the user may check to select the corresponding package to be deployed with the Applications/Services [ 101 ], as described in the Example User Specification Interface (e.g. Web Browser Interface) [ 103 ] example above.
  • Example User Specification Interface e.g. Web Browser Interface
  • Deployment Package Selector [ 106 ] keeps track of all the Applications/Services [ 101 ] and any User Specification Interface [ 102 ] content that the user submits, and selects the best Deployment Packages [ 107 ] for which to deploy them at any given time.
  • the decision of which Deployment Packages [ 107 ] to deploy with may be based on any combination of (a) Application/Service [ 101 ] type, size, granularity, duration, estimated execution time, coupling, modularity; (b) technical fit between each Applications/Services [ 101 ] and Deployment Packages [ 107 ]; (c) User Specification Interface [ 102 ] content and any corresponding user specification; (d) resource availability in the Heterogeneous/Hybrid Server Cluster [ 120 ]; (e) resource utilization metrics gathered by the Global Resource Tracker [ 122 ]; (f) historical data, including the use of machine learning; (g) specifications for any licensed software; (h) security boundaries in the network and/or organization; (i) device driver and protocol library availabilities.
  • Bypassing and Debugging [ 117 ] is a path that the Deployment Package Selector [ 106 ] may also choose for the submitted Applications/Services [ 101 ] and any corresponding User Specification Interface [ 102 ] content from user specification to bypass the Deployment Packages [ 107 ], based on these same aforementioned decision points.
  • This Bypassing and Debugging [ 117 ] path may also be used for debugging purposes, deploying the same Applications/Services [ 101 ] through the Bypassing and Debugging [ 117 ] path that were deployed through Deployment Packages [ 107 ].
  • Condensed Containerization [ 108 ] which is described in detail in FIG. 2 , is a Deployment Package [ 107 ].
  • Condensed Containerization [ 108 ] a unique process that is part of this invention, is able to produce optimized (in terms of size, speed, resource utilization, and performance) container images from Applications/Services [ 101 ] by leveraging traditional containerization technologies.
  • OS operating system
  • VEs virtualization engines
  • VPN virtual private servers
  • jails or zones, and/or any hybrid combination thereof.
  • Some example available technologies for this containerization include chroot, Docker, Linux-VServer, lmctfy (“let me contain that for you”), LXC (Linux containers), Canonical LXD, OpenVZ (Open Virtuozzo), Parallels Virtuozzo Containers, Solaris Containers (and Solaris Zones), FreeBSD Jail, sysjail, WPARs (workload partitions), HP-UX Containers (SRP, secure resource partitions), iCore Virtual Accounts, Sandboxie, Spoon, and VMware ThinApp.
  • Specialized Unikernel Generation [ 109 ] is a Deployment Package [ 107 ].
  • Specialized Unikernel Generation [ 109 ] a unique process that is part of this invention, is able to produce optimized (in terms of most fitting to Applications/Services [ 101 ], for any combination of size, speed, resource utilization, and performance) unikernel images from Applications/Services [ 101 ] by leveraging traditional unikernel technologies.
  • Unikernel technologies which are depicted by Unikernel [ 114 ] in FIG. 1 , may consist of single address space machine images, and may be constructed with the use of library operating systems (OS).
  • OS library operating systems
  • Some example available technologies for this unikernel include ClickOS, Clive, Drawbridge, Graphene, HaLVM (Haskell Lightweight Virtual Machine), IncludeOS, LING, MirageOS, OSv, Rumprun, and runtime.js.
  • Application Requirement Determination [ 110 ] is a step in the Specialized Unikernel Generation [ 109 ] that determines the various requirements and dependencies for deployed Applications/Services [ 101 ] and any corresponding User Specification Interface [ 102 ] content. For example, this may be determining required and user-specified libraries and packages, similar to Application/Service Library and Package Requirement Determination [ 201 ] and User Specification Application/Service Library and Package Requirement Determination [ 208 ], both depicted in FIG. 2 , respectively.
  • Application Requirement Determination [ 110 ] may also determine which programming language(s) and software framework(s) are required. For Applications/Services [ 101 ] that are deployed as source code, the Application Requirement Determination [ 110 ] may also build the Applications/Services [ 101 ] into binaries to gather some or all of these requirements.
  • the Library Operating System Matching [ 111 ] which determines which available library operating system is best suited for the deployed Applications/Services [ 101 ].
  • the selection factors may be taken from the requirements gathered from the Application Requirement Determination [ 110 ], and may also be combined with other factors such as resource utilization metrics gathered by the Global Resource Tracker [ 122 ] and User Specification Interface [ 102 ] content, matching the deployed Applications/Services [ 101 ] with available library operating systems in the Library Operating System Repository [ 112 ].
  • Library Operating System Repository [ 112 ] is the repository where the available library operating systems are stored.
  • Library Operating System Compiler [ 113 ] is a compiler that converts the deployed Applications/Services [ 101 ] into the Unikernel [ 114 ], using the library operating system selected by the Library Operating System Matching [ 111 ] from the Library Operating System Repository [ 112 ]. As part of the compilation, the Library Operating System Compiler [ 113 ] may also include specific language runtimes, any required device drivers and protocol libraries.
  • Application [ 115 ] and Library Operating System Runtime [ 116 ] are parts of the compiled product of the Unikernel [ 114 ].
  • the Application [ 115 ] is unique to each Applications/Services [ 101 ] deployment, and the Library Operating System Runtime [ 116 ] may be unique to each type of library operating system.
  • Orchestrator [ 119 ] may refer to the Application Management Engines ( 106 ), as part of the Elastic Compute Cloud Framework described in the utility patent application Elastic Compute Cloud Based on Underutilized Server Resources Using a Distributed Container System (U.S. patent application Ser. No. 14/640,569, filed on Mar. 6, 2015), as well as the Elastic Workload Orchestrator [ 107 ], as part of the Elastic Workload Orchestration Architecture described in the utility patent application Ubiquitous and Elastic Workload Orchestration Architecture of Hybrid Applications/Services on Hybrid Cloud (U.S. patent application Ser. No. 15/381,427, filed on Dec. 16, 2016), the entireties of which are incorporated by reference herein.
  • Heterogeneous/Hybrid Server Cluster [ 120 ] is a cluster that is formed from any hybrid combination of Server [ 121 ] resources, spanning, for example, on-premises data centers, private clouds, and public clouds.
  • Server [ 121 ] which may operate within a client-server architecture, responds to requests across a computer network to provide, or help to provide a networked service.
  • the client or clients may run on the same physical or virtual Server [ 121 ], or may be connected to the Server [ 121 ] over a network.
  • Some examples of Server [ 121 ] types include computing servers, application servers, database servers, web servers, file servers, mail servers, and game servers.
  • Global Resource Tracker [ 122 ] which keeps track of resource utilization metrics of each Server [ 121 ] in the Heterogeneous/Hybrid Server Cluster [ 120 ].
  • These resource utilization metrics may include, but are not limited to, capacity of its central processing unit (CPU), random-access memory (RAM), graphics processing unit (GPU), storage, network bandwidth, memory bandwidth, and central processing unit (CPU) cache.
  • resource utilization metrics may be gathered for, and differentiated between, physical servers, virtual machines (VMs), containers, unikernels, applications, and processes. These resource utilization metrics may be gathered from agents running on each Server [ 121 ], as well as with agentless approaches.
  • the Global Resource Tracker [ 122 ] shares the resource utilization metrics with the (a) Deployment Package Selector [ 106 ], to assist in selecting the most fitting Deployment Package [ 107 ] for each Application/Service [ 101 ] deployment; (b) Condensed Containerization [ 108 ], for optimized container-based deployment; (c) Specialized Unikernel Generation [ 109 ], for optimized unikernel-based deployment; (d) Orchestrator [ 119 ], for selecting the most fitting Server [ 121 ] onto which to deploy in the Heterogeneous/Hybrid Server Cluster [ 120 ].
  • FIG. 2 an exemplary embodiment of the Architecture for Condensed Containerization [ 108 ] concept is illustrated and includes the following elements.
  • Library and Package Repository [ 200 ] may be comprised of lists corresponding to available libraries and corresponding packages, for each operating system. These lists may leverage any combination of local libraries and packages, those that that are in other designated repositories, and those that are in common public repositories.
  • Library and Package Repository [ 200 ] assists with Application/Service Library and Package Requirement Determination [ 201 ], User Specification Application/Service Library and Package Requirement Determination [ 208 ], as well as Containerization [ 215 ].
  • Application/Service Library and Package Requirement Determination [ 201 ] are steps that determine all the library and corresponding package requirements based on the Applications/Services [ 101 ] deployed by the user.
  • Application/Service Library and Package Requirement Determination [ 201 ] steps include:
  • All Available Shared Libraries and Packages [ 204 ] is an example list that illustrates, as an example, the Application/Service Library and Package Requirement Determination [ 201 ] step results. As described in FIG. 2 , this is an Example List of Shared Libraries with Corresponding Packages for the Application/Service [ 205 ]. All Available Shared Libraries and Packages [ 204 ], as an example, contains lists of Shared Libraries [ 206 ], which are operating system libraries that are loaded by programs when they start, and Packages [ 207 ], which are operating system packages, for an example operating system. Similar lists may be generated for any operating system type and version, as depicted in FIG. 3 for the Library and Package Repository [ 200 ].
  • User Specification Application/Service Library and Package Requirement Determination [ 208 ] are steps that determine all the library and corresponding package requirements based on any User Specification Interface [ 102 ] content interfaced with by the user.
  • User Specification Application/Service Library and Package Requirement Determination [ 208 ] steps include:
  • Library and Package Dependencies and Sub-Dependencies from User Specification [ 210 ] is an example list that illustrates, as an example, the User Specification Application/Service Library and Package Requirement Determination [ 208 ] step results. As described in FIG. 2 , this is an Example List of Libraries and Packages Corresponding to the User Specification Interface [ 211 ], which lists out SSH (Secure Shell) Support [ 212 ] based on the checked Add SSH (Secure Shell) Support [ 105 ] from the Package Selection [ 104 ] example of the User Specification Interface [ 102 ] content.
  • SSH Secure Shell
  • Add SSH Secure Shell
  • Library and Package Dependencies and Sub-Dependencies from User Specification [ 210 ] contains lists of Libraries [ 213 ], which are operating system libraries, and Packages [ 214 ], which are operating system packages, for an example operating system. Similar lists may be generated for any operating system type and version, as well as for other example User Specification Interface [ 102 ] content, as depicted in FIG. 3 for the Library and Package Repository [ 200 ].
  • Containerization [ 215 ] are steps that are able to create the condensed container from the determined library and corresponding package requirements based on the Applications/Services [ 101 ] deployed by the user, as output from the Application/Service Library and Package Requirement Determination [ 201 ], as well as determine the library and corresponding package requirements based on any User Specification Interface [ 102 ] content interfaced with by the user, as output from the User Specification Application/Service Library and Package Requirement Determination [ 208 ], combined with the assistance of the Library and Package Repository [ 200 ].
  • Containerization [ 215 ] steps include:
  • the resulting container image from Containerization [ 215 ], encapsulating the deployed Application/Service [ 101 ] and all the required libraries and packages, may then be deployed to the Orchestrator [ 119 ], which then deploys it to various Server [ 120 ] resources.
  • FIG. 3 an exemplary embodiment of the Library and Package Repository [ 200 ], depicting the Example Repository Lists of Libraries and Packages for Various Example Operating Systems [ 300 ] concept is illustrated and include the following elements.
  • FIG. 3 depicts the Example Repository Lists of Libraries and Packages for Various Example Operating Systems [ 300 ].
  • Some example operating systems are depicted as [ 301 ], specifically Ubuntu ⁇ [ 301 ], Ubuntu ⁇ [ 301 ], and CentOS ⁇ [ 301 ], which, for the purposes of this example, are different types and versions of Linux.
  • FIG. 3 also depicts the Example Repository List of All Available Libraries and Packages, for Each Operating System [ 302 ], which comprises All Available Libraries and Packages [ 204 ], Shared Libraries [ 206 ], and Packages [ 207 ], for each operating system.
  • FIG. 3 also depicts the Example Repository List of All Library and Package Dependencies and Sub-Dependencies from User Specifications, for Each Operating System [ 303 ], which comprises All Library and Package Dependencies and Sub-Dependencies from User Specification [ 210 ], SSH (Secure Shell) Support [ 212 ], Libraries [ 213 ], and Packages [ 214 ], for each operating system.
  • SSH Secure Shell
  • Implementations of the system described herein can include a plurality of software processing modules stored in a memory and executed on a processor.
  • the program modules can be in the form of one or more suitable programming languages, which are converted to machine language or object code to allow the processor or processors to execute the instructions.
  • the software can be in the form of one or more standalone applications implemented in a suitable programming language or framework.
  • Communication among devices can take place over media such as standard telephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25), broadband connections (ISDN, Frame Relay, ATM), wireless links (802.11 (Wi-Fi), Bluetooth, GSM, CDMA, etc.), for example.
  • the network can carry TCP/IP protocol communications, and HTTP/HTTPS requests made by a web browser, and the connection between the user devices and servers can be communicated over such TCP/IP networks.
  • Method steps of the techniques described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. Method steps can also be performed by, and the modules can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Modules can refer to portions of the computer program and/or the processor/special circuitry that implements that functionality.
  • special purpose logic circuitry e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
  • Modules can refer to portions of the computer program and/or the processor/special circuitry that implements that functionality.
  • processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors.
  • a processor will receive instructions and data from a read-only memory or a random access memory or both.
  • the essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data.
  • Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
  • One or more memories can store instructions that, when executed by a processor, form the modules and other components described herein and perform the functionality associated with the components.
  • the processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.
  • implementations of the systems and methods can be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture, including non-transitory storage mediums.
  • the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
  • a computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them.
  • a computer storage medium is not a propagated signal
  • a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal.
  • the computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Stored Programmes (AREA)

Abstract

A bottom-up approach is taken to creating a condensed container, in which an automatic determination is made of the minimum set of requirements and dependencies, in forms of libraries and packages, for deployed applications/services and any corresponding user-specified content. The most fitting libraries and packages are merged with the applications/services, and the merged results are containerized using a minimized version of a container base image. Specialized unikernel generation includes automatically determining the minimum set of requirements and dependencies, in forms of libraries and packages, for the deployed applications/services and any corresponding user-specified content. The best available library operating system is identified, and compilation occurs to generate the unikernel. In effect, optimized (in terms of most fitting to applications/services, for any combination of size, speed, resource utilization, and performance) containers and unikernel images from applications/services by leveraging traditional unikernel technologies are produced.

Description

CROSS-REFERENCE TO RELATED APPLICATION
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/373,147, filed on Aug. 10, 2016, and entitled “Efficient Workload Deployment Using Containers and Unikernels,” the entirety of which is incorporated by reference herein.
TECHNICAL FIELD
This disclosure is directed generally to containerization and unikernel generation, and, more particularly, to a bottom-up approach for forming condensed containers and unikernels that minimize overhead and enhance performance and resource utilization.
BACKGROUND
Containers enable application/service deployment environments with relatively-low overhead compared with other virtualization techniques, while offering portability, compatibility, and isolation. They leverage server virtualization methods such as operating system-level virtualization, where the kernel of an operating system allows for multiple isolated user space instances, instead of just one.
However, traditional application/service deployments using containers have employed a top-down approach, which containerizes applications/services in a container that is inclusive of many standard libraries, packages, and binaries of an operating system (OS). These comprise many of the possible requirements and dependencies for this application/service, many of which will not be used during regular operation of the application/service, but are still included in the container. Even though the container allows for relatively-low overhead by leveraging a shared kernel between containers on the same host, that overhead is still relatively high compared to the minimum set of requirements and dependencies of the application/service during regular operation.
Unikernels consist of single address space machine images, and may be constructed with the use of library operating systems (OS). Traditionally, to deploy applications/services as unikernels, the developers would manually identify the minimum set of libraries corresponding to the operation system (OS) constructs for specific types of unikernels. These libraries would then be compiled with the applications/services and any corresponding configuration code. Many times, device drivers would also be required for specific hardware on which the unikernel is to be deployed, as well as protocol libraries.
Compared with deploying applications/services using traditional operating systems, deploying applications/services using unikernels offers several benefits, which include small footprint, short boot times, and increased security. However, because of their high degree of specialization, combined with their inability to allow developers to debug or modify them after compilation, unikernels are unsuitable for general-purpose, multi-user computing, an area where traditional operating systems excel.
SUMMARY OF THE INVENTION
The present disclosure describes, in various implementations, systems and methods for efficient workload deployment using containers and unikernels. One implementation of condensed containerization, a bottom-up approach, includes a system that first automatically determines the minimum set of requirements and dependencies, in forms of libraries and packages, for deployed applications/services and any corresponding user-specified content. This system determines which versions of libraries and packages are most fitting by comparing lists corresponding to compatible operating systems, merges them with the applications/services, and performs the containerization of the merged results using a minimized version of a container base image. This minimized version of a container base image, which sometimes may be referred to as a “scratch” container, acts as a minimal starting point for building containers. Thus, the system produces optimized (in terms of size, speed, resource utilization, and performance) container images from applications/services by leveraging traditional containerization technologies.
One implementation of specialized unikernel generation includes a system that first automatically determines the minimum set of requirements and dependencies, in forms of libraries and packages, for the deployed applications/services and any corresponding user-specified content. This system then matches the best available library operating system, and performs the compilation to generate the unikernel. Thus, this system produces optimized (in terms of most fitting to applications/services, for any combination of size, speed, resource utilization, and performance) unikernel images from applications/services by leveraging traditional unikernel technologies.
Furthermore, for systems leveraging both deployment packages of condensed containerization and specialized unikernel generation, part of this disclosure describes a method that automatically selects the best deployment packages for which to deploy the applications/services at any given time. This selection may be based on application/service type, size, granularity, duration, estimated execution time, coupling, modularity, technical fit, user-specified content, resource availability, resource utilization metrics, historical data, including the use of machine learning, specifications for any licensed software, security boundaries in the network and/or organization, as well as device driver and protocol library availabilities. Based on these decision points, there is also a path in the system for bypassing the deployment packages, which may also be used for debugging purposes.
Accordingly, in one aspect, a method for creating a condensed container includes receiving an instruction to create a container that includes one or more applications and services, and identifying one or more shared libraries and corresponding packages required for executing applications and services. An optimal operating system for the container is determined, and versions of the shared libraries and the corresponding packages that correspond to the optimal operating system are selected. A container is then created based on a merge formed of the selected versions of the shared libraries and corresponding packages with the applications and services. Other aspects of the foregoing method include corresponding systems and non-transitory computer-readable media.
In one implementation, one or more user-selected libraries and corresponding packages are identified based on functionality specified by a user for inclusion in the container. Versions of these user-selected libraries and corresponding packages that correspond to the optimal operating system are selected and included in the container.
Other implementations include one or more of the following features. A user interface that allows the user to select functionality for inclusion in the container can be provided. The optimal operating system for the container can be determined by identifying one or more operating systems that support each of the applications and services. The container can be created by containerizing the merge using a minimized container based image. The container can be deployed to one or more servers.
In a further implementation, a deployment package is selected from a plurality of deployment packages to deploy one or more applications and services, based on at least one factor associated with the applications and services, specified user content, deployment resources, historical data, licensed software specifications, organization or network security boundaries, or driver or protocol library availability. The applications and services are then deployed using the selected deployment package. Factors associated with the applications and services can include type, size, granularity, duration, estimated execution time, coupling, modularity, and technical fit between each application or service and available deployment packages. Factors associated with the deployment resources can include server cluster resource availability and resource utilization metrics.
In another aspect, a method for creating a unikernel includes identifying one or more requirements or dependencies required for executing one or more deployed applications and services, and identifying a library operating system to use for the deployed applications and services. The deployed applications and services and the library operating system are compiled into a unikernel. In some implementations, a container is created based on the unikernel. Other aspects of the foregoing method include corresponding systems and non-transitory computer-readable media.
BRIEF DESCRIPTION OF THE DRAWINGS
Various objectives, features, and advantages of the disclosed subject matter can be more fully appreciated with reference to the following detailed description of the disclosed subject matter when considered in connection with the following drawings, in which like reference numerals identify like elements.
FIG. 1 is an exemplary Architecture for Efficient Workload Deployment Using Containers and Unikernels.
FIG. 2 is an exemplary Architecture for Condensed Containerization.
FIG. 3 is an exemplary Library and Package Repository, depicting the Example Repository Lists of Libraries and Packages for Various Example Operating Systems.
DETAILED DESCRIPTION OF THE INVENTION
Referring to FIG. 1, an exemplary embodiment of the Architecture for Efficient Workload Deployment Using Containers and Unikernels concept is illustrated and include the following elements.
User Submission Interface [100] is an interface that a user may use to interface with the system. For example, the user may submit Applications/Services [101] to the Deployment Package Selector [106] through the User Submission Interface [100]. This may be accomplished, for example, manually or automatically, through a command-line interface (CLI), graphical user interface (GUI), application program interface (API), web browser interface, or any combination thereof.
Applications/Services [101] are software functionalities that can be reused for different purposes, and may also be coupled with policies that control their usage. For example, in an enterprise setting, Applications/Services [101] may be defined through enterprise architecture (EA) practices for organizing business processes and information technology (IT) infrastructure. Different enterprises may have different types of Applications/Services [101] with various application granularities, which are often also correlated with their size and duration. Applications/Services [101] that may be deployed in any state, including (but not limited to) as any combination of source code, binaries, functional elements, build instructions, container images, virtual machine (VM) images, or unikernel images. Applications/Services [101] may run in architectures that include, for example, any combination of:
    • a) Monolithic applications, which may be single-tiered software applications in which, for example, the user interface and data access code are combined into a single program from a single platform.
    • b) Services, which may follow service-orientation and service-oriented-architecture (SOA) design principles, and multitier (e.g., three-tier) architecture.
    • c) Microservices, which may support complex applications composed of many small, independent processes communicating with each other.
    • d) “Serverless” architectures (or other architectures based functional programming), which tends to abstract the underlying resources from the Applications/Services [101] developers, and are typically more focused on the completion and duration of each event, independent of those underlying resources.
User Specification Interface [102] is an interface that the user may use to interface with the system. For example, the user may select which packages they want to be included with the Applications/Services [101] through the User Specification Interface [102]. This may be accomplished, for example, manually or automatically, through a command-line interface (CLI), graphical user interface (GUI), application program interface (API), web browser interface, or any combination thereof. Some content in the User Specification Interface [102] may not require user engagement in order for the Applications/Services [101] to be deployed, and other content may require user engagement.
Example User Specification Interface (e.g. Web Browser Interface) [103], is one example instantiation of the User Specification Interface [102]. This example includes the Package Selection [104], which has an example list of packages for the user, with corresponding checkboxes the user can check, selecting, in this example, which packages are to be included with the Application/Service [101] deployments.
Add SSH (Secure Shell) Support [105] is an example checkbox the user may check to select the corresponding package to be deployed with the Applications/Services [101], as described in the Example User Specification Interface (e.g. Web Browser Interface) [103] example above.
Deployment Package Selector [106] keeps track of all the Applications/Services [101] and any User Specification Interface [102] content that the user submits, and selects the best Deployment Packages [107] for which to deploy them at any given time. The decision of which Deployment Packages [107] to deploy with may be based on any combination of (a) Application/Service [101] type, size, granularity, duration, estimated execution time, coupling, modularity; (b) technical fit between each Applications/Services [101] and Deployment Packages [107]; (c) User Specification Interface [102] content and any corresponding user specification; (d) resource availability in the Heterogeneous/Hybrid Server Cluster [120]; (e) resource utilization metrics gathered by the Global Resource Tracker [122]; (f) historical data, including the use of machine learning; (g) specifications for any licensed software; (h) security boundaries in the network and/or organization; (i) device driver and protocol library availabilities.
Bypassing and Debugging [117] is a path that the Deployment Package Selector [106] may also choose for the submitted Applications/Services [101] and any corresponding User Specification Interface [102] content from user specification to bypass the Deployment Packages [107], based on these same aforementioned decision points. This Bypassing and Debugging [117] path may also be used for debugging purposes, deploying the same Applications/Services [101] through the Bypassing and Debugging [117] path that were deployed through Deployment Packages [107].
Condensed Containerization [108], which is described in detail in FIG. 2, is a Deployment Package [107]. Condensed Containerization [108], a unique process that is part of this invention, is able to produce optimized (in terms of size, speed, resource utilization, and performance) container images from Applications/Services [101] by leveraging traditional containerization technologies.
Traditional container technologies, which are depicted by Containerization (Optional) [118] in FIG. 1 and Containerize Application and Libraries/Packages Using Scratch Container Image from Container Base Image Repository [219] in FIG. 2, may leverage server virtualization methods such as operating system (OS)-level virtualization, where the kernel of an operating system (OS) allows for multiple isolated user space instances, instead of just one. Some instances of this may include, but are not limited to, containers, virtualization engines (VEs), virtual private servers (VPS), jails, or zones, and/or any hybrid combination thereof. Some example available technologies for this containerization include chroot, Docker, Linux-VServer, lmctfy (“let me contain that for you”), LXC (Linux containers), Canonical LXD, OpenVZ (Open Virtuozzo), Parallels Virtuozzo Containers, Solaris Containers (and Solaris Zones), FreeBSD Jail, sysjail, WPARs (workload partitions), HP-UX Containers (SRP, secure resource partitions), iCore Virtual Accounts, Sandboxie, Spoon, and VMware ThinApp.
Specialized Unikernel Generation [109] is a Deployment Package [107]. Specialized Unikernel Generation [109], a unique process that is part of this invention, is able to produce optimized (in terms of most fitting to Applications/Services [101], for any combination of size, speed, resource utilization, and performance) unikernel images from Applications/Services [101] by leveraging traditional unikernel technologies.
Traditional unikernel technologies, which are depicted by Unikernel [114] in FIG. 1, may consist of single address space machine images, and may be constructed with the use of library operating systems (OS). Some example available technologies for this unikernel include ClickOS, Clive, Drawbridge, Graphene, HaLVM (Haskell Lightweight Virtual Machine), IncludeOS, LING, MirageOS, OSv, Rumprun, and runtime.js.
Traditionally, to deploy Applications/Services [101] as unikernels, the developers of those Applications/Services [101] would manually identify the minimum set of libraries corresponding to the operation system (OS) constructs for specific types of unikernels. These libraries would then be compiled with the Applications/Services [101] and any corresponding configuration code. Many times, device drivers would also be required for specific hardware on which the unikernel is to be deployed, as well as protocol libraries.
Application Requirement Determination [110] is a step in the Specialized Unikernel Generation [109] that determines the various requirements and dependencies for deployed Applications/Services [101] and any corresponding User Specification Interface [102] content. For example, this may be determining required and user-specified libraries and packages, similar to Application/Service Library and Package Requirement Determination [201] and User Specification Application/Service Library and Package Requirement Determination [208], both depicted in FIG. 2, respectively. Application Requirement Determination [110] may also determine which programming language(s) and software framework(s) are required. For Applications/Services [101] that are deployed as source code, the Application Requirement Determination [110] may also build the Applications/Services [101] into binaries to gather some or all of these requirements.
Library Operating System Matching [111], which determines which available library operating system is best suited for the deployed Applications/Services [101]. The selection factors may be taken from the requirements gathered from the Application Requirement Determination [110], and may also be combined with other factors such as resource utilization metrics gathered by the Global Resource Tracker [122] and User Specification Interface [102] content, matching the deployed Applications/Services [101] with available library operating systems in the Library Operating System Repository [112].
Library Operating System Repository [112] is the repository where the available library operating systems are stored.
Library Operating System Compiler [113] is a compiler that converts the deployed Applications/Services [101] into the Unikernel [114], using the library operating system selected by the Library Operating System Matching [111] from the Library Operating System Repository [112]. As part of the compilation, the Library Operating System Compiler [113] may also include specific language runtimes, any required device drivers and protocol libraries.
Application [115] and Library Operating System Runtime [116] are parts of the compiled product of the Unikernel [114]. The Application [115] is unique to each Applications/Services [101] deployment, and the Library Operating System Runtime [116] may be unique to each type of library operating system.
Orchestrator [119] may refer to the Application Management Engines (106), as part of the Elastic Compute Cloud Framework described in the utility patent application Elastic Compute Cloud Based on Underutilized Server Resources Using a Distributed Container System (U.S. patent application Ser. No. 14/640,569, filed on Mar. 6, 2015), as well as the Elastic Workload Orchestrator [107], as part of the Elastic Workload Orchestration Architecture described in the utility patent application Ubiquitous and Elastic Workload Orchestration Architecture of Hybrid Applications/Services on Hybrid Cloud (U.S. patent application Ser. No. 15/381,427, filed on Dec. 16, 2016), the entireties of which are incorporated by reference herein.
Heterogeneous/Hybrid Server Cluster [120] is a cluster that is formed from any hybrid combination of Server [121] resources, spanning, for example, on-premises data centers, private clouds, and public clouds.
Server [121], which may operate within a client-server architecture, responds to requests across a computer network to provide, or help to provide a networked service. The client or clients may run on the same physical or virtual Server [121], or may be connected to the Server [121] over a network. Some examples of Server [121] types include computing servers, application servers, database servers, web servers, file servers, mail servers, and game servers.
Global Resource Tracker [122], which keeps track of resource utilization metrics of each Server [121] in the Heterogeneous/Hybrid Server Cluster [120]. These resource utilization metrics may include, but are not limited to, capacity of its central processing unit (CPU), random-access memory (RAM), graphics processing unit (GPU), storage, network bandwidth, memory bandwidth, and central processing unit (CPU) cache. For example, resource utilization metrics may be gathered for, and differentiated between, physical servers, virtual machines (VMs), containers, unikernels, applications, and processes. These resource utilization metrics may be gathered from agents running on each Server [121], as well as with agentless approaches. Furthermore, the Global Resource Tracker [122] shares the resource utilization metrics with the (a) Deployment Package Selector [106], to assist in selecting the most fitting Deployment Package [107] for each Application/Service [101] deployment; (b) Condensed Containerization [108], for optimized container-based deployment; (c) Specialized Unikernel Generation [109], for optimized unikernel-based deployment; (d) Orchestrator [119], for selecting the most fitting Server [121] onto which to deploy in the Heterogeneous/Hybrid Server Cluster [120].
Referring to FIG. 2, an exemplary embodiment of the Architecture for Condensed Containerization [108] concept is illustrated and includes the following elements.
Library and Package Repository [200], as depicted in more detail in FIG. 3, may be comprised of lists corresponding to available libraries and corresponding packages, for each operating system. These lists may leverage any combination of local libraries and packages, those that that are in other designated repositories, and those that are in common public repositories. Library and Package Repository [200] assists with Application/Service Library and Package Requirement Determination [201], User Specification Application/Service Library and Package Requirement Determination [208], as well as Containerization [215].
Application/Service Library and Package Requirement Determination [201] are steps that determine all the library and corresponding package requirements based on the Applications/Services [101] deployed by the user. Application/Service Library and Package Requirement Determination [201] steps include:
    • a) Determine List of Shared Libraries by Running LDD (List Dynamic Dependencies) Utility on Application/Service [202], which, for example, involves running the LDD (list dynamic dependencies) Linux utility, which prints the shared objects (shared libraries) required by each Application/Service [101] or shared object. For Applications/Services [101] that are deployed as source code, the Application/Service Library and Package Requirement Determination [201] may also first build the Applications/Services [101] into binaries. Running the LDD (list dynamic dependencies) utility is an example, and may also be accomplished with a number of other utilities and tools for a number of operating systems. Determine List of Shared Libraries by Running LDD (List Dynamic Dependencies) Utility on Application/Service [202] may be assisted by the Library and Package Repository [200], for example, in providing library availability for various operating systems.
    • b) Determine (or Build) Corresponding Packages that Support Shared Libraries [203], which, for example, involves determining (or building) corresponding packages that support the shared libraries that are determined in the previous step of the Application/Service Library and Package Requirement Determination [201]. Determine (or Build) Corresponding Packages that Support Shared Libraries [203] may be assisted by the Library and Package Repository [200], for example, in matching libraries to packages for various operating systems.
All Available Shared Libraries and Packages [204] is an example list that illustrates, as an example, the Application/Service Library and Package Requirement Determination [201] step results. As described in FIG. 2, this is an Example List of Shared Libraries with Corresponding Packages for the Application/Service [205]. All Available Shared Libraries and Packages [204], as an example, contains lists of Shared Libraries [206], which are operating system libraries that are loaded by programs when they start, and Packages [207], which are operating system packages, for an example operating system. Similar lists may be generated for any operating system type and version, as depicted in FIG. 3 for the Library and Package Repository [200].
User Specification Application/Service Library and Package Requirement Determination [208] are steps that determine all the library and corresponding package requirements based on any User Specification Interface [102] content interfaced with by the user. User Specification Application/Service Library and Package Requirement Determination [208] steps include:
    • a) Translate User Specification Selection to Corresponding Libraries and Packages [209], which, for example, involves translating User Specification Interface [102] content that has interfaced with the user into corresponding Libraries [213] and Packages [214]. In one example, the user may check Add SSH (Secure Shell) Support [105] in the Package Selection [104] example of the User Specification Interface [102]. The Translate User Specification Selection to Corresponding Libraries and Packages [209] step would then translate this into appropriate Libraries [213] and Packages [214]. Translate User Specification Selection to Corresponding Libraries and Packages [209] may be assisted by the Library and Package Repository [200], for example, in matching User Specification Interface [102] content to available libraries, as well as matching libraries to Packages, for various operating systems.
Library and Package Dependencies and Sub-Dependencies from User Specification [210] is an example list that illustrates, as an example, the User Specification Application/Service Library and Package Requirement Determination [208] step results. As described in FIG. 2, this is an Example List of Libraries and Packages Corresponding to the User Specification Interface [211], which lists out SSH (Secure Shell) Support [212] based on the checked Add SSH (Secure Shell) Support [105] from the Package Selection [104] example of the User Specification Interface [102] content. Library and Package Dependencies and Sub-Dependencies from User Specification [210], as an example, contains lists of Libraries [213], which are operating system libraries, and Packages [214], which are operating system packages, for an example operating system. Similar lists may be generated for any operating system type and version, as well as for other example User Specification Interface [102] content, as depicted in FIG. 3 for the Library and Package Repository [200].
Containerization [215] are steps that are able to create the condensed container from the determined library and corresponding package requirements based on the Applications/Services [101] deployed by the user, as output from the Application/Service Library and Package Requirement Determination [201], as well as determine the library and corresponding package requirements based on any User Specification Interface [102] content interfaced with by the user, as output from the User Specification Application/Service Library and Package Requirement Determination [208], combined with the assistance of the Library and Package Repository [200]. Containerization [215] steps include:
    • a) Determine Most Fitting Compatible Operating System in Library and Package Repository [216], which involves determining the most optimal operating system fit for the deployed Applications/Services [101]. For example, the Determine Most Fitting Compatible Operating System in Library and Package Repository [216] step may search through the permutations of All Available Share Libraries and Packages [204] and Library Package Dependencies and Sub-Dependencies from User Specification [210] for every available operating system, to determine which operating system types and versions would support each deployed Application/Service [101]. If multiple operating systems support the deployed Application/Service [101] in this way, the latest or most stable type and version may be selected as the optimal choice. The user may also specify, for example through the User Specification Interface [102], a preference for specific operating system types and versions.
    • b) Gather All Required Libraries and Packages from Library and Package Repository [217], which involves gathering all the required libraries and packages from the Library and Package Repository [200]. These libraries and packages, required for the deployment of the Application/Service [101], correspond to the selected operating system determined by the Determine Most Fitting Compatible Operating System in Library and Package Repository [216] step.
    • c) Merge Required Libraries and Packages with Application/Service [218], which involves combining the libraries and packages required for the Application/Service [201] deployment, which were gathered in the previous step, with the Application/Service [201], merging everything that is required for the Application/Service [201] deployment.
    • d) Containerize Resulting Application/Service, Libraries, and Packages Using Minimal Container Image from Container Base Image Repository [219], which involves containerizing the resulting merge from the Merge Required Libraries and Packages with Application/Service [218] step. A minimal container image, which sometimes may be referred to as a “scratch” container, may be used for this. The minimal container image is a minimized version of a container base image that is may act as a minimal starting point for building containers, and can be pulled from the Container Base Image Repository [220].
The resulting container image from Containerization [215], encapsulating the deployed Application/Service [101] and all the required libraries and packages, may then be deployed to the Orchestrator [119], which then deploys it to various Server [120] resources.
Referring to FIG. 3, an exemplary embodiment of the Library and Package Repository [200], depicting the Example Repository Lists of Libraries and Packages for Various Example Operating Systems [300] concept is illustrated and include the following elements.
FIG. 3 depicts the Example Repository Lists of Libraries and Packages for Various Example Operating Systems [300].
Some example operating systems are depicted as [301], specifically Ubuntu α [301], Ubuntu β [301], and CentOS γ [301], which, for the purposes of this example, are different types and versions of Linux.
FIG. 3 also depicts the Example Repository List of All Available Libraries and Packages, for Each Operating System [302], which comprises All Available Libraries and Packages [204], Shared Libraries [206], and Packages [207], for each operating system.
FIG. 3 also depicts the Example Repository List of All Library and Package Dependencies and Sub-Dependencies from User Specifications, for Each Operating System [303], which comprises All Library and Package Dependencies and Sub-Dependencies from User Specification [210], SSH (Secure Shell) Support [212], Libraries [213], and Packages [214], for each operating system.
Implementations of the system described herein can include a plurality of software processing modules stored in a memory and executed on a processor. By way of illustration, the program modules can be in the form of one or more suitable programming languages, which are converted to machine language or object code to allow the processor or processors to execute the instructions. The software can be in the form of one or more standalone applications implemented in a suitable programming language or framework.
Communication among devices (e.g., between a deployment package provider and a device on which the package is deployed) can take place over media such as standard telephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25), broadband connections (ISDN, Frame Relay, ATM), wireless links (802.11 (Wi-Fi), Bluetooth, GSM, CDMA, etc.), for example. The network can carry TCP/IP protocol communications, and HTTP/HTTPS requests made by a web browser, and the connection between the user devices and servers can be communicated over such TCP/IP networks.
Method steps of the techniques described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. Method steps can also be performed by, and the modules can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Modules can refer to portions of the computer program and/or the processor/special circuitry that implements that functionality.
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. One or more memories can store instructions that, when executed by a processor, form the modules and other components described herein and perform the functionality associated with the components. The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.
It should also be noted that implementations of the systems and methods can be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture, including non-transitory storage mediums. The program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).
The terms and expressions employed herein are used as terms and expressions of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof. In addition, having described certain implementations in the present disclosure, it will be apparent to those of ordinary skill in the art that other implementations incorporating the concepts disclosed herein can be used without departing from the spirit and scope of the invention. The features and functions of the various implementations can be arranged in various combinations and permutations, and all are considered to be within the scope of the disclosed invention. Accordingly, the described implementations are to be considered in all respects as illustrative and not restrictive. The configurations, materials, and dimensions described herein are also intended as illustrative and in no way limiting. Similarly, although physical explanations have been provided for explanatory purposes, there is no intent to be bound by any particular theory or mechanism, or to limit the claims in accordance therewith.

Claims (16)

The invention claimed is:
1. A computer-implemented method for creating a condensed container, the method comprising:
receiving an instruction to create a container, the container including one or more applications and services;
identifying one or more shared libraries and corresponding packages required for executing the one or more applications and services;
determining an optimal operating system for the container by:
receiving a list comprising a plurality of available operating systems, each available operating system comprising a type and a version, wherein the list specifies, for each available operating system, a permutation of shared libraries and packages provided for the available operating system;
searching through the permutations to identify a subplurality of the available operating systems that support the one or more applications and services; and
if the subplurality of the available operating systems includes more than one of the available operating systems, selecting one of the subplurality of available operating systems as the optimal operating system based on (i) most recent type and/or version of operating system, (ii) most stable type and/or version of operating system, or (ii) user preference for type and/or version of operating system;
selecting versions of the shared libraries and the corresponding packages that correspond to the optimal operating system;
forming a merge of the selected versions of the shared libraries and corresponding packages with the applications and services; and
creating the container based on the merge.
2. The method of claim 1, further comprising:
identifying one or more user-selected libraries and corresponding packages based on functionality specified by a user for inclusion in the container;
selecting versions of the user-selected libraries and corresponding packages that correspond to the optimal operating system; and
including the selected versions of the user-selected libraries and corresponding packages in the container.
3. The method of claim 1, further comprising providing a user interface allowing the user to select functionality for inclusion in the container.
4. The method of claim 1, wherein creating the container comprises containerizing the merge using a minimized container based image.
5. The method of claim 1, further comprising deploying the container to one or more servers.
6. The method of claim 1, further comprising:
selecting a deployment package from a plurality of deployment packages to deploy one or more applications and services, wherein the selecting is based on at least one factor associated with the applications and services, specified user content, deployment resources, historical data, licensed software specifications, organization or network security boundaries, or driver or protocol library availability; and
deploying the one or more applications and services using the selected deployment package.
7. The method of claim 6, wherein a particular factor associated with the applications and services comprises type, size, granularity, duration, estimated execution time, coupling, modularity, or technical fit between each application or service and available deployment packages.
8. The method of claim 6, wherein a particular factor associated with the deployment resources comprises server cluster resource availability or resource utilization metrics.
9. A system for creating a condensed container, the system comprising:
at least one memory for storing computer-executable instructions; and
at least one processor for executing the instructions stored on the at least one memory, wherein execution of the instructions programs the at least one processor to perform operations comprising:
receiving an instruction to create a container, the container including one or more applications and services;
identifying one or more shared libraries and corresponding packages required for executing the one or more applications and services;
determining an optimal operating system for the container by:
receiving a list comprising a plurality of available operating systems, each available operating system comprising a type and a version, wherein the list specifies, for each available operating system, a permutation of shared libraries and packages provided for the available operating system;
searching through the permutations to identify a subplurality of the available operating systems that support the one or more applications and services; and
if the subplurality of the available operating systems includes more than one of the available operating systems, selecting one of the subplurality of available operating systems as the optimal operating system based on (i) most recent type and/or version of operating system, (ii) most stable type and/or version of operating system, or (ii) user preference for type and/or version of operating system;
selecting versions of the shared libraries and the corresponding packages that correspond to the optimal operating system;
forming a merge of the selected versions of the shared libraries and corresponding packages with the applications and services; and
creating the container based on the merge.
10. The system of claim 9, wherein the operations further comprise:
identifying one or more user-selected libraries and corresponding packages based on functionality specified by a user for inclusion in the container;
selecting versions of the user-selected libraries and corresponding packages that correspond to the optimal operating system; and
including the selected versions of the user-selected libraries and corresponding packages in the container.
11. The system of claim 9, wherein the operations further comprise providing a user interface allowing the user to select functionality for inclusion in the container.
12. The system of claim 9, wherein creating the container comprises containerizing the merge using a minimized container based image.
13. The system of claim 9, wherein the operations further comprise deploying the container to one or more servers.
14. The system of claim 9, wherein the operations further comprise:
selecting a deployment package from a plurality of deployment packages to deploy one or more applications and services, wherein the selecting is based on at least one factor associated with the applications and services, specified user content, deployment resources, historical data, licensed software specifications, organization or network security boundaries, or driver or protocol library availability; and
deploying the one or more applications and services using the selected deployment package.
15. The system of claim 14, wherein a particular factor associated with the applications and services comprises type, size, granularity, duration, estimated execution time, coupling, modularity, or technical fit between each application or service and available deployment packages.
16. The system of claim 14, wherein a particular factor associated with the deployment resources comprises server cluster resource availability or resource utilization metrics.
US15/662,649 2016-08-10 2017-07-28 Efficient workload deployment using containers and unikernels Active 2037-09-28 US10402187B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/662,649 US10402187B2 (en) 2016-08-10 2017-07-28 Efficient workload deployment using containers and unikernels

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201662373147P 2016-08-10 2016-08-10
US15/662,649 US10402187B2 (en) 2016-08-10 2017-07-28 Efficient workload deployment using containers and unikernels

Publications (2)

Publication Number Publication Date
US20180046446A1 US20180046446A1 (en) 2018-02-15
US10402187B2 true US10402187B2 (en) 2019-09-03

Family

ID=61159017

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/662,649 Active 2037-09-28 US10402187B2 (en) 2016-08-10 2017-07-28 Efficient workload deployment using containers and unikernels

Country Status (1)

Country Link
US (1) US10402187B2 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190227978A1 (en) * 2019-04-02 2019-07-25 Intel Corporation Edge component computing system having integrated faas call handling capability
US20200044945A1 (en) * 2018-08-03 2020-02-06 Quanta Computer Inc. Systems and methods for cloud service provisioning using software containers
US11789774B2 (en) 2021-02-22 2023-10-17 International Business Machines Corporation Optimization of workload scheduling in a distributed shared resource environment
US11880282B2 (en) 2021-09-15 2024-01-23 Trilio Data, Inc. Container-based application data protection method and system
US12073232B2 (en) 2022-09-28 2024-08-27 Red Hat, Inc. Unikernel for webassembly containers

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10579407B2 (en) * 2016-10-28 2020-03-03 ShieldX Networks, Inc. Systems and methods for deploying microservices in a networked microservices system
US10169028B2 (en) * 2016-12-13 2019-01-01 Ciena Corporation Systems and methods for on demand applications and workflow management in distributed network functions virtualization
US10592215B1 (en) * 2017-08-17 2020-03-17 NanoVMs, Inc. Unikernel cross-compilation
US20220360600A1 (en) * 2017-11-27 2022-11-10 Lacework, Inc. Agentless Workload Assessment by a Data Platform
JP7545895B2 (en) 2018-04-11 2024-09-05 コーネル ユニヴァーシティ Method and system for improving performance and isolation of software containers - Patents.com
US10628177B1 (en) * 2018-04-11 2020-04-21 NanoVMs, Inc. Unikernel provisioning
US11989569B2 (en) * 2018-04-11 2024-05-21 NanoVMs, Inc. Unikernel provisioning
CN108897550B (en) * 2018-06-08 2021-07-30 江苏中威科技软件系统有限公司 System and method for realizing program deployment, access and upgrade through packaging browser
US10901798B2 (en) * 2018-09-17 2021-01-26 International Business Machines Corporation Dependency layer deployment optimization in a workload node cluster
US10838702B2 (en) 2018-11-09 2020-11-17 International Business Machines Corporation Analyzing and optimizing container images in cloud computing
US10983847B2 (en) * 2019-02-15 2021-04-20 Red Hat, Inc. Dynamically loadable unikernel binaries
US10990365B2 (en) * 2019-02-26 2021-04-27 Red Hat, Inc. Container image building using dependency container images
US11036545B2 (en) * 2019-03-15 2021-06-15 Intel Corporation Graphics systems and methods for accelerating synchronization using fine grain dependency check and scheduling optimizations based on available shared memory space
KR102099118B1 (en) * 2019-04-29 2020-04-09 래블업(주) A method of providing a session container with a plurality of libraries requested by a user
WO2021021126A1 (en) 2019-07-30 2021-02-04 Hewlett-Packard Development Company, L.P. Hash map of executable control blocks
US10838716B1 (en) * 2019-11-19 2020-11-17 Capital One Services, Llc Automated dependency detection and response
US10877750B1 (en) * 2019-12-17 2020-12-29 Datacore Software Corporation Containerized storage microservice with direct connection to requesting application container
US11126571B2 (en) * 2020-01-30 2021-09-21 Sap Se Using unikernels to design high performance integration workflows
CN111580834B (en) * 2020-04-10 2021-10-29 北京城市网邻信息技术有限公司 Method and device for generating application installation package
US11321107B1 (en) * 2020-05-13 2022-05-03 NanoVMs, Inc. Running arbitrary binaries as unikernels on embedded processors
US12086262B2 (en) * 2021-07-28 2024-09-10 Red Hat, Inc. Secure container image builds
US11782689B1 (en) * 2022-03-31 2023-10-10 Spectrum Comm Inc. System, method, and choreographic framework for the design, development, deployment, and management of distributed software and computing systems

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060230395A1 (en) * 2005-03-16 2006-10-12 Microsoft Corporation Embedded device update service
US20060277537A1 (en) * 2005-06-01 2006-12-07 International Business Machines Corporation Deployment of containers and container extensions in a modular code runtime platform through a runtime platform extension point
US20080127154A1 (en) * 2006-11-28 2008-05-29 Ulrich Drepper Methods and systems for optimization in a run-time environment
US20090070752A1 (en) * 2007-09-06 2009-03-12 International Business Machines Corporation Method and system for optimization of an application
US7568195B2 (en) * 2003-12-16 2009-07-28 Microsoft Corporation Determining a maximal set of dependent software updates valid for installation
US20100107151A1 (en) * 2008-10-24 2010-04-29 Oracle International Corporation Method and system for implementing performance kits
US20110265164A1 (en) * 2010-04-26 2011-10-27 Vmware, Inc. Cloud platform architecture
US20110265081A1 (en) * 2010-04-26 2011-10-27 Vmware, Inc. Droplet execution engine for dynamic server application deployment
US8707294B2 (en) * 2009-04-07 2014-04-22 Oracle International Corporation Model for system-wide application extension
US20140215452A1 (en) * 2013-01-28 2014-07-31 Red Hat, Inc. Deployment Optimization for High Availability in a Multi-Tenant Platform-as-a-Service (PaaS) System
US8832681B1 (en) * 2008-08-29 2014-09-09 Adobe Systems Incorporated Bundled mobile/desktop application package
US9210141B2 (en) * 2009-11-25 2015-12-08 Novell, Inc System and method for providing scorecards to visualize services in an intelligent workload management system
US20160147529A1 (en) * 2014-11-20 2016-05-26 Red Hat, Inc. Source Code Management for a Multi-Tenant Platform-as-a-Service (PaaS) System
US20170364377A1 (en) * 2016-06-15 2017-12-21 International Business Machines Corporation Specialized micro-hypervisors for unikernels
US20170364794A1 (en) * 2016-06-20 2017-12-21 Telefonaktiebolaget Lm Ericsson (Publ) Method for classifying the payload of encrypted traffic flows
US20180025160A1 (en) * 2016-07-21 2018-01-25 International Business Machines Corporation Generating containers for applications utilizing reduced sets of libraries based on risk analysis
US9916233B1 (en) * 2015-03-27 2018-03-13 Amazon Technologies, Inc. Using containers for update deployment
US20180074852A1 (en) * 2016-09-14 2018-03-15 Salesforce.Com, Inc. Compact Task Deployment for Stream Processing Systems
US9928059B1 (en) * 2014-12-19 2018-03-27 Amazon Technologies, Inc. Automated deployment of a multi-version application in a network-based computing environment
US10007509B1 (en) * 2015-12-08 2018-06-26 Amazon Technologies, Inc. Container handover for device updates

Patent Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7568195B2 (en) * 2003-12-16 2009-07-28 Microsoft Corporation Determining a maximal set of dependent software updates valid for installation
US20060230395A1 (en) * 2005-03-16 2006-10-12 Microsoft Corporation Embedded device update service
US20060277537A1 (en) * 2005-06-01 2006-12-07 International Business Machines Corporation Deployment of containers and container extensions in a modular code runtime platform through a runtime platform extension point
US20080127154A1 (en) * 2006-11-28 2008-05-29 Ulrich Drepper Methods and systems for optimization in a run-time environment
US20090070752A1 (en) * 2007-09-06 2009-03-12 International Business Machines Corporation Method and system for optimization of an application
US8832681B1 (en) * 2008-08-29 2014-09-09 Adobe Systems Incorporated Bundled mobile/desktop application package
US20100107151A1 (en) * 2008-10-24 2010-04-29 Oracle International Corporation Method and system for implementing performance kits
US8707294B2 (en) * 2009-04-07 2014-04-22 Oracle International Corporation Model for system-wide application extension
US9210141B2 (en) * 2009-11-25 2015-12-08 Novell, Inc System and method for providing scorecards to visualize services in an intelligent workload management system
US20110265164A1 (en) * 2010-04-26 2011-10-27 Vmware, Inc. Cloud platform architecture
US20110265081A1 (en) * 2010-04-26 2011-10-27 Vmware, Inc. Droplet execution engine for dynamic server application deployment
US20140215452A1 (en) * 2013-01-28 2014-07-31 Red Hat, Inc. Deployment Optimization for High Availability in a Multi-Tenant Platform-as-a-Service (PaaS) System
US20160147529A1 (en) * 2014-11-20 2016-05-26 Red Hat, Inc. Source Code Management for a Multi-Tenant Platform-as-a-Service (PaaS) System
US9928059B1 (en) * 2014-12-19 2018-03-27 Amazon Technologies, Inc. Automated deployment of a multi-version application in a network-based computing environment
US9916233B1 (en) * 2015-03-27 2018-03-13 Amazon Technologies, Inc. Using containers for update deployment
US10007509B1 (en) * 2015-12-08 2018-06-26 Amazon Technologies, Inc. Container handover for device updates
US20170364377A1 (en) * 2016-06-15 2017-12-21 International Business Machines Corporation Specialized micro-hypervisors for unikernels
US9886303B2 (en) * 2016-06-15 2018-02-06 International Business Machines Corporation Specialized micro-hypervisors for unikernels
US20170364794A1 (en) * 2016-06-20 2017-12-21 Telefonaktiebolaget Lm Ericsson (Publ) Method for classifying the payload of encrypted traffic flows
US20180025160A1 (en) * 2016-07-21 2018-01-25 International Business Machines Corporation Generating containers for applications utilizing reduced sets of libraries based on risk analysis
US20180074852A1 (en) * 2016-09-14 2018-03-15 Salesforce.Com, Inc. Compact Task Deployment for Stream Processing Systems

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
Balogh et al., "Model-Based Optimization of Enterprise Application and Service Deployment", 2005, Springer-Verlag Berlin Heidelberg (Year: 2005). *
Inagaki et al., "Container Management as Emerging Workload for Operating Systems", 2016, IEEE (Year: 2016). *
Kivity et al., "Optimizing the Operating System for Virtual Machines", Jun. 2014, USENIX (Year: 2014). *
Madhavapeddy et al., "Unikernels: Rise of the Virtual Library Operating System", Nov. 2013, ACM Queue, vol. 11 Issue 11 (Year: 2013). *
Milanovic et al., "Service-Oriented Operating System: A Key Element in Improving Service Availability", 2007, Springer-Verlag Berlin Heidelberg (Year: 2007). *
Yi et al., "Design and Implementation of the Container Terminal Operating System Based on Service-Oriented Architecture", 2008, IEEE (Year: 2008). *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200044945A1 (en) * 2018-08-03 2020-02-06 Quanta Computer Inc. Systems and methods for cloud service provisioning using software containers
US10791039B2 (en) * 2018-08-03 2020-09-29 Quanta Computer Inc. Systems and methods for cloud service provisioning using software containers
US20190227978A1 (en) * 2019-04-02 2019-07-25 Intel Corporation Edge component computing system having integrated faas call handling capability
US11055256B2 (en) * 2019-04-02 2021-07-06 Intel Corporation Edge component computing system having integrated FaaS call handling capability
US11650951B2 (en) 2019-04-02 2023-05-16 Intel Corporation Edge component computing system having integrated FaaS call handling capability
US11789774B2 (en) 2021-02-22 2023-10-17 International Business Machines Corporation Optimization of workload scheduling in a distributed shared resource environment
US11880282B2 (en) 2021-09-15 2024-01-23 Trilio Data, Inc. Container-based application data protection method and system
US12073232B2 (en) 2022-09-28 2024-08-27 Red Hat, Inc. Unikernel for webassembly containers

Also Published As

Publication number Publication date
US20180046446A1 (en) 2018-02-15

Similar Documents

Publication Publication Date Title
US10402187B2 (en) Efficient workload deployment using containers and unikernels
US11449365B2 (en) Ubiquitous and elastic workload orchestration architecture of hybrid applications/services on hybrid cloud
Younge et al. A tale of two systems: Using containers to deploy HPC applications on supercomputers and clouds
US10303450B2 (en) Systems and methods for a policy-driven orchestration of deployment of distributed applications
US11537458B2 (en) Reducing service disruptions in a micro-service environment
US11422797B1 (en) Using graph partitioning for software decomposition during modernization processes
US20140245319A1 (en) Method for enabling an application to run on a cloud computing system
KR101594915B1 (en) Method for performing parallel programing in manycore cluster system and manycore cluster sytem
US11119675B2 (en) Polymorphism and type casting in storage volume connections
De Alfonso et al. Infrastructure deployment over the cloud
US10862765B2 (en) Allocation of shared computing resources using a classifier chain
US11934877B2 (en) Workflow resource management for cloud computing services
US11853753B1 (en) Detection, presentation, and resolution of bottlenecks in monolith decomposition
Mehta et al. Calvin constrained—A framework for IoT applications in heterogeneous environments
Petrosyan et al. Serverless high-performance computing over cloud
Denninnart et al. Efficiency in the serverless cloud paradigm: A survey on the reusing and approximation aspects
Harichane et al. KubeSC‐RTP: Smart scheduler for Kubernetes platform on CPU‐GPU heterogeneous systems
US20210191768A1 (en) Computing task scheduling in a computer system utilizing efficient attributed priority queues
US20230185625A1 (en) Workload characterization-based capacity planning for cost-effective and high-performance serverless execution environment
EP4152148A1 (en) Container image file optimization
US20230055276A1 (en) Efficient node identification for executing cloud computing workloads
Spillner et al. Analysis of overhead and profitability in nested cloud environments
US20240362080A1 (en) Power management for virtual network functions or containerized network functions
US20240056853A1 (en) Edge service deployment with network slice invocation
Alves et al. Leveraging Shared Accelerators in Kubernetes Clusters with rOpenCL

Legal Events

Date Code Title Description
AS Assignment

Owner name: JISTO INC., MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TUROVSKY, ANDREY;BIBERMAN, ALEKSANDR;REEL/FRAME:043231/0479

Effective date: 20170807

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

AS Assignment

Owner name: TRILIO DATA INC., MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:JISTO INC.;REEL/FRAME:049395/0545

Effective date: 20190606

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

Year of fee payment: 4