US20200142752A1 - Physical partitioning of computing resources for server virtualization - Google Patents

Physical partitioning of computing resources for server virtualization Download PDF

Info

Publication number
US20200142752A1
US20200142752A1 US16/730,430 US201916730430A US2020142752A1 US 20200142752 A1 US20200142752 A1 US 20200142752A1 US 201916730430 A US201916730430 A US 201916730430A US 2020142752 A1 US2020142752 A1 US 2020142752A1
Authority
US
United States
Prior art keywords
memory
processors
peripheral device
processor
computer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US16/730,430
Inventor
Sape Mullender
David Richard Barach
Jim Mckie
Peter Bosch
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cisco Technology Inc
Original Assignee
Cisco Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cisco Technology Inc filed Critical Cisco Technology Inc
Priority to US16/730,430 priority Critical patent/US20200142752A1/en
Assigned to CISCO TECHNOLOGY, INC. reassignment CISCO TECHNOLOGY, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MULLENDER, SAPE, BOSCH, PETER, MCKIE, JIM, BARACH, DAVID RICHARD
Publication of US20200142752A1 publication Critical patent/US20200142752A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061Partitioning or combining of resources
    • G06F9/5077Logical partitioning of resources; Management or configuration of virtualized resources
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061Improving I/O performance
    • G06F3/0613Improving I/O performance in relation to throughput
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0644Management of space entities, e.g. partitions, extents, pools
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662Virtualisation aspects
    • G06F3/0664Virtualisation aspects at device level, e.g. emulation of a storage device or system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/4401Bootstrapping
    • G06F9/4406Loading of operating system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/4401Bootstrapping
    • G06F9/4411Configuring for operating with peripheral devices; Loading of device drivers
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533Hypervisors; Virtual machine monitors
    • G06F9/45558Hypervisor-specific management and integration aspects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061Partitioning or combining of resources
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533Hypervisors; Virtual machine monitors
    • G06F9/45558Hypervisor-specific management and integration aspects
    • G06F2009/45579I/O management, e.g. providing access to device drivers or storage
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533Hypervisors; Virtual machine monitors
    • G06F9/45558Hypervisor-specific management and integration aspects
    • G06F2009/45583Memory management, e.g. access or allocation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the subject matter of this disclosure relates in general to the field of server virtualization, and more specifically to techniques for physically partitioning resources of a computing system via a baseboard management controller of the system.
  • Virtualization is an important technology used in a modern data center. Virtualization can allow an operator of the data center to consolidate workloads; raise utilization levels; reduce operating, capital, space, power, and cooling expenses; move workloads dynamically within a virtualization pool to increase the flexibility to take servers offline or bring new ones online; manage the relationship of virtual instances to physical machines to optimize performance and maintain service levels; scale existing applications or deploy new ones by creating more virtual instances using an existing pool of resources; and deploy high-availability and disaster-recovery features of a virtualization platform to overcome localized and geographic failures, among other benefits.
  • hypervisor-based virtualization software emulates the hardware of a physical computer so that an entire, unmodified operating system can run with the emulated or virtual environment (i.e., a virtual machine (VM)).
  • VM virtual machine
  • a single physical server can run several VMs at once, and a hypervisor or virtual machine monitor (VMM) can manage the VMs and allocate the resources of the server among them.
  • VMM virtual machine monitor
  • Hypervisor-based virtualization can introduce additional overhead because a server implementing this approach must replicate true hardware behaviors for each VM running on the server.
  • Containers do not require an additional layer of virtual hardware.
  • Container-based virtualization therefore typically does not allow a server to run different operating systems or different versions of the same operating system.
  • FIG. 1 illustrates an example of a network environment in accordance with an embodiment
  • FIG. 2A and FIG. 2B illustrate examples of computing systems in accordance with some embodiments
  • FIG. 3 illustrates an example of a process for physically partitioning resources of a computing system via a baseboard management controller of the system in accordance with an embodiment
  • FIG. 4A , FIG. 4B , and FIG. 4C illustrate examples of physically partitioned computing systems in accordance with some embodiments.
  • a baseboard management controller can physically partition the computing resources (e.g., processing, memory, input/output (I/O), and/or storage) of a server into two or more resource groups for concurrently running a different operating system and one or more applications per resource group.
  • computing resources e.g., processing, memory, input/output (I/O), and/or storage
  • the BMC can allocate at least a first processor (e.g., a pair of processors of a four-socket server, a multi-core processor, a single core of a multi-core processor, etc.) of a physical host to a first resource group and a second processor of the physical host to a second resource group.
  • a first processor e.g., a pair of processors of a four-socket server, a multi-core processor, a single core of a multi-core processor, etc.
  • the BMC can load one or more boot images (e.g., basic input/output system (BIOS), Unified Extensible Firmware Interface (UEFI) boot manager, boot loader, bootstrap, or other software/firmware executed prior to loading of an operating system) and/or other configuration data from a storage area network (SAN) (or other remote storage) or a storage device embedded on the physical host (e.g., read-only memory (ROM), flash memory, or other non-volatile memory) for initializing the hardware of the physical host.
  • boot images e.g., basic input/output system (BIOS), Unified Extensible Firmware Interface (UEFI) boot manager, boot loader, bootstrap, or other software/firmware executed prior to loading of an operating system
  • SAN storage area network
  • ROM read-only memory
  • flash memory or other non-volatile memory
  • the BMC/boot image(s) can partition the memory of the physical host into at least a first memory range for exclusive use by the first processor and a second memory range for exclusive use by the second processor.
  • the first memory range can comprise a first set of (one or more) dual in-line memory modules (DIMMs) mounted on the physical host and the second memory range can comprise a second set of (one or more) DIMMs mounted on the physical host.
  • the BMC/boot image(s) can generate a first memory map that maps the first memory range to the first set of DIMMs and a second memory map that maps the second memory range to the second set of DIMMs.
  • the BMC/boot image(s) can limit access to the memory controller(s) of the first set of DIMMs to the first processor and limit access to the memory controller(s) of the second set of DIMMs to the second processor.
  • the BMC/boot image(s) can also distribute physical or virtual peripheral devices of the physical host between the first processor and the second processor.
  • the BMC/boot image(s) can expose/hide one or more I/O ports from one of the resources groups.
  • the BMC/boot image(s) can generate a Peripheral Component Interconnect Express (PCIe) address memory map for one of the resource groups that gives that resource group access to a particular peripheral device and denies access to a different peripheral device.
  • PCIe Peripheral Component Interconnect Express
  • the BMC/boot image(s) can configure a peripheral device to be a bus master, and that peripheral device can act on I/O requests from a particular processor and ignore I/O requests from a different processor.
  • FIG. 1 illustrates an example of a network environment 100 in accordance with an embodiment.
  • a network environment 100 there can be additional or fewer nodes, devices, links, networks, or components in similar or alternative configurations.
  • Various embodiments may include different numbers and/or types of clients, networks, nodes, cloud components, servers, software components, devices, virtual or physical resources, configurations, topologies, services, appliances, deployments, network devices, etc.
  • the illustrations and examples provided in the present disclosure are for conciseness and clarity.
  • the network environment 100 includes storage area networks (SAN) 102 a and 102 b (collectively, “ 102 ”), local area network (LAN) 104 , management network 106 , network devices 108 a and 108 b (collectively, “ 108 ”), and server chassis 110 .
  • Server chassis 110 is a computing infrastructure device used to interconnect servers in various form factors (e.g., rack servers, blade servers, and other high-density servers) with other network elements.
  • Server chassis 110 can provide power, cooling, connectivity, and mechanical support for servers, such as servers 112 a and 112 b (collectively, “ 112 ”) and network devices, such as switches, routers, network appliances (e.g., load balancers, deep packet inspectors, firewalls, etc.), port extenders (e.g., port extenders 114 a and 114 b (collectively, “ 114 ”)), etc.
  • An example of an implementation of server chassis 110 is Cisco Unified Computing SystemTM (Cisco UCS®) Blade Server Chassis, provided by Cisco Systems®, Inc. of San Jose, Calif.
  • Server chassis 110 includes a number of slots (e.g., 8 half-width slots, 4 full-width slots, or other capacities) for receiving servers 112 .
  • Server chassis 110 can reduce the number of physical components and the amount of cabling relative to conventional rack or blade systems, integrate with existing infrastructure for centralized management, and operate more efficiently with respect to energy consumption than conventional systems.
  • server 112 a is a half-width or half-slot server and server 112 b is a full-width or full-slot server.
  • Other embodiments may utilize servers having other types of form factors, including some embodiments with servers that do not require a chassis.
  • various embodiments can include a server that is a standalone device communicatively coupled to server chassis 110 or to one or more network devices 108 .
  • Various types of interconnections and buses can provide the communicative coupling, including any wired or wireless interconnection line, network, connection, bundle, single bus, multiple buses, crossbar network, single-stage network, multi-stage network, or other conduction medium operable to carry data between parts of a computing device or between computing devices.
  • Half-slot server 112 a includes network adapter 116 a , baseboard management controller (BMC) 118 a , processing subsystem 120 a , memory subsystem 122 a , I/O subsystem 124 a , local storage subsystem 126 a , and boot subsystem 128 a .
  • BMC baseboard management controller
  • Network adapter 116 a e.g., a network interface controller or card (NIC), network adapter, LAN adapter, etc.
  • NIC network interface controller
  • LAN adapter e.g., a network interface controller or card (NIC), network adapter, LAN adapter, etc.
  • server 112 a connects server 112 a to other physically separate and discrete network elements (e.g., network adapters 116 b and 116 c , port extenders 114 , network devices 108 , SANs 102 , LAN 104 , management network 106 , etc.) and logically separate elements of server 112 a (e.g., virtual machines, containers, or other partitions).
  • network adapters 116 b and 116 c e.g., a network interface controller or card (NIC), network adapter, LAN adapter, etc.
  • port extenders 114 e.g., network devices 108 , SANs 102 , LAN 104 , management
  • server 112 a includes the above subsystems for purposes of simplicity and clearness.
  • BMC 118 a monitors and manages the physical state of server 112 a .
  • BMC 118 a includes a specialized service processor (not shown) and firmware (not shown) to provide management and monitoring capabilities independently from processing subsystem 120 a .
  • BMC 118 a is reachable even when processing subsystem 120 a is powered off or non-operational.
  • BMC 118 a supports the standards defined in the Intelligent Platform Management Interface (IPMI) specification.
  • IPMI Intelligent Platform Management Interface
  • An example of an implementation of BMC 118 a is the Cisco® Integrated Management Controller (CIMC). CIMC is compliant with the IPMI specification but also provides additional functionality for providing unified monitoring and management of multiple computing systems.
  • Diagnostic and health monitoring features provided with CIMC include support for Simple Network Management Protocol (SNMP); extensible mark-up language (XML) application programming interface (API) event subscription and configurable alerts; system event logging; audit logging; monitoring of field-replaceable units (FRUs), hard disk drive (HDD) faults, dual inline memory module (DIMM) faults, NIC media access control (MAC) address, CPU, and thermal faults; configurable alerts and thresholds; watchdog timer; redundant array of independent disks (RAID) configuration and monitoring; predictive failure analysis of HDD and DIMM; support for converged network adapters (CNAs); and support for Network Time Protocol (NTP).
  • SNMP Simple Network Management Protocol
  • XML extensible mark-up language
  • API application programming interface
  • FRUs field-replaceable units
  • HDD hard disk drive
  • DIMM dual inline memory module
  • MAC media access control
  • RAID redundant array of independent disks
  • predictive failure analysis of HDD and DIMM support for converged
  • CIMC can operate in a standalone mode to provide users with full control of the server, allowing an administrator to perform server management tasks including powering on, powering off, power-cycling, resetting, and shutting down the server; toggling the locator light-emitting diode (LED); configuring the server boot order; viewing server properties and sensors; configuring out-of-band storage; managing remote presence; managing firmware; creating and managing local user accounts and enabling authentication through Active Directory and Lightweight Directory Access Protocol (LDAP); configuring network-related settings, including NIC properties, Internet Protocol (IP) version 4 (IPv4), IP version 6 (IPv6), virtual local area networks (VLANs), and network security; configure communication services, including Hypertext Transfer Protocol (HTTP), secure shell (SSH), and IPMI over LAN; managing certificates; configuring platform event filters; and monitoring faults, alarms, and server status.
  • IP Internet Protocol
  • IPv6 IP version 6
  • VLANs virtual local area networks
  • IPMI virtual local area networks
  • CIMC may also provide features such as a hypertext mark-up language version 5 (HTMLS) and keyboard, video, and mouse (KVM) user interface (UI); Redfish support; and XML API transactional support.
  • HTMLS and KVM can provide users with a simplified UI, and can eliminate the need for Java to use CIMC.
  • Redfish is an open industry standard specification and schema that specifies a restful stateful transfer (REST) interface and uses Javascript Object Notation (JSON) and Open Data Protocol (OData) to help customers integrate solutions within their existing tool chains.
  • XML API transactional support enables configuration of multiple managed objects in a single transaction, allowing for quicker, simpler deployments.
  • BMC 118 a can perform configuration and management services while server 112 a is in a low-power state, such as a standby state.
  • processing subsystem 120 a may require server 112 a to be in a relatively high power state.
  • a low-power state may include a state where server 112 a is not completely powered on and does not provide all or substantially all of its full functionality
  • a high-power state is a state where server 112 a is powered on and provides all or substantially all of its capabilities, less capabilities that are specifically disabled for purposes of management and configuration.
  • Processing subsystem 120 a connects to other elements of server 112 a via one or more interconnects or buses, and can directly perform instructions stored in memory subsystem 122 a and indirectly perform instructions stored in local storage subsystem 126 a , SANs 102 , and/or other memory locations.
  • Processing subsystem 120 a can include any combination of hardware, software, and/or firmware providing programmable logic. Examples of implementations of processing subsystem 120 a include the Advanced RISC Machine (ARM) architecture provided by ARM Holdings plc of Cambridge, England, United Kingdom; the Microprocessor without Interlocked Pipeline Stages (MIPS) architecture provided by MIPS Technologies, Inc.
  • ARM Advanced RISC Machine
  • MIPS Microprocessor without Interlocked Pipeline Stages
  • Memory subsystem 122 a comprises a collection of random access memories (RAMs), integrated circuits (ICs) that generally allow for access to data stored in the ICs in any order, in constant time, regardless of the data's physical location.
  • RAMs can include static RAMs (SRAMs); dynamic RAMs (DRAMs); and synchronous DRAMs (SDRAMs).
  • SRAMs are generally very fast but typically have a smaller capacity (e.g., a few megabytes) than DRAMs.
  • SRAMs are static because they have a chip structure that maintains data as long as there is power to the SRAMs.
  • main memory typically comprises DRAMs.
  • DRAMs store data on capacitors within an integrated circuit. DRAMs are dynamic because capacitors can discharge over time due to leakage currents and may require recharging to avoid data loss.
  • SDRAMs have a synchronous interface, meaning that their operation synchronizes with a clock signal.
  • the clock can drive an internal finite state machine that “pipelines” memory accesses (i.e., SDRAM can accept a new memory access before it has finished processing the previous one).
  • Pipelining can improve the performance of SDRAMs relative to conventional DRAMs.
  • the first implementation of SDRAM single data rate (SDR), specified transfer of a single unit of data per clock cycle.
  • DDR double data rate
  • DDR could achieve nearly twice the bandwidth of SDR by transferring data on the rising and falling edges of a clock signal, without increasing clock frequency.
  • DDR evolved into second generation DDR (DDR2) and third generation DDR (DDR3).
  • DDR2 is capable of operating the external data bus at twice the data rate of DDR by improved bus signaling.
  • DDR3 improves upon DDR2 by reducing operating voltage, increasing memory density, and increasing memory bandwidth.
  • memory subsystem 122 a can include multiple memory chips assembled together on small boards known as dual inline memory modules (DIMMs).
  • DIMM can have a rank, an arrangement of chips that produce a specified number of bits of useful data (e.g., 64, 128, etc.).
  • a DIMM can be single-rank, double-rank, quad-rank, etc.
  • a memory controller can select a memory rank by chip select instead of by address bits.
  • a typical memory controller can include up to eight separate chip selects and are therefore capable of supporting up to eight ranks.
  • SDRAM DIMMs can include unbuffered DIMMs (UDIMMs) and registered DIMMs (RDIMMs).
  • UDIMMs unbuffered DIMMs
  • RDIMMs registered DIMMs
  • the memory chips directly connect to the address and control buses, without any intermediate component.
  • RDIMMs have additional components, registers, placed between the incoming address and control buses and the SDRAM components. These registers can add one clock cycle of delay but can reduce the electrical load on the memory controller and allow more DIMMs per memory controller.
  • I/O subsystem 124 a includes peripheral devices (other than network adapter 116 a ) and the interfaces for connecting the peripheral devices to server 112 a .
  • I/O subsystem 124 a is generally responsible for moving data from memory subsystem 122 a to accessible peripheral devices and vice versa.
  • PCI Peripheral Component Interconnect
  • Various embodiments support a version of PCI, PCI Express (PCIe).
  • PCIe specifies point-to-point connectivity resulting in a tree structure topology with a single root complex.
  • the root complex can be responsible for system configuration, enumeration of PCIe resources, and management of interrupts and errors for the PCIe tree.
  • a root complex and its endpoints can share a single address space and communicate through memory reads and writes, and interrupts.
  • PCIe connects two components with a point-to-point link. Links comprise N lanes (i.e., a by-N link comprises N lanes), and each lane can include two pairs of wires: one pair for transmission and one pair for reception. Each lane connects to a PCIe endpoint, PCIe switch, or a PCIe to PCI bridge.
  • Local storage subsystem 126 a comprises non-volatile memory and can be a hard disk drive (HDD), solid state device (SSD), or other type of computer readable media which can store data that is accessible by a computing system, such as Universal Serial Bus (USB) flash memory drives, flash memory cards or sticks, optical disks, magnetic tape, standalone RAM disks, read only memory (ROM), and hybrids thereof.
  • HDD hard disk drive
  • SSD solid state device
  • USB Universal Serial Bus
  • Boot subsystem 128 a includes software and/or firmware for performing hardware initialization upon server 112 a powering on or booting up, and to provide runtime services for operating systems and applications.
  • Boot subsystem 128 a may initialize and test the hardware components of server 112 a .
  • Boot subsystem 128 a can also load one or more operating systems from local storage subsystem 126 a or SANs 102 into memory subsystem 122 a .
  • boot subsystem 128 a can discover and setup one or more peripheral devices for access by processing subsystem 120 a .
  • server 112 a may store boot subsystem 128 a as one or more boot images in a peripheral memory device connected to server 112 a .
  • SANs 102 may store one or more boot images from which server 112 a can retrieve the boot image(s). Multiple boot images can represent different hardware, software (e.g., operating systems, applications, etc.), and/or firmware configurations for server 112 a . Examples of implementations of boot subsystem 128 a include basic input/output system (BIOS) for the x86 architecture or the Unified Extensible Firmware Interface (UEFI) or a bootloader for the ARM architecture.
  • BIOS basic input/output system
  • UEFI Unified Extensible Firmware Interface
  • BMC 118 a may program and/or execute boot subsystem 128 a to configure two physical Ethernet links to combine them into one double-capacity Ethernet interface that can mask link failures (at the cost of halving capacity); configure an Ethernet link to split it into an out-of-band management Ethernet interface with its own MAC address for exclusive use by BMC 118 a and one or more in-band Ethernet interfaces with different MAC addresses; configure a group of disks to organize them as a RAID configuration to form one or more fault-tolerant disks; configure PCIe devices to expose them or hide them from one or more operating systems of server 112 a ; and monitor and manage power supplies, voltages, clocks, CPU speed, temperatures, and fans.
  • BMC 118 a may also program and/or execute boot subsystem 128 a to configure memory controllers to limit access to certain portions or ranges of memory subsystem 122 a to certain processors of processing subsystem 120 a .
  • server 112 a may include multiple memory controllers and multiple processors (e.g., multiple sockets and/or multiple cores per processor).
  • BMC 118 a may program and/or execute boot subsystem 128 a to limit the access of certain processors to particular sections of memory to effectively partition the memory among the processors.
  • BMC 118 a can program and/or execute boot subsystem 128 a to limit the access of certain cores to specific portions of memory and control whether the cores can interrupt one another.
  • BMC 118 a may also program and/or execute boot subsystem 128 a to control whether certain virtual or physical peripheral devices are accessible to specific processors or cores.
  • BMC 118 a can load different boot images for each partition of processors and/or cores and thereby each partition can bootstrap independently. In this manner, BMC 118 a can create a number of segregated resource groups with each group comprising one or more processors or cores, a memory range, and one or more accessible peripheral devices (and/or a PCIe address range).
  • BMC 118 a can perform operations similar to a conventional hypervisor without the overhead that may come with operating a host operating system and a guest operating system. This approach is also an improvement upon containerization because server 112 a is no longer limited to a single operating system.
  • Full-slot server 112 b includes network adapters 116 b and 116 c , BMC 118 b , processing subsystem 120 b , memory subsystem 122 b , I/O subsystem 124 b , local storage subsystem 126 b , and boot subsystem 128 b .
  • full-slot server is in many respects similar to full-slot server 112 b .
  • full-slot server 112 b includes more overall resources than half-slot server 112 a .
  • full-slot server 112 b can include more processors (and/or cores), more DIMMs, and more I/O interfaces (including network interfaces), thus providing greater processing power, memory, and networking capabilities relative to half-slot server 112 a.
  • a port extender standardized by the Institute of Electrical and Electronics Engineers (IEEE) 802.1Qbh protocol, can operate as an access device for use in NICs, blade switches, top-of-rack (TOR) switches, hypervisors, single root I/O virtualization (SR-IOV) adapters, virtual switches, etc.
  • a port extender can attach to a MAC port of an 802.1Q bridge (i.e., a controlling bridge) to provide additional MAC ports (downlink interfaces) that are logical ports of the 802.1Q bridge to which the port extender attaches.
  • Packets can flow from a port extender through the controlling bridge to allow consistent forwarding and policy enforcement for traffic.
  • the controlling bridge can operate as a logically centralized management point for its collection of port extenders.
  • Examples of implementations of port extenders 114 include Cisco® Fabric Extender (FEX) Technology, such as the Cisco Nexus® Fabric Extenders (FEX) for providing additional ports for TOR switches, Cisco UCS® Fabric Extenders for providing additional ports for the Cisco UCS® Blade Server Chassis, Cisco® Adapter Fabric Extender (Adapter FEX) for providing additional ports for a server, and the Cisco® Data Center Virtual Machine Fabric Extender (VM-FEX) for providing additional ports for virtual machines.
  • FEX Cisco® Fabric Extender
  • FEX Cisco Nexus® Fabric Extenders
  • Adapter FEX Cisco® Adapter Fabric Extender
  • VM-FEX Cisco® Data Center Virtual Machine Fabric Extender
  • Port extenders 114 can each include interconnect infrastructure I, chassis manager M, and chassis management switch S.
  • Interconnect infrastructure I can operate as a bridge between servers 112 and switch/routers 108 and implement the data plane of the port extenders 114 .
  • Examples of implementations of interconnect infrastructure I are Cisco® FEX ASICs, such as Redwood and Woodside.
  • Chassis manager M can interact with network-wide manager N in switch/routers 108 and BMC 118 in servers 112 .
  • Chassis manager M can perform server discovery and sequencing, power management, temperature monitoring, and fan control looping.
  • when there are multiple port extenders 114 in server chassis 110 as in the example of FIG.
  • chassis managers M may form a cluster with one manager in an active state and another in an inactive state according to a high-availability algorithm. For example, there can be a serial interface between chassis managers M for receiving heartbeats between the two managers. Failover can occur either by failure to detect a heartbeat or unplugging of the active chassis manager. Network-manager N may also force a fail-over. Examples of implementations of chassis managers M include Cisco® Chassis Management Controller (CMC) ASICs. Chassis manager switch S can provide connectivity to BMC 118 present on each server 112 . Examples of implementations of chassis manager switch S include Cisco® Chassis Management Switch (CMS) ASICs.
  • CMC Cisco® Chassis Management Controller
  • Port extenders 114 connect server chassis 110 to switches/routers 108 a and 108 b (collectively, “ 108 ”).
  • Switches/routers 108 can operate as spine switches in a spine-and-leaf topology; aggregation/distribution switches and/or core switches in three-tier, multi-tier, or fat tree topologies; a Level 1+ switch in a BCube network topology; or other switch/router in a suitable network topology (e.g., DCell, CamCube, FiConn, Jelly fish, Scafida, etc.).
  • switch/routers 108 examples include Cisco UCS® Fabric Interconnects, Cisco Catalyst® switches, Cisco Nexus® switches, Cisco® Industrial Ethernet switches, Cisco MerakiTM or Meraki® switches/routers, Cisco® Integrated Services Routers (ISR), Cisco® Network Convergence System (NCS) routers, Cisco® Aggregation Services Routers (ASR), Cisco® Industrial ISR, and Cisco® Connected Grid Routers, among others.
  • ISR Integrated Services Routers
  • NCS Network Convergence System
  • ASR Cisco® Aggregation Services Routers
  • Cisco® Industrial ISR Cisco® Connected Grid Routers
  • Switches/routers 108 include port controller ASICs P, crossbar fabric ASIC X, and network-wide manager N.
  • Port controller ASICs P control a small group of ports (e.g., 4, 8, 16, etc.) for processing packets upon egress or egress for managed ports.
  • Port controller ASICs P can also handle forwarding decisions between ports. Examples of implementations of port controller ASICs P include Cisco® Unified Port Controller (UPC) ASICs.
  • UPC Cisco® Unified Port Controller
  • Crossbar fabric ASIC X operates as a bridge between port controllers P, and can manage packet switching and scheduling. That is, crossbar fabric ASIC X can couple a port controller for an ingress port to the port controller for an egress port to enable traffic flow between the ports. Examples of implementations of crossbar fabric ASIC X include Cisco® Unified Crossbar Fabric (UCF) ASICs.
  • UPF Cisco® Unified Crossbar Fabric
  • Network-wide manager N can include hardware, software, and/or firmware for monitoring and managing server, network, and storage infrastructure of network environment 100 .
  • Network-wide manager N can provision server, fabric, and storage resources as well as perform device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection. For example, network-wide manager N can automatically detect, inventory, manage, and provision system components added to or changed in network environment 100 .
  • Network-wide manager N can also manage clustering, switch redundancy, and otherwise ensure high availability for server, network, and storage resources in a data center and/or a remote network or cloud.
  • network-wide manager N examples include Cisco UCS® Central, Cisco UCS® Director, Cisco UCS® Manager, Cisco UCS® Performance Manager, Cisco® IMC Supervisor, Cisco® Application Policy Infrastructure Controller (APIC), Cisco ONETM Enterprise Cloud, Cisco® Intelligent Automation, Cisco® Intercloud Fabric, Cisco® Network Services Data Center Network Manager, Cisco Prime® Network Services Controller, or other system for monitoring and managing multiple servers, the network fabric, and/or server storage.
  • Cisco UCS® Central Cisco UCS® Director, Cisco UCS® Manager, Cisco UCS® Performance Manager, Cisco® IMC Supervisor, Cisco® Application Policy Infrastructure Controller (APIC), Cisco ONETM Enterprise Cloud, Cisco® Intelligent Automation, Cisco® Intercloud Fabric, Cisco® Network Services Data Center Network Manager, Cisco Prime® Network Services Controller, or other system for monitoring and managing multiple servers, the network fabric, and/or server storage.
  • APIC Cisco® Application Policy Infrastructure Controller
  • Switches/routers 108 can support various types of traffic and include various types of ports and port controllers for connecting servers 112 to other networks, such as SANs 102 , LAN 104 , management network 106 , or any communicative platform operable to exchange data or information within or between computing systems (e.g., Internet, ad-hoc local network, packet data network (PDN), LAN, metropolitan area network (MAN), wide area network (WAN), wireless local area network (WLAN), virtual private network (VPN), intranet, or any other appropriate architecture or system that facilitates electronic communications).
  • computing systems e.g., Internet, ad-hoc local network, packet data network (PDN), LAN, metropolitan area network (MAN), wide area network (WAN), wireless local area network (WLAN), virtual private network (VPN), intranet, or any other appropriate architecture or system that facilitates electronic communications).
  • PDN packet data network
  • MAN metropolitan area network
  • WAN wide area network
  • WLAN wireless local area network
  • VPN virtual private network
  • intranet or any
  • switches/routers 108 can include a number of 40 Gbps (Quad Small Form-Factor Pluggable) (QSFP) or QSFP+ ports that can operate at native-40-Gbps speed, or that can operate as four 10-Gbps ports (e.g., by inserting a QSFP-to-4 small form factor plus pluggable (SFP+) breakout cable) for handling Ethernet/IP traffic (e.g., traffic to/from LAN 104 ), Fibre Channel (FC) or Fibre Channel on Ethernet (FCoE) ports for handling block storage traffic (e.g., traffic to/from SANs 102 ), and serial ports for handling management traffic (e.g., traffic to/from management network 106 ) and inter-process communications (IPC) (e.g., high availability, clustering, virtualization platform services, etc.).
  • QSFP Quality Small Form-Factor Pluggable
  • SFP+ Small Form-Factor Pluggable
  • FIG. 2A and FIG. 2B illustrate computing systems 200 and 250 .
  • computing systems 200 and 250 can be respective implementations of servers 112 a and 112 b of FIG. 1 .
  • This disclosure provides additional details regarding each of the components in FIG. 2A and FIG. 2B in more detail below.
  • computing systems 200 and 250 are each simply one possible configuration and that other configurations with more or fewer components are also possible.
  • Computing system 200 is a half-slot, two-socket server including processors 202 a and 202 b (collectively, “ 202 ”).
  • Sockets are mounting/interconnection structures for installing processors on a printed circuit board (PCB) or mother board of a computing system. Multiple sockets can provide for customization of a server motherboard by enabling mounting of processors using different clock speeds and/or amounts of power.
  • Each processor 202 can include one or more cores (e.g., 2, 4, 6, 8, etc.) (not shown), each of which can replicate a basic central processing unit (CPU).
  • Each core may be associated with a level 1 (L1) cache (not shown).
  • Caches are small fast memories that can reduce the average time to access main memory.
  • the cores generally share a larger level 2 (L2) or level 3 (L3) cache, a bus or interconnection interface, and external die connections.
  • the number of processors of a computing system is the product of the number of sockets and the number of cores per socket.
  • computing system 200 includes two sockets and can include four cores per socket for a total of eight processors.
  • An example of an implementation of processor 202 a or 202 b is the Xeon Processor provided by Intel Corp. of Santa Clara, Calif.
  • processors 202 connect to one another and to I/O hub 204 by interconnections 206 a , 206 b , and 206 c (collectively, “ 206 ”).
  • processors 202 and interconnections 206 implement the QuickPath Interconnect (QPI) architecture provided by Intel.
  • QPI utilizes multiple high-speed uni-directional links for interconnecting processors and a chipset or similar computing system element and integrating multiple distributed memory controllers for multiple cores of the processor chip.
  • the cores inside a socket may share integrated memory controllers (IMCs) (not shown) that have multiple memory interfaces (i.e., memory buses).
  • IMCs integrated memory controllers
  • the IMCs may have various external connections, such as DDR3 memory (or other suitable memory) channels for connecting processors 202 to DDR3 (or other suitable memory) DIMMs D.
  • IMCs and cores in different sockets can talk to one another using QPI.
  • Processors implementing QPI may also have full access to the memory of every other processor while maintaining cache coherency using a cache-coherent Non-Uniform Memory Architecture (NUMA).
  • NUMA Non-Uniform Memory Architecture
  • NUMA Non-Uniform Memory Architecture
  • a computing system that implements QPI can achieve global memory reachability (or reachability by one socket to memory of another socket) using an internal crossbar router in the socket. This route-through capability may allow for computing systems without a fully connected topology.
  • Other embodiments may implement interconnections 206 using other types of buses or interconnections, such as front-side buses (FSBs), dual independent buses (DIBs), Dedicated High-Speed Interconnect (DHSI), etc.
  • FFBs front-side
  • I/O hub 204 (sometimes referred to as a chipset) connects processors 202 to I/O controller hub (ICH) 208 using interconnection 210 , local storage controller 212 using interconnection 216 a , and mezzanine card 214 using interconnection 216 b .
  • I/O hub 204 is the X58 chip provided by Intel.
  • interconnection 210 implements the Direct Media Interface (DMI) provided by Intel
  • interconnections 216 a and 216 b are a PCIe x4 link and a PCIe x16 link, respectively.
  • DMI Direct Media Interface
  • Local storage controller 212 connects computing system 200 to local storage devices (e.g., HDD, SSD, etc.).
  • Local storage controller 212 (sometimes referred to as a SAS controller) can support Serial Attached Small Computer System Interface (SAS) and Serial Advanced Technology Attachment (SATA) transfer rates, and integrate mirroring and striping functions to provide different RAID availability levels for internal drives.
  • SAS Serial Attached Small Computer System Interface
  • SATA Serial Advanced Technology Attachment
  • An example of an implementation of local storage controller 212 is the 1064e storage processor provided by LSI Corp. of San Jose, Calif.
  • Mezzanine card 214 i.e., a network adapter
  • PCI bus e.g., IEEE P1386.1
  • CMC Common Mezzanine Card
  • Mezzanine card 214 can include a number of bus connectors (not shown), such as for connecting to one or more 32-bit PCI buses, 64-bit PCI buses, or other non-specified, non-standardized, and/or proprietary I/O bus.
  • PMC PCI Mezzanine Card
  • PMC-X PCI Mezzanine Card
  • PPMC Processor PMC
  • CCPMC conduction-cooled PMC
  • XMC Switched Mezzanine Card
  • FMC—FPGA Mezzanine Card standards PCI Mezzanine Card
  • PMC-X PCI Mezzanine Card
  • PPMC Processor PMC
  • CCPMC conduction-cooled PMC
  • XMC Switched Mezzanine Card
  • FMC—FPGA Mezzanine Card standards FPGA Mezzanine Card
  • I/O controller hub 208 (sometimes referred to as a Southbridge) connects computing system 200 to relatively low-speed peripheral devices (not shown) (e.g., USB devices or other devices slower than mezzanine card 214 ), BMC 218 , and boot subsystem 220 .
  • I/O controller hub 208 is the ICH 10 I/O controller hub provided by Intel®.
  • interconnection 216 c between ICH 208 and BMC 218 is a PCIe x4 link.
  • BMC 218 provides management access to computing system 200 prior to the loading of an operating system, and can operate as an aggregation point for server hardware.
  • BMC 218 can have two integrated Ethernet connections (not shown) connected in a redundant manner to management components of access devices (e.g., chassis management switches S).
  • BMC 118 of FIG. 1 is an example of an implementation of BMC 218 .
  • Boot subsystem 220 includes software or firmware for initializing hardware upon powering on or booting up computing system 200 (e.g., BIOS, UEFI, bootloader, etc.) or other management software and/or firmware executed prior to loading of one or more operating systems of computing system 200 .
  • Boot subsystem 128 is an example of an implementation of boot subsystem 220 .
  • Computing system 250 is a full-slot, four-socket server including processors 252 a , 252 b , 252 c , and 252 d (collectively, “ 252 ”).
  • Each processor 252 can include one or more cores (e.g., 4, 6, 8, etc.) (not shown), each of which represents a discrete processing element.
  • each processor 252 can include 4 cores such that the CPU of computing system 250 includes a total of 16 processors. Examples of implementations of processors 252 include the Xeon 7500 series CPUs provided by Intel.
  • Processors 252 are fully interconnected with one another using interconnections 254 a , 254 b , 254 c , 254 d , 254 e , and 254 f (collectively, “ 254 ”).
  • Processors 252 c and 252 d also connect to I/O hub 256 using interconnections 254 g and 254 h , respectively.
  • processors 252 a and 252 b may connect to I/O hub 256 through core-to-core QPI links.
  • processors 252 a and 252 b may connect to a second I/O hub symmetrical that enable such a computing system to include additional memory and/or PCIe slots.
  • interconnections 254 are QPI links but other embodiments may utilize other types of buses or interconnections as discussed elsewhere.
  • Each processor 252 also connects to one or more serial memory buffers (SMBs) 258 using Serial Memory Interface (SMI) links 260 .
  • SMB 258 can connect to one or more DIMMs D (e.g., DDR3 DIMM).
  • DIMMs D e.g., DDR3 DIMM.
  • a computing system can include four sockets with each socket connected to four SMBs and each SMB connected to two DIMMs providing a total of 32 DIMMs.
  • Interconnections 262 b and 262 c are the two main I/O paths toward mezzanine cards 264 a and 264 b (collectively, “ 264 ”), respectively.
  • interconnections 262 b and 262 c are PCIe x16 links.
  • Mezzanine cards 264 e.g., network adapters 116 of FIG. 1
  • access devices e.g., port extenders 114 of FIG. 1
  • Local storage controller 266 e.g., an embedded 6 G SAS RAID controller connects to I/O hub 256 through a PCIe x4 link.
  • ICH 268 also connects to I/O hub 256 by interconnection 270 (e.g., an Enhanced Function Interface (EFI) bus).
  • ICH 268 provides connections to BMC 272 , Front panel 274 , and boot subsystem 276 .
  • Front panel 274 can include one or more USB ports and/or ports for other low-speed peripheral devices (not shown).
  • computing system 200 is in many respects similar to computing system 250 .
  • computing system 250 is a full-slot server that includes more overall resources than computing system 200 , a half-slot server.
  • computing system 250 includes more sockets and cores than computing system 200 , and thus the CPU of computing system 250 includes more processors than the CPU of computing system 200 , more DIMMs and thus more main memory, and more I/O interfaces (including mezzanine cards).
  • computing system 250 has greater processing power, memory, and networking capabilities than computing system 200 .
  • computing systems 200 and 250 illustrate examples of the x86 architecture, other embodiments may utilize other server architectures (e.g., ARM, MIPS, Power, SPARC, other Reduced Instruction Set Computer (RISC) architectures, and/or other Complex Instruction Set Computer (CISC) architectures), and one of ordinary skill in the art could readily apply the principles disclosed in this disclosure to these other architectures.
  • server architectures e.g., ARM, MIPS, Power, SPARC, other Reduced Instruction Set Computer (RISC) architectures, and/or other Complex Instruction Set Computer (CISC) architectures
  • FIG. 3 illustrates an example of process 300 for physically partitioning resources of a computing system via a baseboard management controller of the system.
  • a network and particularly a network-wide manager (e.g., network-wide manager N of FIG. 1 ) or other system for monitoring and managing multiple servers, the network fabric, and/or server storage can communicate with a baseboard management controller to perform process 300 .
  • a standalone baseboard management controller can perform process 300 .
  • Process 300 may begin with a physical server or host powering on and plugging into a network (e.g., from an off state, after reset, after plug-in into a chassis/rack, after insertion as a card into a slot, etc.).
  • a baseboard management controller (e.g., BMC 118 a or 118 b ) can determine the availability of multiple processors of the host of the BMC for provisioning to an end user.
  • a processor may include a single-socket central processing unit (CPU), a pair of CPUs or processors of a four-socket server, a multi-core processor, a single core of a multi-core processor, or other discrete physical device capable of executing the instructions of a computer program by performing the basic arithmetic, logical, control, and input/output (I/O) operations specified by the instructions. If there are no processors available, process 300 can come to an end. Otherwise, process 300 can continue onto step 304 at which the BMC can allocate at least a first processor of the physical host to a first resource group and a second processor of the physical host to a second resource group.
  • the BMC can load one or more boot images (e.g., BIOS, UEFI boot manager, boot loader, bootstrap, etc.) and/or other configuration data from a SAN (e.g., SAN 102 a or 102 b of FIG. 1 ) or other remote source or a storage device embedded on the physical host (e.g., ROM, flash memory, or other non-volatile memory) (e.g., boot subsystem 128 a or 128 b , boot subsystem 220 of FIG. 2A , or boot subsystem 276 of FIG. 2B ) for initializing the hardware of the physical host.
  • boot images e.g., BIOS, UEFI boot manager, boot loader, bootstrap, etc.
  • other configuration data e.g., BIOS, UEFI boot manager, boot loader, bootstrap, etc.
  • a SAN e.g., SAN 102 a or 102 b of FIG. 1
  • a storage device embedded on the physical host
  • the BMC can receive a single boot image for configuring multiple physical partitions of the host. In other embodiments, the BMC may receive multiple boot images, including at least one unique boot image for each different resource group. In this manner, each resource group can boot-up independently from one another and apply different programmable code-signing keys for giving the BMC control over which operating system each resource group may load.
  • the BMC/boot image(s) can partition the main memory (i.e., primary storage, internal memory, RAM, etc.) of the physical host into at least a first memory range for exclusive use by the first processor and a second memory range for exclusive use by the second processor.
  • the first memory range can comprise a first set of DIMMs and the second memory range can comprise a second set of DIMMs.
  • the BMC/boot image(s) can generate a first memory map that maps the first memory range to the first set of DIMMs and excludes mappings to other DIMMs.
  • the BMC/boot image(s) can also generate a second memory map that maps the second memory range to the second set of DIMMs and excludes mappings to other DIMMs, including the first set of DIMMs.
  • the BMC/boot image(s) can allocate the resources of the computing system in a variety of configurations.
  • FIG. 4A , FIG. 4B , and FIG. 4C illustrate different ways that a BMC can physically partition the processors and main memory of a computing system.
  • FIG. 4A illustrates a computing system 400 having four sockets for four CPUs 402 a , 402 b , 402 c , and 402 d (collectively, “ 402 ”).
  • Each processor 402 includes integrated memory controller (IMC) 404 for accessing a respective set of DDR3 DIMMs D.
  • IMC integrated memory controller
  • a BMC partitions computing system 400 into resource groups 406 a , 406 b , 406 c , and 406 d (collectively, “ 406 ”).
  • Each resource group (e.g., resource group 406 a ) includes a subset of the processing resources of computing system 400 (e.g., processor 402 a ); a subset of the memory controllers of computing system 400 (e.g., IMC 404 a ); and a subset of memory or a range of the main memory of computing system 400 (e.g., DIMMs D directly connected to IMC 404 a ).
  • FIG. 4A shows that each resource group 406 includes one CPU, a resource group can also include multiple CPUs.
  • the BMC can partition computing system into three resource groups, a first resource group including CPUs 402 a and 402 b , a second resource group including CPU 402 c , and a third resource group including CPU 424 c .
  • a resource group can also include a portion of a CPU (i.e., one or more cores of a multi-core processor) as shown in FIG. 4B and discussed further below.
  • computing system 400 can implement cache coherency by default and disable cache coherency under certain conditions (or vice versa, i.e., disable cache coherency by default and activate cache coherency under certain conditions).
  • a computing system that implements cache coherency seeks uniformity of shared data stored among multiple local caches. Common approaches for achieving cache coherency include snooping and directory-based cache coherency. In snooping, individual caches monitor address lines for access to cached memory locations and invalidate or update a copy of a snooped memory location on write to the corresponding memory location in a different cache. In directory-based cache coherency, each processor stores shared data to a common directory that maintains the coherency between caches.
  • computing system 400 may safely disable cache coherency for improved performance by each resource group.
  • FIG. 4B illustrates computing system 420 including CPU 422 and DIMMs 428 a and 428 b (collectively, “ 428 ”).
  • CPU 422 includes six cores, processors 430 a , 430 b , 430 c , 430 d , 430 e , and 430 f (collectively, “ 430 ”), and IMCs 424 a and 424 b (collectively, “ 424 ”).
  • Computing system 420 may be a single-socket server including only CPU 422 or a multi-socket server including CPU 422 and one or more other CPUs.
  • a BMC separates computing system 420 into at least two resource groups 426 a and 426 b (collectively, “ 426 ”).
  • Resource group 426 a includes cores 430 a and 430 b , IMC 424 a , and DIMMs 428 a .
  • Resource group 426 b includes cores 430 c , 430 d , 430 e , and 430 f , IMC 424 b , and DIMMs 428 b.
  • the BMC of computing system 420 is capable of partitioning the cores of a multi-core processor into two or more resource groups with at least one core in a first resource group and a second core in a second resource group. That is, the BMC can limit the memory accessible by cores 430 a and 430 b to DIMMs 428 a ; and the BMC can limit the memory accessible by cores 430 c , 430 d , 430 e , and 430 f to DIMMs 428 b . In other embodiments, the BMC can address memory differently for each individual core.
  • CPU 422 may include additional pins for encoding which core 430 is requesting a cache line in order to limit a particular core to a particular range of memory.
  • additional pins for encoding which core 430 is requesting a cache line in order to limit a particular core to a particular range of memory.
  • FIG. 4B also shows that the BMC is capable of allocating IMCs 424 a and 424 b among different resource groups.
  • IMCs 424 a and 424 b may be physically or logically separate elements each associated with a memory map to DIMMs 428 a and 428 b , respectively.
  • computing system 420 may disable cache coherency at least between DIMMs 428 a and 428 b because of the physical isolation between these memories and their respective processors.
  • a resource group may also be associated with various priority levels.
  • the BMC may configure whether one core may interrupt another core residing in the same socket or whether one CPU may interrupt another CPU mounted on the same motherboard based on priority level.
  • FIG. 4C illustrates computing system 440 including at least CPUs 442 a and 442 b (collectively, “ 442 ”) and DIMMs 448 a , 448 b , 448 c , and 448 d (collectively, “ 448 ”).
  • CPU 442 a includes four cores, processors 470 a , 470 b , 470 c , and 470 d and IMC 444 a .
  • CPU 442 b also includes four cores, processors 470 e , 470 f , 470 g , and 470 h and IMC 444 b .
  • CPUs 442 include the same number of cores in this example, other embodiments may include multi-core multi-processors having different numbers of cores (e.g., 2 and 4 cores, 4 and 6 cores, 2 and 6 cores, etc.).
  • a BMC partitions computing system 440 into at least two resource groups 446 a and 446 b (collectively, “ 446 ”).
  • Resource group 446 a includes cores 470 a and 470 b , IMC 444 a , and DIMMs 448 a .
  • Resource group 446 b includes cores 470 c , 470 d , 470 e , 470 f , 470 g , and 470 h , IMCs 444 a and 444 b , and DIMMs 448 b , 448 c , and 448 d .
  • the BMC may logically partition IMC 444 a into two virtual memory controllers with one virtual controller dedicated to resource group 446 a and another virtual controller dedicated to resource group 446 b.
  • the BMC of computing system 440 can define a resource group including portions of CPUs 442 a and 442 b (i.e., at least one core from CPU 442 a and one core from CPU 442 b ).
  • computing system 440 may disable cache coherency at least between DIMM 428 a and other DIMMs because of the physical isolation between these memories and their respective processors but maintain cache coherency at least between and among DIMMs 448 b , 448 c , and 448 d.
  • process 300 can continue to step 310 in which the BMC/boot image(s) can distribute access to physical or virtual peripheral devices connected to the physical host between and among the first processor and the second processor.
  • the peripheral devices can include network adapters, graphic processing unit (GPU) adapters, Peripheral Component Interconnect Express (PCIe) Flash adapters, Fibre Channel host bus adapters (HBAs), disks, disk controllers, USB devices, etc.
  • the BMC/boot image(s) can expose one or more PCIe I/O ports to one of the processors and/or hide one or more other PCIe I/O ports from that processor.
  • the BMC/boot image(s) can map one or more peripheral device's memories into one of the processors' main memory range to give that processor access to the peripheral device(s) and/or deny that processor access to one or more other peripheral devices by not mapping the other peripheral devices' memories into the processor's main memory.
  • the BMC/boot image(s) can configure a peripheral device to be a bus master, and that peripheral device can act on I/O requests from one of the processors and ignore I/O requests from another processor.
  • the BMC/boot image(s) may utilize Single Root I/O Virtualization (SR-IOV) to virtualize a physical peripheral device to create multiple virtual peripheral devices accessible by multiple operating systems and their respective processors on a single-socket server.
  • SR-IOV Single Root I/O Virtualization
  • the BMC/boot image(s) may utilize Multi-Root I/O Virtualization (MR-IOV) to virtualize a physical peripheral device to create multiple virtual peripheral devices accessible by multiple operating systems and their respective processors on a multi-socket server.
  • MR-IOV Multi-Root I/O Virtualization
  • the BMC/boot image(s) can load a first operating system into the first range of main memory associated with the first processor and a second operating system into the second range of main memory associated with the second processor.
  • the operating systems may be different operating systems (e.g., Microsoft Windows, UNIX, Linux, Mac OS X, etc.) or different versions of the same operating system (e.g., Microsoft Windows 7.0 and Microsoft Windows 10.0).
  • the BMC/boot image(s) can retrieve or otherwise receive the operating systems from a SAN or other remote storage or a local mass storage device (e.g., HDD, SDD, flash drive, etc.).
  • Process 300 may conclude at step 314 when the BMC cedes control to the first processor to execute the first operating system and to the second processor to execute the second operating system.
  • the disclosure may present various embodiments as including individual functional blocks comprising devices, device components, steps or routines in a method embodied in software or firmware, or combinations of hardware, firmware, and/or software.
  • the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like.
  • non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.
  • Computer-executable instructions stored or otherwise available from computer readable media, can implement methods according to the above-described examples. Such instructions can comprise, for instance, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network.
  • the computer-executable instructions may also include binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media for storing instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.
  • Devices implementing methods according to these disclosures can comprise hardware, firmware, and/or software, and can take any of a variety of form factors.
  • Typical examples of such form factors include servers (e.g., mainframe servers, tower servers, rack-mount servers, blade servers, microservers, etc.), small form factor personal computers, laptops, smart phones, personal digital assistants, and so on.
  • Peripherals or add-in cards can also perform some of the functionality described herein.
  • a circuit board including different chips or different processes executing in a single device can also perform some of the functionality, by way of further example.
  • the instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • Multi Processors (AREA)

Abstract

A baseboard management controller (BMC) can physically partition the computing resources of a physical host into different resource groups for concurrently running a different operating system per resource group. The BMC can allocate a first processor of the host to a first resource group and a second processor of the host to a second resource group. The BMC can separate the memory of the host into a first memory range for the first processor and a second memory range for the second processor, and the BMC can limit access to the first memory range to the first processor and limit access to the second memory range to the second processor. The BMC can also distribute physical or virtual peripheral devices of the host between the first processor and the second processor.

Description

    CROSS-REFERENCE TO RELATED APPLICATION
  • This application is a continuation of U.S. patent application Ser. No. 15/617,190 filed on Jun. 8, 2017, the contents of which is incorporated by reference in its entirety.
  • TECHNICAL FIELD
  • The subject matter of this disclosure relates in general to the field of server virtualization, and more specifically to techniques for physically partitioning resources of a computing system via a baseboard management controller of the system.
  • BACKGROUND
  • Virtualization is an important technology used in a modern data center. Virtualization can allow an operator of the data center to consolidate workloads; raise utilization levels; reduce operating, capital, space, power, and cooling expenses; move workloads dynamically within a virtualization pool to increase the flexibility to take servers offline or bring new ones online; manage the relationship of virtual instances to physical machines to optimize performance and maintain service levels; scale existing applications or deploy new ones by creating more virtual instances using an existing pool of resources; and deploy high-availability and disaster-recovery features of a virtualization platform to overcome localized and geographic failures, among other benefits.
  • Two common approaches to virtualizing a data center are hypervisor-based virtualization and container-based virtualization. In hypervisor-based virtualization, software emulates the hardware of a physical computer so that an entire, unmodified operating system can run with the emulated or virtual environment (i.e., a virtual machine (VM)). A single physical server can run several VMs at once, and a hypervisor or virtual machine monitor (VMM) can manage the VMs and allocate the resources of the server among them. Hypervisor-based virtualization, however, can introduce additional overhead because a server implementing this approach must replicate true hardware behaviors for each VM running on the server. Containers do not require an additional layer of virtual hardware. Instead, a system implementing containers attempts to provide self-contained execution environments by isolating applications that rely on the same kernel. Thus, containers within a server all run on a single operating system kernel, and that kernel must be capable of supporting all applications and software running within the containers. Container-based virtualization therefore typically does not allow a server to run different operating systems or different versions of the same operating system.
  • BRIEF DESCRIPTION OF THE FIGURES
  • FIG. 1 illustrates an example of a network environment in accordance with an embodiment;
  • FIG. 2A and FIG. 2B illustrate examples of computing systems in accordance with some embodiments;
  • FIG. 3 illustrates an example of a process for physically partitioning resources of a computing system via a baseboard management controller of the system in accordance with an embodiment; and
  • FIG. 4A, FIG. 4B, and FIG. 4C illustrate examples of physically partitioned computing systems in accordance with some embodiments.
  • DESCRIPTION OF EXAMPLE EMBODIMENTS Overview
  • Systems and methods in accordance with various embodiments of the present disclosure may overcome one or more of the foregoing or other deficiencies experienced in conventional approaches for hypervisor-based and container-based virtualization. A baseboard management controller (BMC) can physically partition the computing resources (e.g., processing, memory, input/output (I/O), and/or storage) of a server into two or more resource groups for concurrently running a different operating system and one or more applications per resource group. For example, the BMC can allocate at least a first processor (e.g., a pair of processors of a four-socket server, a multi-core processor, a single core of a multi-core processor, etc.) of a physical host to a first resource group and a second processor of the physical host to a second resource group. The BMC can load one or more boot images (e.g., basic input/output system (BIOS), Unified Extensible Firmware Interface (UEFI) boot manager, boot loader, bootstrap, or other software/firmware executed prior to loading of an operating system) and/or other configuration data from a storage area network (SAN) (or other remote storage) or a storage device embedded on the physical host (e.g., read-only memory (ROM), flash memory, or other non-volatile memory) for initializing the hardware of the physical host.
  • In some embodiments, the BMC/boot image(s) can partition the memory of the physical host into at least a first memory range for exclusive use by the first processor and a second memory range for exclusive use by the second processor. For example, the first memory range can comprise a first set of (one or more) dual in-line memory modules (DIMMs) mounted on the physical host and the second memory range can comprise a second set of (one or more) DIMMs mounted on the physical host. The BMC/boot image(s) can generate a first memory map that maps the first memory range to the first set of DIMMs and a second memory map that maps the second memory range to the second set of DIMMs. Alternatively or in addition, the BMC/boot image(s) can limit access to the memory controller(s) of the first set of DIMMs to the first processor and limit access to the memory controller(s) of the second set of DIMMs to the second processor.
  • In some embodiments, the BMC/boot image(s) can also distribute physical or virtual peripheral devices of the physical host between the first processor and the second processor. For example, the BMC/boot image(s) can expose/hide one or more I/O ports from one of the resources groups. Alternatively or in addition, the BMC/boot image(s) can generate a Peripheral Component Interconnect Express (PCIe) address memory map for one of the resource groups that gives that resource group access to a particular peripheral device and denies access to a different peripheral device. Alternatively or in addition, the BMC/boot image(s) can configure a peripheral device to be a bus master, and that peripheral device can act on I/O requests from a particular processor and ignore I/O requests from a different processor.
  • DESCRIPTION
  • FIG. 1 illustrates an example of a network environment 100 in accordance with an embodiment. One of ordinary skill in the art will understand that, for the network environment 100 and any system discussed in the present disclosure, there can be additional or fewer nodes, devices, links, networks, or components in similar or alternative configurations. Various embodiments may include different numbers and/or types of clients, networks, nodes, cloud components, servers, software components, devices, virtual or physical resources, configurations, topologies, services, appliances, deployments, network devices, etc. The illustrations and examples provided in the present disclosure are for conciseness and clarity.
  • In this example, the network environment 100 includes storage area networks (SAN) 102 a and 102 b (collectively, “102”), local area network (LAN) 104, management network 106, network devices 108 a and 108 b (collectively, “108”), and server chassis 110. Server chassis 110 is a computing infrastructure device used to interconnect servers in various form factors (e.g., rack servers, blade servers, and other high-density servers) with other network elements. Server chassis 110 can provide power, cooling, connectivity, and mechanical support for servers, such as servers 112 a and 112 b (collectively, “112”) and network devices, such as switches, routers, network appliances (e.g., load balancers, deep packet inspectors, firewalls, etc.), port extenders (e.g., port extenders 114 a and 114 b (collectively, “114”)), etc. An example of an implementation of server chassis 110 is Cisco Unified Computing System™ (Cisco UCS®) Blade Server Chassis, provided by Cisco Systems®, Inc. of San Jose, Calif. Server chassis 110 includes a number of slots (e.g., 8 half-width slots, 4 full-width slots, or other capacities) for receiving servers 112. Server chassis 110 can reduce the number of physical components and the amount of cabling relative to conventional rack or blade systems, integrate with existing infrastructure for centralized management, and operate more efficiently with respect to energy consumption than conventional systems.
  • In FIG. 1, server 112 a is a half-width or half-slot server and server 112 b is a full-width or full-slot server. Other embodiments may utilize servers having other types of form factors, including some embodiments with servers that do not require a chassis. For example, various embodiments can include a server that is a standalone device communicatively coupled to server chassis 110 or to one or more network devices 108. Various types of interconnections and buses can provide the communicative coupling, including any wired or wireless interconnection line, network, connection, bundle, single bus, multiple buses, crossbar network, single-stage network, multi-stage network, or other conduction medium operable to carry data between parts of a computing device or between computing devices.
  • Half-slot server 112 a includes network adapter 116 a, baseboard management controller (BMC) 118 a, processing subsystem 120 a, memory subsystem 122 a, I/O subsystem 124 a, local storage subsystem 126 a, and boot subsystem 128 a. Network adapter 116 a (e.g., a network interface controller or card (NIC), network adapter, LAN adapter, etc.) connects server 112 a to other physically separate and discrete network elements (e.g., network adapters 116 b and 116 c, port extenders 114, network devices 108, SANs 102, LAN 104, management network 106, etc.) and logically separate elements of server 112 a (e.g., virtual machines, containers, or other partitions). A person of ordinary skill will appreciate that some of these elements are combinable (e.g., an I/O subsystem typically includes a network interface) or further divisible (e.g., cache memory is distinguishable from main memory) but server 112 a includes the above subsystems for purposes of simplicity and clearness.
  • BMC 118 a monitors and manages the physical state of server 112 a. BMC 118 a includes a specialized service processor (not shown) and firmware (not shown) to provide management and monitoring capabilities independently from processing subsystem 120 a. BMC 118 a is reachable even when processing subsystem 120 a is powered off or non-operational. In some embodiments, BMC 118 a supports the standards defined in the Intelligent Platform Management Interface (IPMI) specification. An example of an implementation of BMC 118 a is the Cisco® Integrated Management Controller (CIMC). CIMC is compliant with the IPMI specification but also provides additional functionality for providing unified monitoring and management of multiple computing systems. Diagnostic and health monitoring features provided with CIMC include support for Simple Network Management Protocol (SNMP); extensible mark-up language (XML) application programming interface (API) event subscription and configurable alerts; system event logging; audit logging; monitoring of field-replaceable units (FRUs), hard disk drive (HDD) faults, dual inline memory module (DIMM) faults, NIC media access control (MAC) address, CPU, and thermal faults; configurable alerts and thresholds; watchdog timer; redundant array of independent disks (RAID) configuration and monitoring; predictive failure analysis of HDD and DIMM; support for converged network adapters (CNAs); and support for Network Time Protocol (NTP).
  • In some embodiments, CIMC can operate in a standalone mode to provide users with full control of the server, allowing an administrator to perform server management tasks including powering on, powering off, power-cycling, resetting, and shutting down the server; toggling the locator light-emitting diode (LED); configuring the server boot order; viewing server properties and sensors; configuring out-of-band storage; managing remote presence; managing firmware; creating and managing local user accounts and enabling authentication through Active Directory and Lightweight Directory Access Protocol (LDAP); configuring network-related settings, including NIC properties, Internet Protocol (IP) version 4 (IPv4), IP version 6 (IPv6), virtual local area networks (VLANs), and network security; configure communication services, including Hypertext Transfer Protocol (HTTP), secure shell (SSH), and IPMI over LAN; managing certificates; configuring platform event filters; and monitoring faults, alarms, and server status.
  • In some embodiments, CIMC may also provide features such as a hypertext mark-up language version 5 (HTMLS) and keyboard, video, and mouse (KVM) user interface (UI); Redfish support; and XML API transactional support. HTMLS and KVM can provide users with a simplified UI, and can eliminate the need for Java to use CIMC. Redfish is an open industry standard specification and schema that specifies a restful stateful transfer (REST) interface and uses Javascript Object Notation (JSON) and Open Data Protocol (OData) to help customers integrate solutions within their existing tool chains. XML API transactional support enables configuration of multiple managed objects in a single transaction, allowing for quicker, simpler deployments.
  • In some embodiments, BMC 118 a can perform configuration and management services while server 112 a is in a low-power state, such as a standby state. In contrast, processing subsystem 120 a, memory subsystem 122 a, local storage subsystem 126 a, etc., may require server 112 a to be in a relatively high power state. In general, a low-power state may include a state where server 112 a is not completely powered on and does not provide all or substantially all of its full functionality, whereas a high-power state is a state where server 112 a is powered on and provides all or substantially all of its capabilities, less capabilities that are specifically disabled for purposes of management and configuration.
  • Processing subsystem 120 a connects to other elements of server 112 a via one or more interconnects or buses, and can directly perform instructions stored in memory subsystem 122 a and indirectly perform instructions stored in local storage subsystem 126 a, SANs 102, and/or other memory locations. Processing subsystem 120 a can include any combination of hardware, software, and/or firmware providing programmable logic. Examples of implementations of processing subsystem 120 a include the Advanced RISC Machine (ARM) architecture provided by ARM Holdings plc of Cambridge, England, United Kingdom; the Microprocessor without Interlocked Pipeline Stages (MIPS) architecture provided by MIPS Technologies, Inc. of Sunnyvale, Calif.; the Power architecture provided by IBM of Armonk, North Castle, N.Y.; the Scalable Processor Architecture (SPARC) provided by Sun Microsystems of Menlo Park, Calif.; and the x86 architecture provided by Intel Corp. of Santa Clara, Calif., Advanced Micro Devices (AMD), Inc. of Sunnyvale, Calif., or VIA Technologies Inc. of New Taipei City, Taiwan, Republic of China.
  • Memory subsystem 122 a comprises a collection of random access memories (RAMs), integrated circuits (ICs) that generally allow for access to data stored in the ICs in any order, in constant time, regardless of the data's physical location. RAMs can include static RAMs (SRAMs); dynamic RAMs (DRAMs); and synchronous DRAMs (SDRAMs). SRAMs are generally very fast but typically have a smaller capacity (e.g., a few megabytes) than DRAMs. SRAMs are static because they have a chip structure that maintains data as long as there is power to the SRAMs. However, SRAMs are generally not large enough to operate as the main memory of a server. Instead, main memory typically comprises DRAMs. DRAMs store data on capacitors within an integrated circuit. DRAMs are dynamic because capacitors can discharge over time due to leakage currents and may require recharging to avoid data loss.
  • SDRAMs have a synchronous interface, meaning that their operation synchronizes with a clock signal. The clock can drive an internal finite state machine that “pipelines” memory accesses (i.e., SDRAM can accept a new memory access before it has finished processing the previous one). Pipelining can improve the performance of SDRAMs relative to conventional DRAMs. The first implementation of SDRAM, single data rate (SDR), specified transfer of a single unit of data per clock cycle. The next implementation of SDRAM, double data rate (DDR), could achieve nearly twice the bandwidth of SDR by transferring data on the rising and falling edges of a clock signal, without increasing clock frequency. DDR evolved into second generation DDR (DDR2) and third generation DDR (DDR3). DDR2 is capable of operating the external data bus at twice the data rate of DDR by improved bus signaling. DDR3 improves upon DDR2 by reducing operating voltage, increasing memory density, and increasing memory bandwidth.
  • In some embodiments, memory subsystem 122 a can include multiple memory chips assembled together on small boards known as dual inline memory modules (DIMMs). A DIMM can have a rank, an arrangement of chips that produce a specified number of bits of useful data (e.g., 64, 128, etc.). Thus, a DIMM can be single-rank, double-rank, quad-rank, etc. A memory controller can select a memory rank by chip select instead of by address bits. A typical memory controller can include up to eight separate chip selects and are therefore capable of supporting up to eight ranks.
  • SDRAM DIMMs can include unbuffered DIMMs (UDIMMs) and registered DIMMs (RDIMMs). In UDIMMs, the memory chips directly connect to the address and control buses, without any intermediate component. RDIMMs have additional components, registers, placed between the incoming address and control buses and the SDRAM components. These registers can add one clock cycle of delay but can reduce the electrical load on the memory controller and allow more DIMMs per memory controller.
  • I/O subsystem 124 a includes peripheral devices (other than network adapter 116 a) and the interfaces for connecting the peripheral devices to server 112 a. I/O subsystem 124 a is generally responsible for moving data from memory subsystem 122 a to accessible peripheral devices and vice versa. Historically, computing systems provided I/O using buses compatible with the Peripheral Component Interconnect (PCI) standard, a standard developed to interconnect peripheral devices to a computing system. Various embodiments support a version of PCI, PCI Express (PCIe). PCIe specifies point-to-point connectivity resulting in a tree structure topology with a single root complex. The root complex can be responsible for system configuration, enumeration of PCIe resources, and management of interrupts and errors for the PCIe tree. A root complex and its endpoints can share a single address space and communicate through memory reads and writes, and interrupts. PCIe connects two components with a point-to-point link. Links comprise N lanes (i.e., a by-N link comprises N lanes), and each lane can include two pairs of wires: one pair for transmission and one pair for reception. Each lane connects to a PCIe endpoint, PCIe switch, or a PCIe to PCI bridge.
  • Local storage subsystem 126 a comprises non-volatile memory and can be a hard disk drive (HDD), solid state device (SSD), or other type of computer readable media which can store data that is accessible by a computing system, such as Universal Serial Bus (USB) flash memory drives, flash memory cards or sticks, optical disks, magnetic tape, standalone RAM disks, read only memory (ROM), and hybrids thereof.
  • Boot subsystem 128 a includes software and/or firmware for performing hardware initialization upon server 112 a powering on or booting up, and to provide runtime services for operating systems and applications. Boot subsystem 128 a may initialize and test the hardware components of server 112 a. Boot subsystem 128 a can also load one or more operating systems from local storage subsystem 126 a or SANs 102 into memory subsystem 122 a. In addition, boot subsystem 128 a can discover and setup one or more peripheral devices for access by processing subsystem 120 a. In some embodiments, server 112 a may store boot subsystem 128 a as one or more boot images in a peripheral memory device connected to server 112 a. Alternatively or in addition, SANs 102 may store one or more boot images from which server 112 a can retrieve the boot image(s). Multiple boot images can represent different hardware, software (e.g., operating systems, applications, etc.), and/or firmware configurations for server 112 a. Examples of implementations of boot subsystem 128 a include basic input/output system (BIOS) for the x86 architecture or the Unified Extensible Firmware Interface (UEFI) or a bootloader for the ARM architecture.
  • In some embodiments, BMC 118 a may program and/or execute boot subsystem 128 a to configure two physical Ethernet links to combine them into one double-capacity Ethernet interface that can mask link failures (at the cost of halving capacity); configure an Ethernet link to split it into an out-of-band management Ethernet interface with its own MAC address for exclusive use by BMC 118 a and one or more in-band Ethernet interfaces with different MAC addresses; configure a group of disks to organize them as a RAID configuration to form one or more fault-tolerant disks; configure PCIe devices to expose them or hide them from one or more operating systems of server 112 a; and monitor and manage power supplies, voltages, clocks, CPU speed, temperatures, and fans.
  • In some embodiments, BMC 118 a may also program and/or execute boot subsystem 128 a to configure memory controllers to limit access to certain portions or ranges of memory subsystem 122 a to certain processors of processing subsystem 120 a. For example, server 112 a may include multiple memory controllers and multiple processors (e.g., multiple sockets and/or multiple cores per processor). BMC 118 a may program and/or execute boot subsystem 128 a to limit the access of certain processors to particular sections of memory to effectively partition the memory among the processors. Alternatively or in addition, BMC 118 a can program and/or execute boot subsystem 128 a to limit the access of certain cores to specific portions of memory and control whether the cores can interrupt one another. In some embodiments, BMC 118 a may also program and/or execute boot subsystem 128 a to control whether certain virtual or physical peripheral devices are accessible to specific processors or cores. In some embodiments, BMC 118 a can load different boot images for each partition of processors and/or cores and thereby each partition can bootstrap independently. In this manner, BMC 118 a can create a number of segregated resource groups with each group comprising one or more processors or cores, a memory range, and one or more accessible peripheral devices (and/or a PCIe address range). Thus, BMC 118 a can perform operations similar to a conventional hypervisor without the overhead that may come with operating a host operating system and a guest operating system. This approach is also an improvement upon containerization because server 112 a is no longer limited to a single operating system.
  • Full-slot server 112 b includes network adapters 116 b and 116 c, BMC 118 b, processing subsystem 120 b, memory subsystem 122 b, I/O subsystem 124 b, local storage subsystem 126 b, and boot subsystem 128 b. One of ordinary skill in the art will appreciate that full-slot server is in many respects similar to full-slot server 112 b. However, full-slot server 112 b includes more overall resources than half-slot server 112 a. For example, full-slot server 112 b can include more processors (and/or cores), more DIMMs, and more I/O interfaces (including network interfaces), thus providing greater processing power, memory, and networking capabilities relative to half-slot server 112 a.
  • Servers 112 connect to port extender 114 via network adapters 116 a and 116 b. A port extender, standardized by the Institute of Electrical and Electronics Engineers (IEEE) 802.1Qbh protocol, can operate as an access device for use in NICs, blade switches, top-of-rack (TOR) switches, hypervisors, single root I/O virtualization (SR-IOV) adapters, virtual switches, etc. A port extender can attach to a MAC port of an 802.1Q bridge (i.e., a controlling bridge) to provide additional MAC ports (downlink interfaces) that are logical ports of the 802.1Q bridge to which the port extender attaches. Packets can flow from a port extender through the controlling bridge to allow consistent forwarding and policy enforcement for traffic. The controlling bridge can operate as a logically centralized management point for its collection of port extenders. Examples of implementations of port extenders 114 include Cisco® Fabric Extender (FEX) Technology, such as the Cisco Nexus® Fabric Extenders (FEX) for providing additional ports for TOR switches, Cisco UCS® Fabric Extenders for providing additional ports for the Cisco UCS® Blade Server Chassis, Cisco® Adapter Fabric Extender (Adapter FEX) for providing additional ports for a server, and the Cisco® Data Center Virtual Machine Fabric Extender (VM-FEX) for providing additional ports for virtual machines.
  • Port extenders 114 can each include interconnect infrastructure I, chassis manager M, and chassis management switch S. Interconnect infrastructure I can operate as a bridge between servers 112 and switch/routers 108 and implement the data plane of the port extenders 114. Examples of implementations of interconnect infrastructure I are Cisco® FEX ASICs, such as Redwood and Woodside. Chassis manager M can interact with network-wide manager N in switch/routers 108 and BMC 118 in servers 112. Chassis manager M can perform server discovery and sequencing, power management, temperature monitoring, and fan control looping. In some embodiments, when there are multiple port extenders 114 in server chassis 110, as in the example of FIG. 1, chassis managers M may form a cluster with one manager in an active state and another in an inactive state according to a high-availability algorithm. For example, there can be a serial interface between chassis managers M for receiving heartbeats between the two managers. Failover can occur either by failure to detect a heartbeat or unplugging of the active chassis manager. Network-manager N may also force a fail-over. Examples of implementations of chassis managers M include Cisco® Chassis Management Controller (CMC) ASICs. Chassis manager switch S can provide connectivity to BMC 118 present on each server 112. Examples of implementations of chassis manager switch S include Cisco® Chassis Management Switch (CMS) ASICs.
  • Port extenders 114 connect server chassis 110 to switches/ routers 108 a and 108 b (collectively, “108”). Switches/routers 108 can operate as spine switches in a spine-and-leaf topology; aggregation/distribution switches and/or core switches in three-tier, multi-tier, or fat tree topologies; a Level 1+ switch in a BCube network topology; or other switch/router in a suitable network topology (e.g., DCell, CamCube, FiConn, Jelly fish, Scafida, etc.). Examples of implementations of switch/routers 108 include Cisco UCS® Fabric Interconnects, Cisco Catalyst® switches, Cisco Nexus® switches, Cisco® Industrial Ethernet switches, Cisco Meraki™ or Meraki® switches/routers, Cisco® Integrated Services Routers (ISR), Cisco® Network Convergence System (NCS) routers, Cisco® Aggregation Services Routers (ASR), Cisco® Industrial ISR, and Cisco® Connected Grid Routers, among others.
  • Switches/routers 108 include port controller ASICs P, crossbar fabric ASIC X, and network-wide manager N. Port controller ASICs P control a small group of ports (e.g., 4, 8, 16, etc.) for processing packets upon egress or egress for managed ports. Port controller ASICs P can also handle forwarding decisions between ports. Examples of implementations of port controller ASICs P include Cisco® Unified Port Controller (UPC) ASICs.
  • Crossbar fabric ASIC X operates as a bridge between port controllers P, and can manage packet switching and scheduling. That is, crossbar fabric ASIC X can couple a port controller for an ingress port to the port controller for an egress port to enable traffic flow between the ports. Examples of implementations of crossbar fabric ASIC X include Cisco® Unified Crossbar Fabric (UCF) ASICs.
  • Network-wide manager N can include hardware, software, and/or firmware for monitoring and managing server, network, and storage infrastructure of network environment 100. Network-wide manager N can provision server, fabric, and storage resources as well as perform device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection. For example, network-wide manager N can automatically detect, inventory, manage, and provision system components added to or changed in network environment 100. Network-wide manager N can also manage clustering, switch redundancy, and otherwise ensure high availability for server, network, and storage resources in a data center and/or a remote network or cloud. Examples of implementations of network-wide manager N include Cisco UCS® Central, Cisco UCS® Director, Cisco UCS® Manager, Cisco UCS® Performance Manager, Cisco® IMC Supervisor, Cisco® Application Policy Infrastructure Controller (APIC), Cisco ONE™ Enterprise Cloud, Cisco® Intelligent Automation, Cisco® Intercloud Fabric, Cisco® Network Services Data Center Network Manager, Cisco Prime® Network Services Controller, or other system for monitoring and managing multiple servers, the network fabric, and/or server storage.
  • Switches/routers 108 can support various types of traffic and include various types of ports and port controllers for connecting servers 112 to other networks, such as SANs 102, LAN 104, management network 106, or any communicative platform operable to exchange data or information within or between computing systems (e.g., Internet, ad-hoc local network, packet data network (PDN), LAN, metropolitan area network (MAN), wide area network (WAN), wireless local area network (WLAN), virtual private network (VPN), intranet, or any other appropriate architecture or system that facilitates electronic communications). For example, switches/routers 108 can include a number of 40 Gbps (Quad Small Form-Factor Pluggable) (QSFP) or QSFP+ ports that can operate at native-40-Gbps speed, or that can operate as four 10-Gbps ports (e.g., by inserting a QSFP-to-4 small form factor plus pluggable (SFP+) breakout cable) for handling Ethernet/IP traffic (e.g., traffic to/from LAN 104), Fibre Channel (FC) or Fibre Channel on Ethernet (FCoE) ports for handling block storage traffic (e.g., traffic to/from SANs 102), and serial ports for handling management traffic (e.g., traffic to/from management network 106) and inter-process communications (IPC) (e.g., high availability, clustering, virtualization platform services, etc.).
  • FIG. 2A and FIG. 2B illustrate computing systems 200 and 250. In some embodiments, computing systems 200 and 250 can be respective implementations of servers 112 a and 112 b of FIG. 1. This disclosure provides additional details regarding each of the components in FIG. 2A and FIG. 2B in more detail below. However, one skilled in art will understand that computing systems 200 and 250 are each simply one possible configuration and that other configurations with more or fewer components are also possible.
  • Computing system 200 is a half-slot, two-socket server including processors 202 a and 202 b (collectively, “202”). Sockets are mounting/interconnection structures for installing processors on a printed circuit board (PCB) or mother board of a computing system. Multiple sockets can provide for customization of a server motherboard by enabling mounting of processors using different clock speeds and/or amounts of power. Each processor 202 can include one or more cores (e.g., 2, 4, 6, 8, etc.) (not shown), each of which can replicate a basic central processing unit (CPU). Each core may be associated with a level 1 (L1) cache (not shown). Caches are small fast memories that can reduce the average time to access main memory. The cores generally share a larger level 2 (L2) or level 3 (L3) cache, a bus or interconnection interface, and external die connections. The number of processors of a computing system is the product of the number of sockets and the number of cores per socket. For example, computing system 200 includes two sockets and can include four cores per socket for a total of eight processors. An example of an implementation of processor 202 a or 202 b is the Xeon Processor provided by Intel Corp. of Santa Clara, Calif.
  • Processors 202 connect to one another and to I/O hub 204 by interconnections 206 a, 206 b, and 206 c (collectively, “206”). In this example, processors 202 and interconnections 206 implement the QuickPath Interconnect (QPI) architecture provided by Intel. QPI utilizes multiple high-speed uni-directional links for interconnecting processors and a chipset or similar computing system element and integrating multiple distributed memory controllers for multiple cores of the processor chip. In this example, the cores inside a socket may share integrated memory controllers (IMCs) (not shown) that have multiple memory interfaces (i.e., memory buses). The IMCs may have various external connections, such as DDR3 memory (or other suitable memory) channels for connecting processors 202 to DDR3 (or other suitable memory) DIMMs D. IMCs and cores in different sockets can talk to one another using QPI. Processors implementing QPI may also have full access to the memory of every other processor while maintaining cache coherency using a cache-coherent Non-Uniform Memory Architecture (NUMA). However, various embodiments may limit the memory range each processor 202 can access as discussed further below. A computing system that implements QPI can achieve global memory reachability (or reachability by one socket to memory of another socket) using an internal crossbar router in the socket. This route-through capability may allow for computing systems without a fully connected topology. Other embodiments may implement interconnections 206 using other types of buses or interconnections, such as front-side buses (FSBs), dual independent buses (DIBs), Dedicated High-Speed Interconnect (DHSI), etc.
  • I/O hub 204 (sometimes referred to as a chipset) connects processors 202 to I/O controller hub (ICH) 208 using interconnection 210, local storage controller 212 using interconnection 216 a, and mezzanine card 214 using interconnection 216 b. An example of an implementation of I/O hub 204 is the X58 chip provided by Intel. In this example, interconnection 210 implements the Direct Media Interface (DMI) provided by Intel, and interconnections 216 a and 216 b are a PCIe x4 link and a PCIe x16 link, respectively.
  • Local storage controller 212 connects computing system 200 to local storage devices (e.g., HDD, SSD, etc.). Local storage controller 212 (sometimes referred to as a SAS controller) can support Serial Attached Small Computer System Interface (SAS) and Serial Advanced Technology Attachment (SATA) transfer rates, and integrate mirroring and striping functions to provide different RAID availability levels for internal drives. An example of an implementation of local storage controller 212 is the 1064e storage processor provided by LSI Corp. of San Jose, Calif.
  • Mezzanine card 214 (i.e., a network adapter) is a PCB assembly that combines the electrical characteristics of the PCI bus (e.g., IEEE P1386.1) with the mechanical characteristics of the Common Mezzanine Card (CMC) format (e.g., IEEE 1386). Mezzanine card 214 can include a number of bus connectors (not shown), such as for connecting to one or more 32-bit PCI buses, 64-bit PCI buses, or other non-specified, non-standardized, and/or proprietary I/O bus. Other card standards supported can include the PCI Mezzanine Card (PMC) eXtended (PMC-X), Processor PMC (PPMC), conduction-cooled PMC (CCPMC), Switched Mezzanine Card (XMC), or FMC—FPGA Mezzanine Card standards.
  • I/O controller hub 208 (sometimes referred to as a Southbridge) connects computing system 200 to relatively low-speed peripheral devices (not shown) (e.g., USB devices or other devices slower than mezzanine card 214), BMC 218, and boot subsystem 220. An example of an implementation of I/O controller hub 208 is the ICH10 I/O controller hub provided by Intel®. In this example, interconnection 216 c between ICH 208 and BMC 218 is a PCIe x4 link. BMC 218 provides management access to computing system 200 prior to the loading of an operating system, and can operate as an aggregation point for server hardware. In some embodiments, BMC 218 can have two integrated Ethernet connections (not shown) connected in a redundant manner to management components of access devices (e.g., chassis management switches S). BMC 118 of FIG. 1 is an example of an implementation of BMC 218.
  • Boot subsystem 220 includes software or firmware for initializing hardware upon powering on or booting up computing system 200 (e.g., BIOS, UEFI, bootloader, etc.) or other management software and/or firmware executed prior to loading of one or more operating systems of computing system 200. Boot subsystem 128 is an example of an implementation of boot subsystem 220.
  • Computing system 250 is a full-slot, four-socket server including processors 252 a, 252 b, 252 c, and 252 d (collectively, “252”). Each processor 252 can include one or more cores (e.g., 4, 6, 8, etc.) (not shown), each of which represents a discrete processing element. In this example, each processor 252 can include 4 cores such that the CPU of computing system 250 includes a total of 16 processors. Examples of implementations of processors 252 include the Xeon 7500 series CPUs provided by Intel. Processors 252 are fully interconnected with one another using interconnections 254 a, 254 b, 254 c, 254 d, 254 e, and 254 f (collectively, “254”). Processors 252 c and 252 d also connect to I/O hub 256 using interconnections 254 g and 254 h, respectively. In this example, processors 252 a and 252 b may connect to I/O hub 256 through core-to-core QPI links. In other embodiments, processors 252 a and 252 b may connect to a second I/O hub symmetrical that enable such a computing system to include additional memory and/or PCIe slots. In this example, interconnections 254 are QPI links but other embodiments may utilize other types of buses or interconnections as discussed elsewhere.
  • Each processor 252 also connects to one or more serial memory buffers (SMBs) 258 using Serial Memory Interface (SMI) links 260. Each SMB 258 can connect to one or more DIMMs D (e.g., DDR3 DIMM). In an embodiment, a computing system can include four sockets with each socket connected to four SMBs and each SMB connected to two DIMMs providing a total of 32 DIMMs.
  • Interconnections 262 b and 262 c (collectively, “262”) are the two main I/O paths toward mezzanine cards 264 a and 264 b (collectively, “264”), respectively. In this example, interconnections 262 b and 262 c are PCIe x16 links. Mezzanine cards 264 (e.g., network adapters 116 of FIG. 1) can provide connections to access devices (e.g., port extenders 114 of FIG. 1). Local storage controller 266 (e.g., an embedded 6G SAS RAID controller connects to I/O hub 256 through a PCIe x4 link.
  • ICH 268 also connects to I/O hub 256 by interconnection 270 (e.g., an Enhanced Function Interface (EFI) bus). ICH 268 provides connections to BMC 272, Front panel 274, and boot subsystem 276. Front panel 274 can include one or more USB ports and/or ports for other low-speed peripheral devices (not shown). One of ordinary skill in the art will appreciate that computing system 200 is in many respects similar to computing system 250. However, computing system 250 is a full-slot server that includes more overall resources than computing system 200, a half-slot server. For example, computing system 250 includes more sockets and cores than computing system 200, and thus the CPU of computing system 250 includes more processors than the CPU of computing system 200, more DIMMs and thus more main memory, and more I/O interfaces (including mezzanine cards). Thus, computing system 250 has greater processing power, memory, and networking capabilities than computing system 200. Although computing systems 200 and 250 illustrate examples of the x86 architecture, other embodiments may utilize other server architectures (e.g., ARM, MIPS, Power, SPARC, other Reduced Instruction Set Computer (RISC) architectures, and/or other Complex Instruction Set Computer (CISC) architectures), and one of ordinary skill in the art could readily apply the principles disclosed in this disclosure to these other architectures.
  • FIG. 3 illustrates an example of process 300 for physically partitioning resources of a computing system via a baseboard management controller of the system. One of ordinary skill will understood that, for any method discussed herein, there can be additional, fewer, or alternative steps performed in similar or alternative orders, or in parallel, within the scope of the various embodiments unless otherwise stated. In some embodiments, a network, and particularly a network-wide manager (e.g., network-wide manager N of FIG. 1) or other system for monitoring and managing multiple servers, the network fabric, and/or server storage can communicate with a baseboard management controller to perform process 300. In other embodiments, a standalone baseboard management controller can perform process 300. Process 300 may begin with a physical server or host powering on and plugging into a network (e.g., from an off state, after reset, after plug-in into a chassis/rack, after insertion as a card into a slot, etc.).
  • At step 302, a baseboard management controller (BMC) (e.g., BMC 118 a or 118 b) can determine the availability of multiple processors of the host of the BMC for provisioning to an end user. A processor may include a single-socket central processing unit (CPU), a pair of CPUs or processors of a four-socket server, a multi-core processor, a single core of a multi-core processor, or other discrete physical device capable of executing the instructions of a computer program by performing the basic arithmetic, logical, control, and input/output (I/O) operations specified by the instructions. If there are no processors available, process 300 can come to an end. Otherwise, process 300 can continue onto step 304 at which the BMC can allocate at least a first processor of the physical host to a first resource group and a second processor of the physical host to a second resource group.
  • Proceeding to step 306, the BMC can load one or more boot images (e.g., BIOS, UEFI boot manager, boot loader, bootstrap, etc.) and/or other configuration data from a SAN (e.g., SAN 102 a or 102 b of FIG. 1) or other remote source or a storage device embedded on the physical host (e.g., ROM, flash memory, or other non-volatile memory) (e.g., boot subsystem 128 a or 128 b, boot subsystem 220 of FIG. 2A, or boot subsystem 276 of FIG. 2B) for initializing the hardware of the physical host. In some embodiments, the BMC can receive a single boot image for configuring multiple physical partitions of the host. In other embodiments, the BMC may receive multiple boot images, including at least one unique boot image for each different resource group. In this manner, each resource group can boot-up independently from one another and apply different programmable code-signing keys for giving the BMC control over which operating system each resource group may load.
  • After loading the boot image(s), at step 308 the BMC/boot image(s) can partition the main memory (i.e., primary storage, internal memory, RAM, etc.) of the physical host into at least a first memory range for exclusive use by the first processor and a second memory range for exclusive use by the second processor. For example, the first memory range can comprise a first set of DIMMs and the second memory range can comprise a second set of DIMMs. The BMC/boot image(s) can generate a first memory map that maps the first memory range to the first set of DIMMs and excludes mappings to other DIMMs. The BMC/boot image(s) can also generate a second memory map that maps the second memory range to the second set of DIMMs and excludes mappings to other DIMMs, including the first set of DIMMs. The BMC/boot image(s) can allocate the resources of the computing system in a variety of configurations.
  • FIG. 4A, FIG. 4B, and FIG. 4C illustrate different ways that a BMC can physically partition the processors and main memory of a computing system. In particular, FIG. 4A illustrates a computing system 400 having four sockets for four CPUs 402 a, 402 b, 402 c, and 402 d (collectively, “402”). Each processor 402 includes integrated memory controller (IMC) 404 for accessing a respective set of DDR3 DIMMs D. In this example, a BMC (not shown) partitions computing system 400 into resource groups 406 a, 406 b, 406 c, and 406 d (collectively, “406”). Each resource group (e.g., resource group 406 a) includes a subset of the processing resources of computing system 400 (e.g., processor 402 a); a subset of the memory controllers of computing system 400 (e.g., IMC 404 a); and a subset of memory or a range of the main memory of computing system 400 (e.g., DIMMs D directly connected to IMC 404 a). Although FIG. 4A shows that each resource group 406 includes one CPU, a resource group can also include multiple CPUs. For example, in an embodiment, the BMC can partition computing system into three resource groups, a first resource group including CPUs 402 a and 402 b, a second resource group including CPU 402 c, and a third resource group including CPU 424 c. A resource group can also include a portion of a CPU (i.e., one or more cores of a multi-core processor) as shown in FIG. 4B and discussed further below.
  • In some embodiments, computing system 400 can implement cache coherency by default and disable cache coherency under certain conditions (or vice versa, i.e., disable cache coherency by default and activate cache coherency under certain conditions). A computing system that implements cache coherency seeks uniformity of shared data stored among multiple local caches. Common approaches for achieving cache coherency include snooping and directory-based cache coherency. In snooping, individual caches monitor address lines for access to cached memory locations and invalidate or update a copy of a snooped memory location on write to the corresponding memory location in a different cache. In directory-based cache coherency, each processor stores shared data to a common directory that maintains the coherency between caches. When an entry in the directory changes, the directory either updates or invalidates the other caches. In the example of FIG. 4A, actively maintaining cache coherency is unnecessary because of the isolation between processors 402 and their respective ranges of main memory. Thus, computing system 400 may safely disable cache coherency for improved performance by each resource group.
  • FIG. 4B illustrates computing system 420 including CPU 422 and DIMMs 428 a and 428 b (collectively, “428”). CPU 422 includes six cores, processors 430 a, 430 b, 430 c, 430 d, 430 e, and 430 f (collectively, “430”), and IMCs 424 a and 424 b (collectively, “424”). Computing system 420 may be a single-socket server including only CPU 422 or a multi-socket server including CPU 422 and one or more other CPUs. In this example, a BMC (not shown) separates computing system 420 into at least two resource groups 426 a and 426 b (collectively, “426”). Resource group 426 a includes cores 430 a and 430 b, IMC 424 a, and DIMMs 428 a. Resource group 426 b includes cores 430 c, 430 d, 430 e, and 430 f, IMC 424 b, and DIMMs 428 b.
  • As shown in FIG. 4B, the BMC of computing system 420 is capable of partitioning the cores of a multi-core processor into two or more resource groups with at least one core in a first resource group and a second core in a second resource group. That is, the BMC can limit the memory accessible by cores 430 a and 430 b to DIMMs 428 a; and the BMC can limit the memory accessible by cores 430 c, 430 d, 430 e, and 430 f to DIMMs 428 b. In other embodiments, the BMC can address memory differently for each individual core. For example, in an embodiment, CPU 422 may include additional pins for encoding which core 430 is requesting a cache line in order to limit a particular core to a particular range of memory. A person of ordinary skill in the art will understand that other embodiments may utilize other techniques for addressing memory differently for each core of a multi-core processor.
  • FIG. 4B also shows that the BMC is capable of allocating IMCs 424 a and 424 b among different resource groups. IMCs 424 a and 424 b may be physically or logically separate elements each associated with a memory map to DIMMs 428 a and 428 b, respectively. In some embodiments, computing system 420 may disable cache coherency at least between DIMMs 428 a and 428 b because of the physical isolation between these memories and their respective processors.
  • In some embodiments, a resource group may also be associated with various priority levels. The BMC may configure whether one core may interrupt another core residing in the same socket or whether one CPU may interrupt another CPU mounted on the same motherboard based on priority level.
  • FIG. 4C illustrates computing system 440 including at least CPUs 442 a and 442 b (collectively, “442”) and DIMMs 448 a, 448 b, 448 c, and 448 d (collectively, “448”). CPU 442 a includes four cores, processors 470 a, 470 b, 470 c, and 470 d and IMC 444 a. CPU 442 b also includes four cores, processors 470 e, 470 f, 470 g, and 470 h and IMC 444 b. Although CPUs 442 include the same number of cores in this example, other embodiments may include multi-core multi-processors having different numbers of cores (e.g., 2 and 4 cores, 4 and 6 cores, 2 and 6 cores, etc.). In this example, a BMC (not shown) partitions computing system 440 into at least two resource groups 446 a and 446 b (collectively, “446”). Resource group 446 a includes cores 470 a and 470 b, IMC 444 a, and DIMMs 448 a. Resource group 446 b includes cores 470 c, 470 d, 470 e, 470 f, 470 g, and 470 h, IMCs 444 a and 444 b, and DIMMs 448 b, 448 c, and 448 d. In some embodiments, the BMC may logically partition IMC 444 a into two virtual memory controllers with one virtual controller dedicated to resource group 446 a and another virtual controller dedicated to resource group 446 b.
  • As shown in FIG. 4C, the BMC of computing system 440 can define a resource group including portions of CPUs 442 a and 442 b (i.e., at least one core from CPU 442 a and one core from CPU 442 b). In some embodiments, it may also be possible to logically partition IMCs 444 into multiple virtual memory controllers to limit the access of each processor 442 to a specific memory range or set of DIMMs 448. In this example, computing system 440 may disable cache coherency at least between DIMM 428 a and other DIMMs because of the physical isolation between these memories and their respective processors but maintain cache coherency at least between and among DIMMs 448 b, 448 c, and 448 d.
  • Returning to FIG. 3, process 300 can continue to step 310 in which the BMC/boot image(s) can distribute access to physical or virtual peripheral devices connected to the physical host between and among the first processor and the second processor. The peripheral devices can include network adapters, graphic processing unit (GPU) adapters, Peripheral Component Interconnect Express (PCIe) Flash adapters, Fibre Channel host bus adapters (HBAs), disks, disk controllers, USB devices, etc. In some embodiments, the BMC/boot image(s) can expose one or more PCIe I/O ports to one of the processors and/or hide one or more other PCIe I/O ports from that processor. In some embodiments, the BMC/boot image(s) can map one or more peripheral device's memories into one of the processors' main memory range to give that processor access to the peripheral device(s) and/or deny that processor access to one or more other peripheral devices by not mapping the other peripheral devices' memories into the processor's main memory. In some embodiments, the BMC/boot image(s) can configure a peripheral device to be a bus master, and that peripheral device can act on I/O requests from one of the processors and ignore I/O requests from another processor. In some embodiments, the BMC/boot image(s) may utilize Single Root I/O Virtualization (SR-IOV) to virtualize a physical peripheral device to create multiple virtual peripheral devices accessible by multiple operating systems and their respective processors on a single-socket server. In some embodiments, the BMC/boot image(s) may utilize Multi-Root I/O Virtualization (MR-IOV) to virtualize a physical peripheral device to create multiple virtual peripheral devices accessible by multiple operating systems and their respective processors on a multi-socket server.
  • At step 312, the BMC/boot image(s) can load a first operating system into the first range of main memory associated with the first processor and a second operating system into the second range of main memory associated with the second processor. The operating systems may be different operating systems (e.g., Microsoft Windows, UNIX, Linux, Mac OS X, etc.) or different versions of the same operating system (e.g., Microsoft Windows 7.0 and Microsoft Windows 10.0). As discussed elsewhere herein, the BMC/boot image(s) can retrieve or otherwise receive the operating systems from a SAN or other remote storage or a local mass storage device (e.g., HDD, SDD, flash drive, etc.). Process 300 may conclude at step 314 when the BMC cedes control to the first processor to execute the first operating system and to the second processor to execute the second operating system.
  • For clarity of explanation, in some instances the disclosure may present various embodiments as including individual functional blocks comprising devices, device components, steps or routines in a method embodied in software or firmware, or combinations of hardware, firmware, and/or software.
  • In some embodiments, the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.
  • Computer-executable instructions, stored or otherwise available from computer readable media, can implement methods according to the above-described examples. Such instructions can comprise, for instance, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer-executable instructions may also include binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media for storing instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.
  • Devices implementing methods according to these disclosures can comprise hardware, firmware, and/or software, and can take any of a variety of form factors. Typical examples of such form factors include servers (e.g., mainframe servers, tower servers, rack-mount servers, blade servers, microservers, etc.), small form factor personal computers, laptops, smart phones, personal digital assistants, and so on. Peripherals or add-in cards can also perform some of the functionality described herein. A circuit board including different chips or different processes executing in a single device can also perform some of the functionality, by way of further example.
  • The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.
  • Although a variety of examples and other information explain aspects within the scope of the appended claims, no limitation of the claims are implicit based on particular features or arrangements in such examples as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although the disclosure may describe some subject matter in language specific to examples of structural features and/or method steps, a person having ordinary skill in the art will understand that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps disclosed are examples of components of systems and methods within the scope of the appended claims.

Claims (20)

What is claimed is:
1. A computer-implemented method comprising:
partitioning a portion of a memory for exclusive use by one of a plurality of processors;
mapping a peripheral device to the portion of the memory to provide access to the one of the plurality of processors; and
executing an operating system via the portion of the memory.
2. The computer-implemented method of claim 1, wherein one or more memory controllers are configured to control the portion of the memory.
3. The computer-implemented method of claim 2, further comprising:
configuring one or more other memory controllers to deny the access of the one of the plurality of processors to other ranges of memory, the one or more other memory controllers not having control over the portion of the memory.
4. The computer-implemented method of claim 1, further comprising:
generating a memory map that maps the portion of the memory to a first set of dual inline memory modules (DIMMs) of a physical host and that excludes mappings to other DIMMs of the physical host.
5. The computer-implemented method of claim 1, further comprising:
disabling cache coherency between the portion of the memory and another portion of the memory.
6. The computer-implemented method of claim 1, further comprising:
receiving at least a first boot image including first instructions for loading a first operating system and a second boot image including second instructions for loading a second operating system;
loading the first boot image into the portion of the memory and the second boot image into another portion of the memory;
executing the first instructions for loading the first operating system; and
executing the second instructions for loading the second operating system.
7. The computer-implemented method of claim 1, further comprising:
providing access to the one of the plurality of processors to an input/output (I/O) port by exposing the I/O port to the one of the plurality of processors.
8. The computer-implemented method of claim 1, further comprising:
denying access to the one of the plurality of processors to an I/O port by hiding the I/O port from the one of the plurality of processors.
9. The computer-implemented method of claim 1, further comprising:
mapping memory of a peripheral device to the portion of the memory to provide access to the one of the plurality of processors to the peripheral device.
10. The computer-implemented method of claim 1, further comprising:
denying access to a peripheral device by excluding a mapping of memory of the peripheral device to the portion of the memory.
11. The computer-implemented method of claim 1, further comprising:
sending, by the one of the plurality of processors, an I/O request to a peripheral device connected to a physical host; and
receiving, by the one of the plurality of processors, an I/O response from the peripheral device.
12. The computer-implemented method of claim 1, further comprising:
sending, the one of the plurality of processors, an I/O request to a peripheral device connected to a physical host; and
ignoring, by the peripheral device, the I/O request.
13. A server comprising:
a processor; and
a memory including instructions that, upon execution by the processor, cause the processor to:
partition a portion of a memory for exclusive use by one of a plurality of processors;
map a peripheral device to the portion of the memory to provide access to the one of the plurality of processors; and
execute an operating system via the portion of the memory.
14. The server of claim 13,
wherein,
the one of the plurality of processors include a first central processing unit (CPU), and
one or more other processors of the plurality of processors include a second CPU.
15. The server of claim 13,
wherein,
the one of the plurality of processors include a first core of a multi-core processor, and
one or more other processors of the plurality of processors include a second core of the multi-core processor.
16. The server of claim 13,
wherein,
the one of the plurality of processors include a first core of a first multi-core processor, and
one or more other processors of the plurality of processors include a second core of a second multi-core processor.
17. A non-transitory computer-readable medium having instructions that, upon execution by a processor, cause the processor to:
partition a portion of a memory for exclusive use by one of a plurality of processors;
map a peripheral device to the portion of the memory to provide access to the one of the plurality of processors; and
execute an operating system via the portion of the memory.
18. The non-transitory computer-readable medium of claim 17, wherein the instructions upon execution further cause the processor to:
virtualize a physical memory controller to create a first virtual memory controller and a second virtual memory controller;
allocate the first virtual memory controller to a first resource group; and
allocate the second virtual memory controller to a second resource group.
19. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the processor to:
virtualize a physical peripheral device using Single Root I/O Virtualization to create at least a first virtual peripheral device and a second virtual peripheral device;
allocate the first virtual peripheral device to a first resource group; and
allocate the second virtual peripheral device to a second resource group.
20. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the processor to:
virtualize a physical peripheral device using Multi I/O Virtualization to create at least a first virtual peripheral device and a second virtual peripheral device;
allocate the first virtual peripheral device to a first resource group; and
allocate the second virtual peripheral device to a second resource group.
US16/730,430 2017-06-08 2019-12-30 Physical partitioning of computing resources for server virtualization Abandoned US20200142752A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/730,430 US20200142752A1 (en) 2017-06-08 2019-12-30 Physical partitioning of computing resources for server virtualization

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/617,190 US10521273B2 (en) 2017-06-08 2017-06-08 Physical partitioning of computing resources for server virtualization
US16/730,430 US20200142752A1 (en) 2017-06-08 2019-12-30 Physical partitioning of computing resources for server virtualization

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US15/617,190 Continuation US10521273B2 (en) 2017-06-08 2017-06-08 Physical partitioning of computing resources for server virtualization

Publications (1)

Publication Number Publication Date
US20200142752A1 true US20200142752A1 (en) 2020-05-07

Family

ID=62779070

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/617,190 Active 2037-12-22 US10521273B2 (en) 2017-06-08 2017-06-08 Physical partitioning of computing resources for server virtualization
US16/730,430 Abandoned US20200142752A1 (en) 2017-06-08 2019-12-30 Physical partitioning of computing resources for server virtualization

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US15/617,190 Active 2037-12-22 US10521273B2 (en) 2017-06-08 2017-06-08 Physical partitioning of computing resources for server virtualization

Country Status (4)

Country Link
US (2) US10521273B2 (en)
EP (1) EP3635542A1 (en)
CN (1) CN110998523B (en)
WO (1) WO2018227042A1 (en)

Families Citing this family (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10649792B1 (en) 2018-02-09 2020-05-12 American Megatrends International, Llc Cloning of firmware configuration settings using rest over IPMI interface
US10416988B1 (en) 2018-02-09 2019-09-17 American Megatrends International, Llc Peripheral device firmware update using rest over IPMI interface firmware shell utility
US10628176B1 (en) 2018-02-09 2020-04-21 American Megatrends International, Llc Firmware configuration using REST over IPMI interface
US10776286B1 (en) 2018-02-09 2020-09-15 American Megatrends International, Llc Rest over IPMI interface for firmware to BMC communication
US10409584B1 (en) 2018-02-09 2019-09-10 American Megatrends International, Llc Peripheral device firmware update using rest over IPMI interface firmware update module
US10489142B1 (en) 2018-02-09 2019-11-26 American Megatrends International, Llc Secure firmware integrity monitoring using rest over IPMI interface
US10572242B1 (en) * 2018-02-09 2020-02-25 American Megatrends International, Llc Firmware update using rest over IPMI interface
US11614986B2 (en) * 2018-08-07 2023-03-28 Marvell Asia Pte Ltd Non-volatile memory switch with host isolation
US11544000B2 (en) 2018-08-08 2023-01-03 Marvell Asia Pte Ltd. Managed switching between one or more hosts and solid state drives (SSDs) based on the NVMe protocol to provide host storage services
CN111092918B (en) * 2018-10-23 2023-08-18 联想企业解决方案(新加坡)有限公司 Computing node and method for establishing cloud cluster
US11526405B1 (en) 2018-11-18 2022-12-13 Pure Storage, Inc. Cloud-based disaster recovery
US11379254B1 (en) * 2018-11-18 2022-07-05 Pure Storage, Inc. Dynamic configuration of a cloud-based storage system
US11340837B1 (en) 2018-11-18 2022-05-24 Pure Storage, Inc. Storage system management via a remote console
US10540185B1 (en) * 2019-01-11 2020-01-21 Liqid Inc. Software deployment in disaggregated computing platforms
US11392525B2 (en) 2019-02-01 2022-07-19 Liqid Inc. Specialized device instantiation onto PCIe fabrics
US11068424B2 (en) * 2019-02-04 2021-07-20 American Megatrends International, Llc Enablement of software defined storage solution for NVME over ethernet fabric management on a processor
US10849223B2 (en) 2019-03-06 2020-11-24 Cisco Technology, Inc. Multi-socket server assembly
US10997029B2 (en) * 2019-03-07 2021-05-04 International Business Machines Corporation Core repair with failure analysis and recovery probe
US20200342109A1 (en) * 2019-04-29 2020-10-29 Hewlett Packard Enterprise Development Lp Baseboard management controller to convey data
US20210042255A1 (en) * 2019-08-09 2021-02-11 Sony Interactive Entertainment LLC Methods for Using High-Speed Data Communication Fabric to Enable Cross-System Command Buffer Writing for Data Retrieval in Cloud Gaming
DK3787230T3 (en) * 2019-08-29 2021-08-23 Ovh METHOD AND SYSTEM FOR PUSHING A NEW RACK IN OPERATING MODE IN A DATA CENTER
US11010173B2 (en) * 2019-09-17 2021-05-18 Dell Products L.P. Adjusting a processing state of an information handling system from multi-socket mode to multi-single socket mode
US11349733B2 (en) * 2020-03-23 2022-05-31 Quanta Computer Inc. Method and system for automatic detection and alert of changes of computing device components
CN111797439A (en) * 2020-05-18 2020-10-20 联想企业解决方案(新加坡)有限公司 Method and apparatus for providing virtual device
US11232537B2 (en) * 2020-06-03 2022-01-25 Dell Products L.P. System and method for pre-boot dynamic video rendering and graphics interpretation by a virtual graphics browser
CN112134752B (en) * 2020-09-10 2022-05-13 苏州浪潮智能科技有限公司 Method, system, equipment and medium for monitoring switch based on BMC
US11500994B2 (en) * 2020-09-23 2022-11-15 Dell Products L.P. Communication system personality provisioning system
US11847226B1 (en) * 2020-11-25 2023-12-19 American Megatrends International, Llc Baseboard Management Controller (BMC)-based security processor
US11294847B1 (en) * 2020-11-30 2022-04-05 Dell Products L.P. Fibre channel host onboarding system
US11645616B1 (en) * 2020-12-03 2023-05-09 American Megatrends International, Llc Verifying the integrity of data transmitted between a firmware and a baseboard management controller (BMC)
CN113220423B (en) * 2021-06-04 2022-07-19 恒为科技(上海)股份有限公司 Multi-chip management method and device based on container
CN113507405B (en) * 2021-06-22 2022-07-29 电子科技大学 Virtual network node rapid construction method based on virtual resource pool
WO2022272191A1 (en) * 2021-06-23 2022-12-29 Intel Corporation Computing devices and method and computing device for initializing a computing device
US11836100B1 (en) * 2022-06-16 2023-12-05 Dell Products L.P. Redundant baseboard management controller (BMC) system and method
US20240073131A1 (en) * 2022-08-25 2024-02-29 Keysight Technologies, Inc. Methods, systems, and computer readable media for implementing routing path groups between emulated switches
US20240143773A1 (en) * 2022-10-31 2024-05-02 Cisco Technology, Inc. Fabric-based root-of-trust
US11863414B1 (en) * 2022-12-29 2024-01-02 Lenovo Enterprise Solutions (Singapore) Pte Ltd. Running network diagnostics on a server
CN117056275B (en) * 2023-10-10 2024-02-09 苏州元脑智能科技有限公司 Communication control method, device and server based on hardware partition system

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8621458B2 (en) * 2004-12-21 2013-12-31 Microsoft Corporation Systems and methods for exposing processor topology for virtual machines
US7334076B2 (en) * 2005-03-08 2008-02-19 Microsoft Corporation Method and system for a guest physical address virtualization in a virtual machine environment
US20070226795A1 (en) * 2006-02-09 2007-09-27 Texas Instruments Incorporated Virtual cores and hardware-supported hypervisor integrated circuits, systems, methods and processes of manufacture
US8286195B2 (en) * 2007-10-31 2012-10-09 Microsoft Corporation Controlling hardware across two or more simultaneously running operating systems
US7865762B2 (en) * 2007-12-04 2011-01-04 Intel Corporation Methods and apparatus for handling errors involving virtual machines
CN101471820B (en) * 2007-12-28 2011-06-15 英业达股份有限公司 Test method for substrate management controller
WO2010021630A2 (en) 2008-08-22 2010-02-25 Hewlett-Packard Development Company, L.P. Server virtualized using virtualization platform
US9459928B2 (en) * 2008-08-22 2016-10-04 Hewlett Packard Enterprise Development Lp Remote graphics console and virtual media access to virtual machine guests
US8352952B2 (en) * 2008-12-01 2013-01-08 Citrix Systems, Inc. Systems and methods for facilitating virtualization of a heterogeneous processor pool
WO2010127365A1 (en) * 2009-05-01 2010-11-04 Citrix Systems, Inc. Systems and methods for establishing a cloud bridge between virtual storage resources
CN102567275B (en) * 2010-12-08 2014-01-08 中国科学院声学研究所 Method and system for memory access among multiple operation systems on multi-core processor
US9021472B2 (en) 2010-12-10 2015-04-28 Lenovo Enterprise Solutions (Singapore) Pte. Ltd. Virtualizing baseboard management controller operation
US8893147B2 (en) * 2012-01-13 2014-11-18 Ca, Inc. Providing a virtualized replication and high availability environment including a replication and high availability engine
US8978031B2 (en) * 2012-08-21 2015-03-10 International Business Machines Corporation Processing of overlay networks using an accelerated network interface card
CN103034526B (en) * 2012-12-06 2016-04-13 中国电信股份有限公司 A kind of implementation method of virtualization services and device
CN103150217B (en) * 2013-03-27 2016-08-10 无锡江南计算技术研究所 Multicore processor operating system method for designing
TWI588751B (en) 2013-05-31 2017-06-21 聯想企業解決方案(新加坡)有限公司 Computer host with a baseboard management controller to manage virtual machines and method thereof
JP6111181B2 (en) 2013-10-31 2017-04-05 株式会社日立製作所 Computer control method and computer
CN104714846B (en) * 2013-12-17 2018-06-05 华为技术有限公司 Method for processing resource, operating system and equipment
US9672058B2 (en) * 2014-03-13 2017-06-06 Unisys Corporation Reduced service partition virtualization system and method
US9727451B2 (en) 2014-03-28 2017-08-08 Fortinet, Inc. Virtualization in a multi-host environment
US9703726B2 (en) * 2014-06-24 2017-07-11 Bitdefender IPR Management Ltd. Systems and methods for dynamically protecting a stack from below the operating system
US9665309B2 (en) * 2014-06-27 2017-05-30 International Business Machines Corporation Extending existing storage devices in virtualized environments
US10044795B2 (en) 2014-07-11 2018-08-07 Vmware Inc. Methods and apparatus for rack deployments for virtual computing environments
CN104202195B (en) 2014-09-10 2018-05-04 华为技术有限公司 Method, baseboard management controller and the server of server Unified Communication
US9747121B2 (en) * 2015-04-14 2017-08-29 Dell Products L.P. Performance optimization of workloads in virtualized information handling systems
US20170206110A1 (en) * 2016-01-18 2017-07-20 American Megatrends Inc. Computer System for BMC resource management

Also Published As

Publication number Publication date
CN110998523B (en) 2024-05-17
US20180357108A1 (en) 2018-12-13
US10521273B2 (en) 2019-12-31
EP3635542A1 (en) 2020-04-15
CN110998523A (en) 2020-04-10
WO2018227042A1 (en) 2018-12-13

Similar Documents

Publication Publication Date Title
US10521273B2 (en) Physical partitioning of computing resources for server virtualization
US11625335B2 (en) Adaptive address translation caches
US20230208731A1 (en) Techniques to control system updates and configuration changes via the cloud
US20180213669A1 (en) Micro data center (mdc) in a box system and method thereof
US11775464B2 (en) Computer system and a computer device
US11372787B2 (en) Unified address space for multiple links
US10404800B2 (en) Caching network fabric for high performance computing
EP3575969B1 (en) Reducing cache line collisions
US20140047156A1 (en) Hybrid computing system
US20230051825A1 (en) System supporting virtualization of sr-iov capable devices
US11886356B2 (en) Local instantiation of remote peripheral devices
US20240078196A1 (en) Cxl persistent memory module link topology
US20240020174A1 (en) Memory disaggregation in a multi-node environment
US11314455B2 (en) Mapping of RAID-CLI requests to vSAN commands by an out-of-band management platform using NLP
US20230169017A1 (en) Dynamic server rebalancing
WO2023177982A1 (en) Dynamic server rebalancing

Legal Events

Date Code Title Description
AS Assignment

Owner name: CISCO TECHNOLOGY, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MULLENDER, SAPE;BARACH, DAVID RICHARD;MCKIE, JIM;AND OTHERS;SIGNING DATES FROM 20170527 TO 20170608;REEL/FRAME:051388/0258

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION