[<< Back](../../openstack)
# 4. Cloud Infrastructure + VIM Component Level Architecture
## Table of Contents
* [4.1 Introduction](#4.1)
* [4.2 Underlying Resources](#4.2)
* [4.2.1 Virtualisation](#4.2.1)
* [4.2.2 Compute](#4.2.2)
* [4.2.3 Network Fabric](#4.2.3)
* [4.2.4 Storage Backend](#4.2.4)
* [4.3 Virtualised Infrastructure Manager (VIM)](#4.3)
* [4.3.1 VIM Services](#4.3.1)
* [4.3.2 Containerised OpenStack Services](#4.3.2)
* [4.4 Consumable Infrastructure Resources and Services](#4.4)
* [4.4.1 Support for Cloud Infrastructure Profiles and flavors](#4.4.1)
* [4.4.2 Logical segregation and high availability](#4.4.2)
* [4.4.3 Transaction Volume Considerations](#4.4.3)
* [4.5 Cloud Topology and Control Plane Scenarios](#4.5)
* [4.5.1 Edge Cloud Topology](#4.5.1)
## 4.1 Introduction.
Chapter 3 introduced the components of an OpenStack-based IaaS
- Consumable Infrastructure Resources and Services
- Cloud Infrastructure Management Software (VIM: OpenStack) core services and architectural constructs needed to consume and manage the consumable resources
- Underlying physical compute, storage and networking resources
This chapter delves deeper into the capabilities of these different resources and their needed configurations to create and operate an OpenStack-based IaaS cloud. This chapter specifies details on the structure of control and user planes, operating systems, hypervisors and BIOS configurations, and architectural details of underlay and overlay networking, and storage, and the distribution of OpenStack service components among nodes. The chapter also covers implementation support for the [Reference Model profiles and flavours](../../../ref_model/chapters/chapter02.md#24-profiles--flavours); the OpenStack flavor types capture both the sizing and the profile configuration (of the host).
## 4.2 Underlying Resources
### 4.2.1 Virtualisation
In OpenStack, KVM is configured as the default hypervisor for compute nodes.
- Configuration: [OpenStack](https://docs.openstack.org/nova/wallaby/admin/configuration/hypervisor-kvm.html) specifies the steps/instructions to configure KVM:
- Enable KVM based hardware virtualisation in BIOS. OpenStack provides instructions on how to enable hardware virtualisation for different hardware platforms (x86, Power)
- QEMU is similar to KVM in that both are libvirt controlled, have the same feature set and utilise compatible virtual machine images
- Configure Compute backing storage
- Specify the CPU Model for KVM guests (VMs)
- KVM Performance Tweaks
- [Hardening the virtualisation layers](https://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html)
- OpenStack recommends minimizing the code base by removing unused components
- sVirt (Secure Virtualisation) provides isolation between VM processes, devices, data files and system processes
### 4.2.2 Compute
#### 4.2.2.1 Cloud Deployment (Foundation/management) Node
Minimal configuration: 1 node
#### 4.2.2.2 OpenStack Control Plane Servers (Control Nodes)
- BIOS Requirements
For OpenStack control nodes we use the BIOS parameters for the basic profile defined in [Chapter 5.4 of the Reference Model](../../../ref_model/chapters/chapter05.md#5.4). Additionally, for OpenStack we need to set the following boot parameters:
| BIOS/boot Parameter | Value |
|--------------------|--------------------|
| Boot disks |RAID 1 |
| CPU reservation for host (kernel) |1 core per NUMA |
| CPU allocation ratio |2:1 |
- How many nodes to meet SLA
- Minimum 3 nodes for high availability
- HW specifications
- Boot disks are dedicated with Flash technology disks
- Sizing rules
- It is easy to horizontally scale the number of control nodes
- The number of control nodes is determined by a minimum number needed for high availability (viz., 3 nodes) and the extra nodes needed to handle the transaction volumes, in particular, for Messaging service (e.g., RabbitMQ) and Database (e.g., MySQL) to track state.
- The number of control nodes only needs to be increased in environments with a lot of changes, such as a testing lab, or a very large cloud footprint (rule of thumb: number of control nodes = 3 + quotient(number of compute nodes/1000)).
- The [Services Placement Summary table](https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html) specifies the number of instances that are required based upon the cloud size (number of nodes).
#### 4.2.2.3 Network nodes
Networks nodes are mainly used for L3 traffic management for overlay tenant network (see more detail in section 4.3.1.5 Neutron).
- BIOS requirements
| BIOS/boot Parameter | Value |
|--------------------|--------------------|
| Boot disks |RAID 1 |
- How many nodes to meet SLA
- Minimum 2 nodes for high availibility using VRRP.
- HW specifications
- 3 NICs card are needed if we want to isolate the different flows:
- 1 NIC for Tenant Network
- 1 NIC for External Network
- 1 NIC for Other Networks (PXE, Mngt ...)
- Sizing rules
- Scale out of network node is not easy
- DVR can be an option for large deployment (see more detail in chapter 4.3.1.5 - Neutron)
#### 4.2.2.4 Storage nodes
- BIOS requirements
| BIOS/boot Parameter | Value |
|--------------------|--------------------|
| Boot disks |RAID 1 |
- HW specifications: please see [RM Chapter 3.6](../../../ref_model/chapters/chapter03.md#3.6)
- How many nodes to meet SLA: Active-Passive is the default and recently OpenStack started to support Active-Active
- Sizing rules: minimum 2 x 1 TB; recommended 2 x 10 TB
#### 4.2.2.5 Compute Nodes
This section specifies the compute node configurations to support the Basic and High-Performance profiles; in OpenStack this would be accomplished by specifying the configurations when creating "flavors". The cloud operator may choose to implement certain profile-extensions ([RM 2.4 Profile Extensions](../../../ref_model/chapters/chapter02.md#242-profile-extensions-specialisations)) as a set of standard configurations, of a given profile, capturing some of the variability through different values or extra specifications.
- The software and hardware configurations are as specified in the [Reference Model chapter 5.4](../../../ref_model/chapters/chapter05.md#5.4)
- BIOS requirement
- The general BIOS requirements are described in the [Reference Model chapter 5.4](../../../ref_model/chapters/chapter05.md#5.4)
**Example Profiles and their Extensions**
The Reference Model specifies the Basic (B) and High-Performance (H) profile types. The Reference Model also provides a choice of network acceleration capabilities utilising, for example, DPDK and SR-IOV technologies. Table 4-2 lists a few simple examples of profile extensions and some of their capabilities.
| Profile Extensions | Description | CPU Allocation Ratio | SMT | CPU Pinning | NUMA | Huge pages | Data Traffic |
|----|----|----|----|----|----|----|----|
| B1 | Basic Profile
No CPU over-subscription profile extension | 1:1 | Y | N | N | N | OVS-kernel |
| B4 | Basic Profile
4x CPU over-subscription profile extension | 4:1 | Y | N | N | N | OVS-kernel |
| HV | High Performance Profile | 1:1 | Y | Y | Y | Y | OVS-kernel |
| HD | High Performance Profile
with DPDK profile extension | 1:1 | Y | Y | Y | Y | OVS-DPDK |
| HS | High Performance Profile
with SR-IOV profile extension | 1:1 | Y | Y | Y | Y | SR-IOV |
Table 4-2: Profile Extensions and Capabilities
**BIOS Settings** A number of capabilities need to be enabled in the BIOS (such as NUMA and SMT); the Reference Model section on "[Cloud Infrastructure Software profile description](../../../ref_model/chapters/chapter05.md#5.1)" specifies the capabilities required to be configured. Please note that capabilities may need to be configured in multiple systems. For OpenStack, we also need to set the following boot parameters: | BIOS/boot Parameter | Basic | High Performance | |---------------|-----------|------------------| | Boot disks | RAID 1 | RAID 1 | - How many nodes to meet SLA - minimum: two nodes per profile - HW specifications - Boot disks are dedicated with Flash technology disks - In case of DPDK usage: | Layer | Description | |-------------|--| | Cloud infrastructure | Important is placement of NICs to get NUMA-balanced system (balancing the I/O, memory, and storage across both sockets), and configuration of NIC features. Server BIOS and Host OS kernel command line settings are described in [DPDK release notes](http://doc.dpdk.org/guides/rel_notes/) and [DPDK performance reports](http://core.dpdk.org/perf-reports/). Disabling power settings (like Intel Turbo Boost Technology) brings stable performance results, although understanding if and when they benefit workloads and enabling them can achieve better performance results. | | Workload | DPDK uses core affinity along with 1G or 2M huge pages, NUMA settings (to avoid crossing inteconnect between CPUs), and DPDK Poll Mode Drivers (PMD, on reserved cores) to get the best performance. DPDK versions xx.11 are Long-Term Support maintained stable release with back-ported bug fixes for a two-year period. | - Sizing rules | Description | Mnemonic | | ------------|--| | Number of CPU sockets| s | | Number of cores| c | | SMT| t | | RAM| rt | | Storage| d | | Overcommit| o | | Average vCPU per instance | v | | Average RAM per instance | ri | | | | Basic | High-Performance | |---------------|------------|------------|------------| | # of VMs per node (vCPU) | (s*c*t*o)/v | 4*(s*c*t)/v | (s*c*t)/v| | # of VMs per node (RAM) | rt/ri | rt/ri | rt/ri | | Max # of VMs per node| | min(4*(s*c*t)/v, rt/ri)| min((s*c*t)/v, rt/ri)| Caveats: - These are theoretical limits - Affinity and anti-affinity rules, among other factors, affect the sizing #### 4.2.2.6 Compute Resource Pooling Considerations - Multiple pools of hardware resources where each resource pool caters for workloads of a specific profile (for example, High-Performance) leads to inefficient use of the hardware as the server resources are configured specifically for a profile. If not properly sized or when demand changes, this can lead to oversupply/starvation scenarios; reconfiguration may not be possible because of the underlying hardware or inability to vacate servers for reconfiguration to support another profile type. - Single pool of hardware resources including for controllers have the same CPU configuration. This is operationally efficient as any server can be utilised to support any profile or controller. The single pool is valuable with unpredictable workloads or when the demand of certain profiles is insufficient to justify individual hardware selection. #### 4.2.2.7 Reservation of Compute Node Cores The [RA-1 2.3.2 Infrastructure Requirements](./chapter02.md#232-infrastructure-requirements) `inf.com.08` requires the allocation of "certain number of host cores/threads to non-tenant workloads such as for OpenStack services." A number ("n") of random cores can be reserved for host services (including OpenStack services) by specifying the following in nova.conf: reserved_host_cpus = n where n is any positive integer. If we wish to dedicate specific cores for host processing we need to consider two different usage scenarios: 1. Require dedicated cores for Guest resources 2. No dedicated cores are required for Guest resources Scenario #1, results in compute nodes that host both pinned and unpinned workloads. In the OpenStack Wallaby release, scenario #1 is not supported; it may also be something that operators may not allow. Scenario #2 is supported through the specification of the cpu_shared_set configuration. The cores and their sibling threads dedicated to the host services are those that do not exist in the cpu_shared_set configuration. Let us consider a compute host with 20 cores with SMT enabled (let us disregard NUMA) and the following parameters specified. The physical cores are numbered '0' to '19' while the sibling threads are numbered '20' to '39' where the vCPUs numbered '0' and '20', '1' and '21', etc. are siblings: cpu_shared_set = 1-7,9-19,21-27,29-39 (can also be specified as cpu_shared_set = 1-19,^8,21-39,^28) This implies that the two physical cores '0' and '8' and their sibling threads '20' and '28' are dedicated to the host services, and 19 cores and their sibling threads are available for Guest instances and can be over allocated as per the specified cpu_allocation_ratio in nova.conf. #### 4.2.2.8 Pinned and Unpinned CPUs When a server (viz., an instance) is created the vCPUs are, by default, not assigned to a particular host CPU. Certain workloads require real-time or near real-time behavior viz., uninterrupted access to their cores. For such workloads, CPU pinning allows us to bind an instance’s vCPUs to particular host cores or SMT threads. To configure a flavor to use pinned vCPUs, we use a dedicated CPU policy. openstack flavor set .xlarge --property hw:cpu_policy=dedicated While an instance with pinned CPUs cannot use CPUs of another pinned instance, this does not apply to unpinned instances; an unpinned instance can utilise the pinned CPUs of another instance. To prevent unpinned instances from disrupting pinned instances, the hosts with CPU pinning enabled are pooled in their own host aggregate and hosts with CPU pinning disabled are pooled in another non-overlapping host aggregate. #### 4.2.2.9 Compute node configurations for Profiles and OpenStack Flavors This section specifies the compute node configurations to support profiles and flavors. ##### 4.2.2.9.1 Cloud Infrastructure Hardware Profile The Cloud Infrastructure Hardware (or simply “host”) profile and configuration parameters are utilised in the reference architecture to define different hardware profiles; these are used to configure the BIOS settings on a physical server and configure utility software (such as Operating System and Hypervisor). An OpenStack flavor defines the characteristics (“capabilities”) of a server (viz., VMs or instances) that will be deployed on hosts assigned a host-profile. A many to many relationship exists between flavors and host profiles. Multiple flavors can be defined with overlapping capability specifications with only slight variations that servers of these flavor types can be hosted on similary configured (host profile) compute hosts. Similarly, a server can be specified with a flavor that allows it to be hosted on, say, a host configured as per the Basic profile or a host configured as per the High-Performance profile. Please note that workloads that specify a server flavor so as to be hosted on a host configured as per the High-Performance profile, may not be able to run (adequately with expected performance) on a host configured as per the Basic profile. A given host can only be assigned a single host profile; a host profile can be assigned to multiple hosts. Host profiles are immutable and hence when a configuration needs to be changed, a new host profile is created. ##### 4.2.2.9.2 CPU Allocation Ratio and CPU Pinning A given host (compute node) can only support a single CPU Allocation Ratio. Thus, to support the B1 and B4 Basic profile extensions (Section 4.2.2.5) with CPU Allocation Ratios of 1.0 and 4.0 we will need to create 2 different host profiles and separate host aggregates for each of the host profiles. The CPU Allocation Ratio is set in the hypervisor on the host. > When the CPU Allocation Ratio exceeds 1.0 then CPU Pinning also needs to be disabled. ##### 4.2.2.9.3 Server Configurations The different networking choices – OVS-Kernel, OVS-DPDK, SR-IOV – result in different NIC port, LAG (Link Aggregation Group), and other configurations. Some of these are shown diagrammatically in section 4.2.9.5. ##### 4.2.2.9.4 Leaf and Compute Ports for Server Flavors must align Compute hosts have varying numbers of Ports/Bonds/LAGs/Trunks/VLANs connected with Leaf ports. Each Leaf port (in A/B pair) must be configured to align with the interfaces required for the compute flavor. Physical Connections/Cables are generally the same within a zone, regardless of these specific L2/L3/SR-IOV configurations for the compute. **Compute Bond Port:** TOR port maps VLANs directly with IRBs on the TOR pair for tunnel packets and Control Plane Control and Storage packets. These packets are then routed on the underlay network GRT. Server Flavors: B1, B4, HV, HD **Compute SR-IOV Port:** TOR port maps VLANs with bridge domains that extend to IRBs, using VXLAN VNI. The TOR port associates each packet’s outer VLAN tag with a bridge domain to support VNF interface adjacencies over the local EVPN/MAC bridge domain. This model also applies to direct physical connections with transport elements. Server Flavors: HS **Notes on SR-IOV** SR-IOV, at the compute server, routes Guest traffic directly with a partitioned NIC card, bypassing the hypervisor and vSwitch software, which provides higher bps/pps throughput for the Guest server. OpenStack and MANO manage SR-IOV configurations for Tenant server interfaces. - Server, Linux, and NIC card hardware standards include SR-IOV and VF requirements - High Performance profile for SR-IOV (hs series) with specific NIC/Leaf port configurations - OpenStack supports SR-IOV provisioning - Implement Security Policy, Tap/Mirror, QoS, etc. functions in the NIC, Leaf, and other places Because SR-IOV involves Guest VLANs between the compute server and the ToR/Leafs, Guest automation and server placement necessarily involves the Leaf switches (e.g., access VLAN outer tag mapping with VXLAN EVPN). - Local VXLAN tunneling over IP-switched fabric implemented between VTEPs on Leaf switches. - Leaf configuration controlled by SDN-Fabric/Global Controller. - Underlay uses VXLAN-enabled switches for EVPN support SR-IOV-based networking for Tenant Use Cases is required where vSwitch-based networking throughput is inadequate. ##### 4.2.2.9.5 Example Host Configurations _Host configurations for B1, B4 Profile Extensions_Figure 4-1: Basic Profile Host Configuration (example and simplified).
Let us refer to the data traffic networking configuration of Figure 4-1 to be part of the hp-B1-a and hp-B4-a host profiles and this requires the configurations as Table 4-3. | | Configured in | Host profile: hp-B1-a | Host profile: hp-B4-a | |----|----|----|----| | CPU Allocation Ratio | Hypervisor | 1:1 | 4:1 | | CPU Pinning | BIOS | Disable | Disable | | SMT | BIOS | Enable | Enable | | NUMA | BIOS | Disable | Disable | | Huge pages | BIOS | No | No | | Profile Extensions | | B1 | B4 |Table 4-3: Configuration of Basic Flavor Capabilities
Figure 4-2 shows the networking configuration where the storage and OAM share networking but are independent of the PXE network.Figure 4-2: Basic Profile Host Configuration with shared Storage and OAM networking (example and simplified).
Let us refer to the above networking set up to be part of the hp-B1-b and hp-B4-b host profiles but the basic configurations as specified in Table 4-3. In our example, the Profile Extensions B1 and B4, are each mapped to two different host profiles hp-B1-a and hp-B1-b, and hp-B4-a and hp-B4-b respectively. Different network configurations, reservation of CPU cores, Lag values, etc. result in different host profiles. To ensure Tenant CPU isolation from the host services (Operating System (OS), hypervisor and OpenStack agents), the following needs to be configured: | GRUB bootloader Parameter |Description |Values | |----|----|----| | isolcpus (Applicable only on Compute Servers) | A set of cores isolated from the host processes. Contains vCPUs reserved for Tenants | isolcpus=1-19, 21-39, 41-59, 61-79 | _Host configuration for HV Profile Extensions_ The above examples of host networking configurations for the B1 and B4 Profile Extensions are also suitable for the HV Profile Extensions; however, the hypervisor and BIOS settings will be different (see table below) and hence there will be a need for different host profiles. Table 4-4 gives examples of three different host profiles; one each for HV, HD and HS Profile Extensions. | | Configured in | Host profile: hp-hv-a | Host profile: hp-hd-a | Host profile: hp-hs-a | |----|----|----|----|----| | Profile Extensions | | HV | HD | HS | | CPU Allocation Ratio | Hypervisor | 1:1 | 1:1 | 1:1 | | NUMA | BIOS, Operating System, Hypervisor and OpenStack Nova Scheduler | Enable | Enable | Enable | | CPU Pinning (requires NUMA) | OpenStack Nova Scheduler | Enable | Enable | Enable | | SMT | BIOS | Enable | Enable | Enable | | Huge pages | BIOS | Yes | Yes | Yes |Table 4-4: Configuration of High Performance Flavor Capabilities
_Host Networking configuration for HD Profile Extensions_ An example of the data traffic configuration for the HD (OVS-DPDK) Profile Extensions is shown in Figure 4-3.Figure 4-3: High Performance Profile Host Configuration with DPDK acceleration (example and simplified).
To ensure Tenant and DPDK CPU isolation from the host services (Operating System (OS), hypervisor and OpenStack agents), the following needs to be configured: | GRUB bootloader Parameter |Description |Values | |----|----|----| | isolcpus (Applicable only on Compute Servers) | A set of cores isolated from the host processes. Contains vCPUs reserved for Tenants and DPDK | isolcpus=3-19, 23-39, 43-59, 63-79 | _Host Networking configuration for HS Profile Extensions_ An example of the data traffic configuration for the HS (SR-IOV) Profile Extensions is shown in Figure 4-4.Figure 4-4: High Performance Profile Host Configuration with SR-IOV (example and simplified).
To ensure Tenant CPU isolation from the host services (Operating System (OS), hypervisor and OpenStack agents), the following needs to be configured: | GRUB bootloader Parameter |Description |Values | |----|----|----| | isolcpus (Applicable only on Compute Servers) | A set of cores isolated from the host processes. Contains vCPUs reserved for Tenants | isolcpus=1-19, 21-39, 41-59, 61-79 | ##### 4.2.2.9.6 Using Hosts of a Host Profile type As we have seen Profile Extensions are supported by configuring hosts in accordance with the Profile Extensions specifications. For example, an instance of flavor type B1 can be hosted on a compute node that is configured as an hp-B1-a or hp-B1-b host profile. All compute nodes configured with hp-B1-a or hp-B1-b host profile are made part of a host aggregate, say, ha-B1 and, thus, during server instantiation of B1 flavor hosts from the ha-B1 host aggregate will be selected. ### 4.2.3. Network Fabric Networking Fabric consists of: - Physical switches, routers… - Switch OS - Minimum number of switches - Dimensioning for East/West and North/South - Spine / Leaf topology – east – west - Global Network parameters - OpenStack control plane VLAN / VXLAN layout - Provider VLANs #### 4.2.3.1 Physical Network Topology #### 4.2.3.2 High Level Logical Network LayoutFigure 4-5: Indicative OpenStack Network Layout.
| Network | Description | Characteristics | |----------|---------|--------------| | Provisioning & Management | Initial OS bootstrapping of the servers via PXE, deployment of software and thereafter for access from within the control plane. | Security Domain: Management
Figure 4-6: Ceph Storage System.
Table 4-2: Neutron Services Placement
##### 4.3.1.5.1 Issues with the standard networking (centralised routing) approach The network node performs both routing and NAT functions and represents both a scaling bottleneck and a single point of failure. Consider two servers on different compute nodes and using different project networks (a.k.a. tenant networks) where the both of the project networks are connected by a project router. For communication between the two servers (instances with a fixed or floating IP address), the network node routes East-West network traffic among project networks using the same project router. Even though the instances are connected by a router, all routed traffic must flow through the network node, and this becomes a bottleneck for the whole network. While the separation of the routing function from the controller node to the network node provides a degree of scaling it is not a truly scalable solution. We can either add additional cores/compute-power or network node to the network node cluster, but, eventually, it runs out of processing power especially with high throughput requirement. Therefore, for scaled deployments, there are multiple options including use of Dynamic Virtual Routing (DVR) and Software Defined Networking (SDN). ##### 4.3.1.5.2 Distributed Virtual Routing (DVR) With DVR, each compute node also hosts the L3-agent (providing the distributed router capability) and this then allows direct instance to instance (East-West) communications. The OpenStack “[High Availability Using Distributed Virtual Routing (DVR)](https://docs.openstack.org/liberty/networking-guide/scenario-dvr-ovs.html)” provides an in-depth view into how DVR works and the traffic flow between the various nodes and interfaces for three different use cases. Please note that DVR was introduced in the OpenStack Juno release and, thus, its detailed analysis in the Liberty release documentation is not out of character for OpenStack documentation. DVR addresses both scalability and high availability for some L3 functions but is not fully fault tolerant. For example, North/South SNAT traffic is vulnerable to single node (network node) failures. [DVR with VRRP](https://docs.openstack.org/neutron/wallaby/admin/config-dvr-ha-snat.html) addresses this vulnerability. ##### 4.3.1.5.3 Software Defined Networking (SDN) For the most reliable solution that addresses all the above issues and Telco workload requirements requires SDN to offload Neutron calls. SDN provides a truly scalable and preferred solution to suport dynamic, very large-scale, high-density, telco cloud environments. OpenStack Neutron, with its plugin architecture, provides the ability to integrate SDN controllers ( [3.2.5. Virtual Networking – 3rd party SDN solution](./chapter03.md#325-virtual-networking--3rd-party-sdn-solution)). With SDN incorporated in OpenStack, changes to the network is triggered by workloads (and users), translated into Neutron APIs and then handled through neutron plugins by the corresponding SDN agents. #### 4.3.1.6 Nova [Nova](https://docs.openstack.org/nova/wallaby/) is the compute management service, depends on all above components and is deployed after their deployment. Nova has services running on the control nodes and the compute nodes: - nova-metadata-api - nova-compute api - nova-consoleauth - nova-scheduler - nova-conductor - nova-novncproxy - nova-compute-agent which runs on Compute node Please note that the Placement-API must have been installed and configured prior to nova compute starts. #### 4.3.1.7 Ironic [Ironic](https://docs.openstack.org/ironic/wallaby/) is the bare metal provisioning service. Ironic depends on all above components and is deployed after them. Ironic has services running on the control nodes and the compute nodes: - Ironic API - ironic-conductor which executes operation on bare metal nodes Note: This is an optional service. The [Ironic APIs](https://docs.openstack.org/api-ref/baremetal/) are still under development. #### 4.3.1.8 Heat [Heat](https://docs.openstack.org/heat/wallaby/) is the orchestration service using templates to provision cloud resources, Heat integrates with all OpenStack services. Heat has services running on the control nodes and no services running on the compute nodes: - heat-api - heat-cfn-api - heat-engine #### 4.3.1.9 Horizon [Horizon](https://docs.openstack.org/horizon/wallaby/) is the Web User Interface to all OpenStack services. Horizon has services running on the control nodes and no services running on the compute nodes. #### 4.3.1.10 Placement The OpenStack [Placement service](https://docs.openstack.org/placement/wallaby/index.html) enables tracking (or accounting) and scheduling of resources. It provides a RESTful API and a data model for the managing of resource provider inventories and usage for different classes of resources. In addition to standard resource classes, such as vCPU, MEMORY_MB and DISK_GB, the Placement service supports custom resource classes (prefixed with “CUSTOM_”) provided by some external resource pools such as a shared storage pool provided by, say, Ceph. The placement service is primarily utilised by nova-compute and nova-scheduler. Other OpenStack services such as Neutron or Cyborg can also utilise placement and do so by creating [Provider Trees]( https://docs.openstack.org/placement/latest/user/provider-tree.html). The following data objects are utilised in the [placement service]( https://docs.openstack.org/placement/latest/user/index.html): - Resource Providers provide consumable inventory of one or more classes of resources (CPU, memory or disk). A resource provider can be a compute host, for example. - Resource Classes specify the type of resources (vCPU, MEMORY_MB and DISK_GB or CUSTOM_\*) - Inventory: Each resource provider maintains the total and reserved quantity of one or more classes of resources. For example, RP_1 has available inventory of 16 vCPU, 16384 MEMORY_MB and 1024 DISK_GB. - Traits are qualitative characteristics of the resources from a resource provider. For example, the trait for RPA_1 “is_SSD” to indicate that the DISK_GB provided by RP_1 are solid state drives. - Allocations represent resources that have been assigned/used by some consumer of that resource. - Allocation candidates is the collection of resource providers that can satisfy an allocation request. The Placement API is stateless and, thus, resiliency, availability and scaling, it is possible to deploy as many servers as needed. On start, the nova-compute service will attempt to make a connection to the Placement API and keep attempting to connect to the Placement API, logging and warning periodically until successful. Thus, the Placement API must be installed and enabled prior to Nova compute. Placement has services running on the control node: - nova-placement-api #### 4.3.1.11 Barbican [Barbican](https://docs.openstack.org/barbican/wallaby/) is the OpenStack Key Manager service. It is an optional service hosted on controller nodes. It provides secure storage, provisioning, and management of secrets as passwords, encryption keys and X.509 Certificates. Barbican API is used to centrally manage secrets used by OpenStack services, e.g., symmetric encryption keys used for Block storage encryption or Object Storage encryption or asymmetric keys and certificates used for Glance image signing and verification. Barbican usage provides a means to fulfill security requirements such as sec.sys.012 “The Platform **must** protect all secrets by using strong encryption techniques and storing the protected secrets externally from the component” and sec.ci.001 “The Platform **must** support Confidentiality and Integrity of data at rest and in transit.”. #### 4.3.1.12 Cyborg [Cyborg](https://docs.openstack.org/cyborg/wallaby/) is the OpenStack project for the general purpose management framework for accelerators (including GPUs, FPGAs, ASIC-based devices, etc.), and their lifecycle management. Cyborg will support only a subset of the [Nova operations](https://docs.openstack.org/api-guide/compute/server_concepts.html); the set of Nova operations supported in Cyborg depends upon the merge of a set of Nova patches in Cyborg. In Wallaby, not all the required Nova patches have been merged. The list of Cyborg operations with Nova dependencies supported in Wallaby is listed [here](https://docs.openstack.org/cyborg/wallaby/reference/support-matrix.html); the Nova operations supported in Cyborg at any given time is also [available](https://docs.openstack.org/cyborg/latest/reference/support-matrix.html). Cyborg supports: * Acceleration Resource Discovery * Accelerator Life Cycle Management Accelerators can be of type: * Software: dpdk/spdk, pmem, … * Hardware (device types): FPGA, GPU, ARM SoC, NVMe SSD, CCIX based Caches, … The [Cyborg architecture](https://docs.openstack.org/cyborg/latest/user/architecture.html) consists of the cyborg-api, cyborg-conductor, cyborg-db, cyborg-agent, and generic device type drivers. cyborg-api, cyborg-conductor and cyborg-db are hosted on control nodes. cyborg-agent, which runs on compute nodes, interacts with generic device type drivers on those nodes. These generic device type drivers are an abstraction of the vendor specific drivers; there is a generic device type driver for each device type (see above for list of some of the device types). The current list of the supported vendor drivers is listed under "[Driver Support](https://docs.openstack.org/cyborg/latest/reference/support-matrix.html)". ### 4.3.2. Containerised OpenStack Services Containers are lightweight compared to Virtual Machines and leads to efficient resource utilisation. Kubernetes auto manages scaling, recovery from failures, etc. Thus, it is recommended that the OpenStack services be containerised for resiliency and resource efficiency. In Chapter 3, [Figure 3.2](../figures/RA1-Ch03-OpenStack-Services-Topology.png) shows a high level Virtualised OpenStack services topology. The containerised OpenStack services topology version is shown in Figure 4-7.
Figure 4-7: Containerised OpenStack Services Topology.
Table 4-5: Distribution of OpenStack services on different nodes depending upon Control Plane Scenario
### 4.5.1 Edge Cloud Topology The Reference Model Chapter 8.3 "[Telco Edge Cloud](../../../ref_model/chapters/chapter08.md#8.3)", presents the deployment environment characteristics, infrastructure characteristics and new values for the Infrastructure Profiles at the Edge. The [Edge computing whitepaper](https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing/) includes information such as the services that run on various nodes. The information from the whitepaper coupled with that from the [OpenStack Reference Architecture](https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html#services-placement-summary) for 100, 300 and 500 nodes will help in deciding which OpenStack and other services (such as database, messaging) run on which nodes in what Cloud Centre and the number of copies that should be deployed. These references also present the pros and cons of DCP and CCP and designs to address some of the challenges of each of the models. Table 8-4 in the Reference Model Chapter 8.3.4 "[Telco Edge Cloud: Platform Services Deployment](../../../ref_model/chapters/chapter08.md#8.3.4)" lists the Platform Services that may be placed in the different node types (control, compute and storage). Depending upon the capacity and resources available only the compute nodes may exist at the Edge thereby impacting operations. Table 8-3 in the Reference Model Chapter 8.3.3 "[Telco Edge Cloud Infrastructure Profiles](../../../ref_model/chapters/chapter08.md#8.3.3)" lists a number of Infrastructure Profile characteristics and the changes that may need to be made for certain Edge clouds depending upon their resource capabilities. It should be noted that none of these changes affect the definition of OpenStack flavors. #### 4.5.1.1 Edge Cloud Deployment Deployment at the Edge requires support for large scale deployment. A number of open-source tools are available for the purpose including: - [Airship](https://docs.airshipit.org/): declaratively configure, deploy and maintain an integrated virtualisation and containerisation platform - [Starling-X](https://www.starlingx.io/): cloud infrastructure software stack for the edge - [Triple-O](https://wiki.openstack.org/wiki/TripleO): for installing, upgrading and operating OpenStack clouds The Reference Implementation (RI-1) is responsible to choose the tools for the implementation and shall specify implementation and usage details of the chosen tools.