# 4 Infrastructure Capabilities, Measurements and Catalogue ## Table of Contents * [4.1 Capabilities and Performance Measurements](#41-Capabilities-and-Performance-Measurements) * [4.1.1 Exposed vs Internal](#411-Exposed-vs-Internal) * [4.1.2 Exposed Infrastructure Capabilities](#412-Exposed-Infrastructure-Capabilities) * [4.1.3 Exposed Infrastructure Measurements](#413-Exposed-Infrastructure-Measurements) * [4.1.4 Internal Infrastructure Capabilities](#414-Internal-Infrastructure-Capabilities) * [4.1.5 Cloud infrastructure management Capabilities](#415-Cloud-infrastructure-management-Capabilities) * [4.1.6 Cloud infrastructure management Measurements](#416-Cloud-infrastructure-management-Measurements) * [4.1.7 Acceleration/Offload API Requirements](#417-Acceleration/Offload-API-Requirements) * [4.2 Profiles and Workload Flavours](#42-Profiles-and-Workload-Flavours) * [4.2.1 Profiles](#421-Profiles) * [4.2.2 Profile Specifications & Capabilities Mapping](#422-Profile-Specifications-&-Capabilities-Mapping) * [4.2.3 Profile Extensions](#423-Profile-Extensions) * [4.2.4 Workload Flavours and Other Capabilities Specifications](#424-Workload-Flavours-and-Other-Capabilities-Specifications) * [4.2.5 Virtual Network Interface Specifications](#425-Virtual-Network-Interface-Specifications) * [4.2.6 Storage Extensions](#426-Storage-Extensions) ## 4.1 Capabilities and Performance Measurements This section describes the Capabilities provided by the Cloud Infrastructure, and the Performance Measurements (PMs) generated by the Cloud Infrastructure (i.e., without the use of external instrumentation). The Capability and PM identifiers conform to the following schema: **a.b.c** (E.g., "e.pm.001") a = Scope <(e)xternal | (i)nternal | (t)hird_party_instrumentation> b = Type <(cap) capability | (man) management | (pm) performance | (man-pm)> c = Serial Number ### 4.1.1 Exposed vs Internal The following definitions specify the context of the Cloud Infrastructure Resources, Capabilities and Performance Measurements (PMs). **Exposed:** Refers to any object (e.g., resource discovery/configuration/consumption, platform telemetry, interface, etc.) that exists in or pertains to, the domain of the Cloud Infrastructure and is made visible (aka “Exposed”) to a workload. When an object is exposed to a given workload, the scope of visibility within a given workload is at the discretion of the workload’s designer. From an infrastructure perspective, the Infra-resident object is simply being exposed to one or more virtual environments (i.e., workloads). It is the responsibility of the kernel or supervisor/executive within the resource instance (VM or container) to control how, when and where the object is further exposed within the resource instance, with regard to permissions, security, etc. An object(s) is by definition visible within its domain of origin. **Internal:** Effectively the opposite of Exposed; objects are exclusively available for use within the Cloud Infrastructure.
Figure 4-1: Exposed vs. Internal Scope
As illustrated in the figure above, objects designated as "Internal" are only visible within the area inside the blue oval (the Cloud Infrastructure), and only when the entity accessing the object has the appropriate permissions. Whereas objects designated as "Exposed" are potentially visible from both the area within the green oval (the Workloads), as well as from within the Cloud Infrastructure, again provided the entity accessing the object has appropriate permissions. Note: The figure above indicates the areas from where the objects are _visible_. It is not intended to indicate where the objects are _instantiated_. For example, the virtual resources are instantiated within the Cloud Infrastructure (the blue area), but are Exposed, and therefore are _visible_ to the Workloads, within the green area. ### 4.1.2 Exposed Infrastructure Capabilities This section describes a set of exposed Cloud Infrastructure capabilities and performance measurements. These capabilities and PMs are well known to workloads as they provide capabilities which workloads rely on. > _**Note**: It is expected that Cloud Infrastructure capabilities and measurements will expand over time as more capabilities are added and technology enhances and matures._ #### 4.1.2.1 Exposed Resource Capabilities **Table 4-1** below shows resource capabilities of the Cloud Infrastructure available to workloads. | Ref | Cloud Infrastructure Capability | Unit | Definition/Notes | |-----------|----------------------------------------|--------|-------------------------------------------------------------------------------| | e.cap.001 | # vCPU | number | Max number of vCPUs that can be assigned to a single VM or Pod 1 | | e.cap.002 | RAM Size | MB | Max memory in MB that can be assigned to a single VM or Pod by the Cloud Infrastructure 2 | | e.cap.003 | Total per-instance (ephemeral) storage | GB | Max storage in GB that can be assigned to a single VM or Pod by the Cloud Infrastructure | | e.cap.004 | # Connection points | number | Max number of connection points that can be assigned to a single VM or Pod by the Cloud Infrastructure | | e.cap.005 | Total external (persistent) storage | GB | Max storage in GB that can be attached/mounted to VM or Pod by the Cloud Infrastructure |Table 4-1: Exposed Resource Capabilities of Cloud Infrastructure
1. In a Kubernetes based environment this means the CPU limit of a pod. 2. In a Kubernetes based environment this means the memory limit of a pod. #### 4.1.2.2 Exposed Performance Optimisation Capabilities **Table 4-2** lists performance optimisation capabilities exposed to workloads by the Cloud Infrastructure. | Ref | Cloud Infrastructure Capability | Unit | Definition/Notes | |-----------|-------------------------------------------|--------|-------------------------------------------------------------| | e.cap.006 | CPU pinning | Yes/No | Indicates if Cloud Infrastructure supports CPU pinning | | e.cap.007 | NUMA alignment | Yes/No | Indicates if Cloud Infrastructure supports NUMA alignment | | e.cap.008 | IPSec Acceleration | Yes/No | IPSec Acceleration | | e.cap.009 | Crypto Acceleration | Yes/No | Crypto Acceleration | | e.cap.010 | Transcoding Acceleration | Yes/No | Transcoding Acceleration | | e.cap.011 | Programmable Acceleration | Yes/No | Programmable Acceleration | | e.cap.012 | Enhanced Cache Management | Yes/No | If supported, L=Lean; E=Equal; X=eXpanded. L and X cache policies require CPU pinning to be active | | e.cap.013 | SR-IOV over PCI-PT | Yes/No | Traditional SR-IOV. These Capabilities generally require hardware-dependent drivers be injected into workloads | | e.cap.014 | GPU/NPU | Yes/No | Hardware coprocessor. These Capabilities generally require hardware-dependent drivers be injected into workloads | | e.cap.015 | SmartNIC | Yes/No | Network Acceleration | | e.cap.016 | FPGA/other Acceleration HW | Yes/No | These Capabilities generally require hardware-dependent drivers be injected into workloads |Table 4-2: Exposed Performance Optimisation Capabilities of Cloud Infrastructure
Enhanced Cache Management is a compute performance enhancer that applies a cache management policy to the socket hosting a given virtual compute instance, provided the associated physical CPU microarchitecture supports it. Cache management policy can be used to specify the static allocation of cache resources to cores within a socket. The "Equal" policy distributes the available cache resources equally across all of the physical cores in the socket. The "eXpanded" policy provides additional resources to the core pinned to a workload that has the "X" attribute applied. The "Lean" attribute can be applied to workloads which do not realise significant benefit from a marginal cache size increase and are hence willing to relinquish unneeded resources. In addition to static allocation, an advanced Reference Architecture implementation can implement dynamic cache management control policies, operating with tight (~ms) or standard (10s of seconds) control loop response times, thereby achieving higher overall performance for the socket. #### 4.1.2.3 Exposed Monitoring Capabilities Monitoring capabilities are used for the passive observation of workload-specific traffic traversing the Cloud Infrastructure. As with all capabilities, Monitoring may be unavailable or intentionally disabled for security reasons in a given Cloud Infrastructure deployment. If this functionality is enabled, it must be subject to strict security policies. Refer to the Reference Model Security chapter for additional details. **Table 4-3** shows possible monitoring capabilities available from the Cloud Infrastructure for workloads. | Ref | Cloud Infrastructure Capability | Unit | Definition/Notes | |-----------|---------------------------------|--------|---------------------------------------------| | e.cap.017 | Monitoring of L2-7 data | Yes/No | Ability to monitor L2-L7 data from workload |Table 4-3: Exposed Monitoring Capabilities of Cloud Infrastructure
### 4.1.3 Exposed Infrastructure Performance Measurements The intent of the following PMs is to be available for and well known to workloads. #### 4.1.3.1 Exposed Performance Measurements The following table of exposed Performance Measurements shows PMs per VM or Pod, vNIC or vCPU. Network test setups are aligned with ETSI GS NFV-TST 009 [14]. Specifically exposed PMs use a single workload (PVP) dataplane test setup in a single host. | Ref | Cloud Infrastructure Measurement | Unit | Definition/Notes | |----------|----------------------------------|-------|---------------------| | e.pm.xxx | Place Holder | Units | Concise description |Table 4-4: Exposed Performance Measurements of Cloud Infrastructure
### 4.1.4 Internal Infrastructure Capabilities This section covers a list of implicit Cloud Infrastructure capabilities and measurements. These capabilities and metrics are hidden from workloads (i.e., workloads may not know about them) but they will impact the overall performance and capabilities of a given Cloud Infrastructure solution. >_**Note**: It is expected that implicit Cloud Infrastructure capabilities and metrics will evolve with time as more capabilities are added as technology enhances and matures._ #### 4.1.4.1 Internal Resource Capabilities **Table 4-5** shows resource capabilities of Cloud Infrastructure. These include capabilities offered to workloads and resources consumed internally by Cloud Infrastructure. | Ref | Cloud Infrastructure Capability | Unit | Definition/Notes | |-----------|-------------------------------------------------------|------------------------|--------------------------------------------------------| | i.cap.014 | CPU cores consumed by the Cloud Infrastructure overhead on a worker (compute) node | % | The ratio of cores consumed by the Cloud Infrastructure components (including host OS) in a compute node to the total number of cores available expressed as a percentage | | i.cap.015 | Memory consumed by the Cloud Infrastructure overhead on a worker (compute) node | % | The ratio of memory consumed by the Cloud Infrastructure components (including host OS) in a worker (compute) node to the total available memory expressed as a percentage |Table 4-5: Internal Resource Capabilities of Cloud Infrastructure
#### 4.1.4.2 Internal SLA capabilities **Table 4-6** below shows SLA (Service Level Agreement) capabilities of Cloud Infrastructure. These include Cloud Infrastructure capabilities required by workloads as well as required internal to Cloud Infrastructure. Application of these capabilities to a given workload is determined by its Cloud Infrastructure Profile. | Ref | Cloud Infrastructure capability | Unit | Definition/Notes | |-----------|----------------------|--------|--------------------------------------------------------------------------------| | i.cap.016 | CPU allocation ratio | N:1 | Number of virtual cores per physical core; also known as CPU overbooking ratio | | i.cap.017 | Connection point QoS | Yes/No | QoS enablement of the connection point (vNIC or interface) |Table 4-6: Internal SLA capabilities to Cloud Infrastructure
#### 4.1.4.3 Internal Performance Optimisation Capabilities **Table 4-7** below shows possible performance optimisation capabilities that can be provided by the Cloud Infrastructure. These include capabilities exposed to workloads as well as internal capabilities to Cloud Infrastructure. These capabilities will be determined by the Cloud Infrastructure Profile used by the Cloud Infrastructure. | Ref | Cloud Infrastructure capability | Unit | Definition/Notes | |-----------|---------------------------------|--------|---------------------------------------| | i.cap.018 | Huge pages | Yes/No | Indicates if the Cloud Infrastructure supports huge pages |Table 4-7: Internal performance optimisation capabilities of Cloud Infrastructure
#### 4.1.4.4 Internal Performance Measurement Capabilities **Table 4-8** shows possible performance measurement capabilities for the Cloud Infrastructure. The availability of these capabilities will be determined by the Cloud Infrastructure Profile used by the workloads. | Ref | Cloud Infrastructure Measurement | Unit | Definition/Notes | |----------|--------------------------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | i.pm.001 | Host CPU usage | nanoseconds | Per Compute node. It maps to ETSI GS NFV-TST 008 V3.2.1 [5] clause 6, processor usage metric (Cloud Infrastructure internal). | | i.pm.002 | Virtual compute resource (vCPU) usage | nanoseconds | Per VM or Pod. It maps to ETSI GS NFV-IFA 027 v2.4.1 [6] Mean vCPU usage and Peak vCPU usage (Cloud Infrastructure external). | | i.pm.003 | Host CPU utilisation | % | Per Compute node. It maps to ETSI GS NFV-TST 008 V3.2.1 [5] clause 6, processor usage metric (Cloud Infrastructure internal). | | i.pm.004 | Virtual compute resource (vCPU) utilisation | % | Per VM or Pod. It maps to ETSI GS NFV-IFA 027 v2.4.1 [6] Mean vCPU usage and Peak vCPU usage (Cloud Infrastructure external). | | i.pm.005 | Measurement of external storage IOPS | Yes/No | | | i.pm.006 | Measurement of external storage throughput | Yes/No | | | i.pm.007 | Available external storage capacity | Yes/No | |Table 4-8: Internal Measurement Capabilities of Cloud Infrastructure
### 4.1.5 Cloud Infrastructure Management Capabilities The Cloud Infrastructure Manager (CIM) is responsible for controlling and managing the Cloud Infrastructure compute, storage, and network resources. Resources are dynamically allocated based on workload requirements. This section covers the list of capabilities offered by the CIM to workloads or service orchestrator. Table 4-9 shows capabilities related to resources allocation. | Ref | Cloud Infrastructure Management Capability | Unit | Definition/Notes | |-----------|--------------------------------------------|-----------------|------------------------------------------------------------------| | e.man.001 | Virtual Compute allocation | Yes/No | Capability to allocate virtual compute resources to a workload | | e.man.002 | Virtual Storage allocation | Yes/No | Capability to allocate virtual storage resources to a workload | | e.man.003 | Virtual Networking resources allocation | Yes/No | Capability to allocate virtual networking resources to a workload | | e.man.004 | Multi-tenant isolation | Yes/No | Capability to isolate resources between tenants | | e.man.005 | Images management | Yes/No | Capability to manage workload software images | | e.man.010 | Compute Availability Zones | list of strings | The names of each Compute Availability Zone that was defined to separate failure domains | | e.man.011 | Storage Availability Zones | list of strings | The names of each Storage Availability Zone that was defined to separate failure domains |Table 4-9: Cloud Infrastructure Management Resource Allocation Capabilities
### 4.1.6 Cloud Infrastructure Management Performance Measurements Table 4-10 shows performance measurement capabilities. | Ref | Cloud Infrastructure Management Capability | Unit | Definition/Notes | |-----------|--------------------------------------------|--------|---------------------------------------------------------------------------------------------| | e.man.006 | Virtual resources inventory per tenant | Yes/No | Capability to provide information related to allocated virtualised resources per tenant | | e.man.007 | Resources Monitoring | Yes/No | Capability to notify state changes of allocated resources | | e.man.008 | Virtual resources Performance | Yes/No | Capability to collect and expose performance information on virtualised resources allocated | | e.man.009 | Virtual resources Fault information | Yes/No | Capability to collect and notify fault information on virtualised resources |Table 4-10: Cloud Infrastructure Management Performance Measurement Capabilities
#### 4.1.6.1 Resources Management Measurements **Table 4-11** shows resource management measurements of CIM as aligned with ETSI GR NFV IFA-012 [15]. The intention of this table is to provide a list of measurements to be used in the Reference Architecture specifications, where the values allowed for these measurements in the context of a particular Reference Architecture will be defined. | Ref | Cloud Infrastructure Management Measurement | Unit | Definition/Notes | |--------------|---------------------------------------------------------------|--------|------------------| | e.man-pm.001 | Time to create Virtual Compute resources (VM/container) for a given workload | Max ms | | | e.man-pm.002 | Time to delete Virtual Compute resources (VM/container) of a given workload | Max ms | | | e.man-pm.003 | Time to start Virtual Compute resources (VM/container) of a given workload | Max ms | | | e.man-pm.004 | Time to stop Virtual Compute resources (VM/container) of a given workload | Max ms | | | e.man-pm.005 | Time to pause Virtual Compute resources (VM/container) of a given workload | Max ms | | | e.man-pm.006 | Time to create internal virtual network | Max ms | | | e.man-pm.007 | Time to delete internal virtual network | Max ms | | | e.man-pm.008 | Time to update internal virtual network | Max ms | | | e.man-pm.009 | Time to create external virtual network | Max ms | | | e.man-pm.010 | Time to delete external virtual network | Max ms | | | e.man-pm.011 | Time to update external virtual network | Max ms | | | e.man-pm.012 | Time to create external storage ready for use by workload | Max ms | |Table 4-11: Cloud Infrastructure Resource Management Measurements
### 4.1.7 Acceleration/Offload API Requirements HW Accelerators and Offload functions with abstracted interfaces are preferred and can functionally be interchanged, but their characteristics might vary. It is also likely that the CNFs/VNFs and the Cloud Infrastructure will have certification requirements for the implementations. A SW implementation of these functions is also often needed to have the same abstracted interfaces for the deployment situations when there are no more HW Accelerator or Offload resources available. For Accelerators and Offload functions with externally exposed differences in their capabilities or management functionality these differences must be clear through the management API either explicit for the differing functions or implicit through the use of a unique APIs. Regardless of the exposed or internal capabilities and characteristics, the operators generally require a choice of implementations for Accelerators and Offload function realisation, and, thus, the need for ease of portability between implementations and vendors. The following table of requirements are derived from the VNF/CNF applications, Cloud Infrastructure and Telco Operators needs to have multiple realisations of HW Acceleration and Offload functions that can also be implemented through SW when no special hardware is available. These requirements should be adopted in Reference Architectures to ensure that the different implementations on the market are as aligned as possible in their interfaces and that HW Acceleration and Offload functions get an efficient ecosystem of accelerators that compete on their technical merits and not through obscure or proprietary interfaces. **Table 4-12** shows Acceleration/Offload API Capabilities. | Ref | Acceleration/Offload API Capability | Unit | Definition/Notes | |-----------|--------------------------------------------|--------|---------------------------------------------------------------------------------------------| | e.api.001 | VNF/CNF usage of Accelerator standard i/f | Yes/No | VNF/CNF shall use abstracted standardised interfaces to the Acceleration/Offload functions. This would enable use of HW and SW implementations of the accelerated/offloaded functions from multiple vendors in the Cloud Infrastructure. | | e.api.002 | Virtualisation Infrastructure SW usage of Accelerator standard i/f | Yes/No | Virtualisation Infrastructure SW shall use abstracted standardised interfaces to the HW-Acceleration/Offload function enabling multiple HW and SW implementations in the HW Infrastructure Layer of the accelerated functions from multiple vendors. | | e.api.003 | Accelerators offering standard i/f to HW Infra Layer | Yes/No | Acceleration/Offload functions shall offer abstracted standardised interfaces for the Virtualisation Infrastructure and VNF/CNF applications. | | e.api.004 | Accelerators offering virtualised functions | Yes/No | Acceleration/Offload functions for VNFs/CNFs should be virtualised to allow multiple VNFs/CNFs to use the same Acceleration/Offload instance. | | e.api.005 | VNF/CNF Accelerator management functions access rights | Yes/No | VNF/CNF management functions shall be able to request Acceleration/Offload invocation without requiring elevated access rights. | | e.api.006 | Accelerators offering standard i/f to VNF/CNF management | Yes/No | VNF/CNF management functions should be able to request Acceleration/Offload invocation through abstracted standardised Management interfaces. | | e.api.007 | VNFs/CNFs and Virtualisation Infrastructure Accelerator portability | Yes/No | VNFs/CNFs and Virtualisation Infrastructure SW should be designed to handle multiple types of Accelerator or Offload Function realisations even when their differences are exposed to the infrastructure or applications layers. | | e.api.008 | VNFs/CNFs and Virtualisation Infrastructure Accelerator flexibility | Yes/No | VNFs/CNFs and Virtualisation Infrastructure SW shall be able to use any assigned instance and type of Accelerator or Offload Function that they are certified for. |Table 4-12: Acceleration/Offload API Capabilities
## 4.2 Profiles and Workload Flavours Section 4.1 enumerates the different capabilities exposed by the infrastructure resources. Not every workload is sensitive to all listed capabilities of the cloud infrastructure. In Chapter 2, the analysis of the use cases led to the definition of two [profiles](./chapter02.md#241-profiles-top-level-partitions) and the need for specialisation through [profile extensions](./chapter02.md#242-profile-extensions-specialisations). Profiles and Profile Extensions are used to configure the cloud infrastructure nodes. They are also used by workloads to specify the infrastructure capabilities needed by them to run on. Workloads would specify the [flavours and additional capabilities](#424-workload-flavours-and-other-capabilities-specifications) information. In this section we will specify the capabilities and features associated with each of the defined profiles and extensions. Each Profile (for example, Figure 4-2), and each Extension associated with that profile, specifies a predefined standard set of infrastructure capabilities that workload vendors can use to build their workloads for deployment on conformant cloud infrastructure. A workload can use several profiles and associated Extensions to build its overall functionality as discussed below.Figure 4-2:Cloud infrastructure Profiles.
The two [profiles]( ./chapter02.md#241-node-profiles-top-level-partitions) are: Basic (B): for Workloads that can tolerate resource over-subscription and variable latency. High Performance (H): for Workloads that require predictable computing performance, high network throughput and low network latency. The availability of these two (2) profiles will facilitate and accelerate workload deployment. The intent of the above profiles is to match the cloud infrastructure to the workloads most common needs, and allow for a more comprehensive configuration using profile-extensions when needed. These profiles are offered with [extensions](#423-profile-extensions), that specify capability deviations, and allow for the specification of even more capabilities. The Cloud Infrastructure will have nodes configured as with options, such as virtual interface options, storage extensions, and acceleration extensions. The justification for defining these two profiles and a set of extensible profile-extensions was provided in Section [2.4 Profiles, Profile Extensions & Flavours](./chapter02.md#24-profiles--flavours) and includes: * Workloads can be deployed by requesting compute hosts configured as per a specific profile (Basic or High Performance) * Profile extensions allow a more granular compute host configuration for the workload (e.g., GPU, high, speed network, Edge deployment) * Cloud infrastructure "scattering" is minimised * Workload development and testing optimisation by using pre-defined and commonly supported (telco operators) profiles and extensions * Better usage of Cloud Objects (Memory; Processor; Network; Storage) Workload flavours specify the resource sizing information including network and storage (size, throughput, IOPS). Figure 4.3 shows three resources (VM or Pod) on nodes configured as per the specified profile ('B' and 'H'), and the resource sizes.Figure 4-3:Workloads built against Cloud Infrastructure Profiles and Workload Flavours.
A node configuration can be specified using the syntax: > \Table 4-12: Workload Flavour Geometry Specification.
The flavours syntax consists of specifying using theTable 4-14: Virtual Network Interface Specification Examples
### 4.2.6 Storage Extensions Persistent storage is associated with workloads via Storage Extensions. The storage qualities specified by the Storage Extension pertain to "Platform Native - Hypervisor Attached" and "Platform Native - Container Persistent" storage types (as defined in section "3.6.3 Storage for Tenant Consumption"). The size of an extension can be specified explicitly in increments of 100GB (Table 4-15), ranging from a minimum of 100GB to a maximum of 16TB. Extensions are configured with the required performance category, as per Table 4-15. Multiple persistent Storage Extensions can be attached to virtual compute instances. >_*Note:*_ This specification uses GB and GiB to refer to a Gibibyte (230 bytes), except where explicitly stated otherwise. | .conf | Read IO/s | Write IO/s | Read Throughput (MB/s) | Write Throughput (MB/s) | Max Ext Size | |---------|------------|------------|------------------------|-------------------------|--------------| | .bronze | Up to 3K | Up to 1.5K | Up to 180 | Up to 120 | 16TB | | .silver | Up to 60K | Up to 30K | Up to 1200 | Up to 400 | 1TB | | .gold | Up to 680K | Up to 360K | Up to 2650 | Up to 1400 | 1TB |Table 4-15: Storage Extensions
>_*Note:*_ Performance is based on a block size of 256KB or larger.