4. Component Level Architecture ¶
Table of Contents ¶
4.1 Introduction ¶
This chapter describes in detail the Kubernetes Reference Architecture in terms of the functional capabilities and how they relate to the Reference Model requirements, i.e. how the infrastructure profiles are determined, documented and delivered.
The specifications defined in this chapter will be detailed with unique
identifiers, which will follow the pattern:
ra2.<section>.<index>
, e.g.
ra2.ch.001
for the first requirement in the Kubernetes Node section. These
specifications will then be used as requirements input for the Kubernetes
Reference Implementation and any Vendor or Community Implementations.
Figure 4-1 below shows the architectural components that are described in the subsequent sections of this chapter.
Figure 4-1: Kubernetes Reference Architecture
4.2 Kubernetes Node ¶
This section describes the configuration that will be applied to the physical or virtual machine and an installed Operating System. In order for a Kubernetes Node to be conformant with the Reference Architecture it must be implemented as per the following specifications:
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.ch.001
|
Huge Pages |
When hosting workloads matching the Network Intensive profile, it must be possible to enable Huge Pages (2048KiB and 1048576KiB) within the Kubernetes Node OS, exposing schedulable resources
hugepages-2Mi
and
hugepages-1Gi
.
|
infra.com.cfg.004 |
ra2.ch.002
|
SR-IOV capable NICs | When hosting workloads matching the Network Intensive profile, the physical machines on which the Kubernetes Nodes run must be equipped with NICs that are SR-IOV capable. | e.cap.013 |
ra2.ch.003
|
SR-IOV Virtual Functions | When hosting workloads matching the Network Intensive profile, SR-IOV virtual functions (VFs) must be configured within the Kubernetes Node OS, as the SR-IOV Device Plugin does not manage the creation of these VFs. | e.cap.013 |
ra2.ch.004
|
CPU Simultaneous Multi-Threading (SMT) | SMT must be enabled in the BIOS on the physical machine on which the Kubernetes Node runs. | infra.hw.cpu.cfg.004 |
ra2.ch.005
|
CPU Allocation Ratio - VMs | For Kubernetes nodes running as Virtual Machines, ensure the CPU allocation ratio between vCPU and physical CPU core is 1:1. | infra.com.cfg.001 |
ra2.ch.006
|
CPU Allocation Ratio - Pods |
To ensure the CPU allocation ratio between vCPU and physical CPU core is 1:1, the sum of CPU requests and limits by containers in Pod specifications must remain less than the allocatable quantity of CPU resources (i.e.
requests.cpu < allocatable.cpu
and
limits.cpu < allocatable.cpu
).
|
infra.com.cfg.001 |
ra2.ch.007
|
IPv6DualStack | To support IPv4/IPv6 dual stack networking, the Kubernetes Node OS must support and be allocated routable IPv4 and IPv6 addresses. | req.inf.ntw.04 |
ra2.ch.008
|
Physical CPU Quantity | The physical machines on which the Kubernetes Nodes run must be equipped with at least 2 physical sockets, each of at least 20 CPU cores. |
infra.hw.cpu.cfg.001
infra.hw.cpu.cfg.002 |
ra2.ch.009
|
Physical Storage | The physical machines on which the Kubernetes Nodes run should be equipped with Sold State Drives (SSDs). | infra.hw.stg.ssd.cfg.002 |
ra2.ch.010
|
Local Filesystem Storage Quantity | The Kubernetes Nodes must be equipped with local filesystem capacity of at least 320GB for unpacking and executing containers. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc. | e.cap.003 |
ra2.ch.011
|
Virtual Node CPU Quantity | If using VMs, the Kubernetes Nodes must be equipped with at least 16 vCPUs. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc. | e.cap.001 |
ra2.ch.012
|
Kubernetes Node RAM Quantity | The Kubernetes Nodes must be equipped with at least 32GB of RAM. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc. | e.cap.002 |
ra2.ch.013
|
Physical NIC Quantity | The physical machines on which the Kubernetes Nodes run must be equipped with at least four (4) Network Interface Card (NIC) ports. | infra.hw.nic.cfg.001 |
ra2.ch.014
|
Physical NIC Speed - Basic Profile | The NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Basic Profile must be at least 10Gbps. | infra.hw.nic.cfg.002 |
ra2.ch.015
|
Physical NIC Speed - Network Intensive Profile | The NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Network Intensive profile must be at least 25Gbps. | infra.hw.nic.cfg.002 |
ra2.ch.015
|
Physical PCIe slots | The physical machines on which the Kubernetes Nodes run must be equipped with at least eight (8) Gen3.0 PCIe slots, each with at least eight (8) lanes. | |
ra2.ch.016
|
Immutable infrastructure | Whether physical or virtual machines are used, the Kubernetes Node is not changed after it is made ready for use. New changes to the Kubernetes Node are rolled out as new instances. This covers any changes from BIOS through Operating System to running processes and all associated configurations. |
req.gen.cnt.02
|
ra2.ch.017
|
|||
ra2.ch.018
|
|||
ra2.ch.019
|
|||
ra2.ch.020
|
|||
ra2.ch.021
|
Table 4-1: Host OS Specifications
4.3 Kubernetes ¶
- The version of version range of Kubernetes and the mandatory components needed for Kubernetes (e.g.: etcd, cadvisor)
- Which optional features are used and which optional API-s are available
- Which alpha or beta features are used
In order for the Kubernetes components to be conformant with the Reference Architecture they must be implemented as per the following specifications:
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.k8s.001
|
Kubernetes Conformance | The Kubernetes distribution, product, or installer used in the implementation must be listed in the Kubernetes Distributions and Platforms document and marked (X) as conformant for the Kubernetes version that is being used. | req.gen.cnt.03 |
ra2.k8s.002
|
Highly available etcd | An implementation must consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes). | req.gen.rsl.02 req.gen.avl.01 |
ra2.k8s.003
|
Highly available control plane | An implementation must consist of at least one master node per availability zone or fault domain to ensure the high availability and resilience of the Kubernetes control plane services | req.gen.rsl.02 req.gen.avl.01 |
ra2.k8s.004
|
Highly available worker nodes | An implementation must consist of at least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes | req.gen.rsl.01 req.gen.avl.01 req.kcm.gen.02 req.inf.com.01 |
ra2.k8s.005
|
Kubernetes API Version |
In alignment with the
Kubernetes version support policy
, an implementation must use one of three latest minor versions (
n-2
). e.g. if the latest version is 1.17 then the RI must use either 1.17, 1.16 or 1.15.
|
TBC |
ra2.k8s.006
|
NUMA Support |
When hosting workloads matching the Network Intensive profile, the
TopologyManager
and
CPUManager
feature gates must be enabled and configured on the kubelet (note, TopologyManager is enabled by default in Kubernetes v1.18 and later, with CPUManager enabled by default in Kubernetes v1.10 and later).
--feature-gates="...,TopologyManager=true,CPUManager=true" --topology-manager-policy=single-numa-node --cpu-manager-policy=static
|
e.cap.007 infra.com.cfg.002 infra.hw.cpu.cfg.004 |
ra2.k8s.007
|
DevicePlugins Feature Gate |
When hosting workloads matching the Network Intensive profile, the DevicePlugins feature gate must be enabled (note, this is enabled by default in Kubernetes v1.10 or later).
--feature-gates="...,DevicePlugins=true,..."
|
Various, e.g. e.cap.013, e.cap.014 |
ra2.k8s.008
|
System Resource Reservations |
To avoid resource starvation issues on nodes, reserve compute resources for system daemons and Kubernetes system daemons such as kubelet, container runtime, etc. (requires Kubernetes version 1.17 or later). Use the following kubelet flags:
--reserved-cpus=[a-z]
|
TBC |
ra2.k8s.009
|
CPU Pinning |
When hosting workloads matching the Network Intensive profile, in order to support CPU Pinning, the kubelet must be started with the
--cpu-manager-policy=static
option. (Note, only containers in
Guaranteed
pods - where CPU resource
requests
and
limits
are identical - and configured with positive-integer CPU
requests
will take advantage of this. All other Pods will run on CPUs in the remaining shared pool.)
|
infra.com.cfg.003 |
ra2.k8s.010
|
IPv6DualStack |
To support IPv6 and IPv4, the
IPv6DualStack
feature gate must be enabled on various components (requires Kubernetes v1.16 or later). kube-apiserver:
--feature-gates="IPv6DualStack=true"
. kube-controller-manager:
--feature-gates="IPv6DualStack=true" --cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR> --service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR> --node-cidr-mask-size-ipv4 ¦ --node-cidr-mask-size-ipv6
defaults to /24 for IPv4 and /64 for IPv6. kubelet:
--feature-gates="IPv6DualStack=true"
. kube-proxy:
--cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR> --feature-gates="IPv6DualStack=true"
|
req.inf.ntw.04 |
ra2.k8s.011
|
|||
ra2.k8s.012
|
|||
ra2.k8s.013
|
|||
ra2.k8s.014
|
|||
ra2.k8s.015
|
Table 4-2: Kubernetes Specifications
4.4 Container runtimes ¶
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.crt.001
|
Conformance with OCI 1.0 runtime spec | The container runtime must be implemented as per the OCI 1.0 (Open Container Initiative 1.0) specification. | TBC |
ra2.crt.002
|
Kubernetes Container Runtime Interface (CRI) | The kubernetes container runtime must be implemented as per the Kubernetes Container Runtime Interface (CRI) | TBC |
Table 4-3: Container Runtime Specifications
4.5 Networking solutions ¶
In order for the networking solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications:
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.ntw.001
|
CNI multiplexer/metaplugin |
As the selected CNI multiplexer/metapulgin MUST support other CNI plugins (
req.inf.ntw.06
) and SHOULD provide an API based solution to administer the networks from a central point (
req.inf.ntw.03
) the selected CNI multiplexer/metapulgin may be
DANM
. For more detailed feature comparision of CNI multiplexers/metaplugins see Table 4-5
|
req.inf.ntw.06 , req.inf.ntw.03 |
ra2.ntw.002
|
CNI to implement a default network which implements the Kubernetes network model |
A CNI plugin may be used which implements the Kubernetes network model and have capability to handle
NetworkPolicies
|
req.inf.ntw.08 |
ra2.ntw.003
|
NAT less connectivity | An IPVLAN CNI plugin or the MACVLAN CNI may be used | |
ra2.ntw.004
|
User plane networks | User Space CNI may be used. The User Space CNI may use VPP or OVS-DPDK as a backend. | |
ra2.ntw.005
|
SR-IOV | SR-IOV CNI plugin may be used | |
ra2.ntw.006
|
SR-IOV | SR-IOV Device Plugin may be used |
Table 4-4: Networking Solution Specifications
| Requirement | Support in Multus | Support in DANM |
|---|---|---|
The overlay network encapsulation protocol needs to enable ECMP in the underlay (
infra.net.cfg.002
)
|
Supported via another CNI plugin | Supported via another CNI plugin |
NAT (
infra.net.cfg.003
)
|
Supported via another CNI plugin | Supported |
Security Groups (
infra.net.cfg.004
)
|
Not supported | Not supported 1) |
SFC support (
infra.net.cfg.005
)
|
Not relevant | Not relevant |
Traffic patterns symmetry (
infra.net.cfg.006
)
|
Not relevant | Not relevant |
Network resiliency (
req.inf.ntw.01
)
|
Supported | Supported |
Centrally administrated and configured (
req.inf.ntw.03
)
|
Not supported | Partially suported |
Dual stack IPv4 and IPv6 for Kubernetes workloads (
req.inf.ntw.04
)
|
Supported via another CNI plugin | Suported |
Integrating SDN controllers (
req.inf.ntw.05
)
|
Supported via another CNI plugin | Supported via another CNI plugin |
More than one networking solution (
req.inf.ntw.06
)
|
Supported | Supported |
Choose whether or not to deploy more than one networking solution (
req.inf.ntw.07
)
|
Supported | Supported |
Kubernetes network model (
req.inf.ntw.08
)
|
Supported via another CNI plugin | Supported via another CNI plugin |
Do not interfere with or cause interference to any interface or network it does not own (
req.inf.ntw.09
)
|
Supported | Supported |
Cluster wide coordination of IP address assignment (
req.inf.ntw.10
)
|
Supported via another CNI plugin | Supported |
Table 4-5: Comparision of CNI multiplexers/metaplugins
1): Under implementation in the current release.
4.6 Storage components ¶
In order for the storage solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications:
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.stg.001
|
Ephemeral Storage |
An implementation must support ephemeral storage, for the unpacked container images to be stored and executed from, as a directory in the filesystem on the worker node on which the container is running.
See the Container runtimes section above for more information on how this meets the requirement for ephemeral storage for containers. |
|
ra2.stg.002
|
Kubernetes Volumes | An implementation may attach additional storage to containers using Kubernetes Volumes. | |
ra2.stg.003
|
Kubernetes Volumes |
An implementation may use Volume Plugins (see
ra2.stg.005
below) to allow the use of a storage protocol (e.g. iSCSI, NFS) or management API (e.g. Cinder, EBS) for the attaching and mounting of storage into a Pod.
|
|
ra2.stg.004
|
Persistent Volumes |
An implementation may support Kubernetes Persistent Volumes (PV) to provide persistent storage for Pods (requirement 'req.inf.stg.01'.
Persistent Volumes exist independent of the lifecycle of containers and/or pods. |
req.inf.stg.01 |
ra2.stg.005
|
Storage Extension | Volume plugins must allow for the use of a range of backend storage systems. | |
ra2.stg.006
|
Container Storage Interface (CSI) |
An implementation may support the Container Storage Interface (CSI), an Out-of-tree plugin.
In order to support CSI, the feature gates
CSIDriverRegistry
and
CSINodeInfo
must be enabled.
The implementation must use a CSI driver (a full list of CSI drivers can be found here ). An implementation may support ephemeral storage through a CSI-compatible volume plugin in which case the
CSIInlineVolume
feature gate must be enabled.
An implementation may support Persistent Volumes through a CSI-compatible volume plugin in which case the
CSIPersistentVolume
feature gate must be enabled.
|
|
ra2.stg.007
|
An implementation should use Kubernetes Storage Classes to support automation and the separation of concerns between providers of a service and consumers of the service. |
Table 4-6: Storage Solution Specifications
A note on object storage:
- This Reference Architecture does not include any specifications for object storage, as this is neither a native Kubernetes object, nor something that is required by CSI drivers. Object storage is an application-level requirement that would ordinarily be provided by a highly scalable service offering rather than being something an individual Kubernetes cluster could offer.
Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest).
4.7 Service meshes ¶
Application service meshes are not in scope for the architecture. Network service mesh specifications are handled in section 4.5 Networking solutions .
4.8 Kubernetes Application package manager ¶
In order for the storage solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications:
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.pkg.001
|
API-based package management | A package manager must use the Kubernetes APIs to manage application artefacts. Cluster-side components such as Tiller are not supported. | req.int.api.02 |
Table 4-7: Kubernetes Application Package Management Specifications
4.9 Kubernetes workloads ¶
In order for the Kubernetes workloads to be conformant with the Reference Architecture they must be implemented as per the following specifications:
| Ref | Specification | Details | Requirement Trace |
|---|---|---|---|
ra2.app.001
|
Root Parameter Group (OCI Spec) | Specifies the container's root filesystem. | TBD |
ra2.app.002
|
Mounts Parameter Group (OCI Spec) | Specifies additional mounts beyond root | TBD |
ra2.app.003
|
Process Parameter Group (OCI Spec) | Specifies the container process | TBD |
ra2.app.004
|
Hostname Parameter Group (OCI Spec) | Specifies the container's hostname as seen by processes running inside the container | TBD |
ra2.app.005
|
User Parameter Group (OCI Spec) | User for the process is a platform-specific structure that allows specific control over which user the process runs as | TBD |
Table 4-8: Kubernetes Workload Specifications
4.10 Additional required components ¶
This chapter should list any additional components needed to provide the services defined in Chapter 3.2 (e.g: Prometheus)