[<< Back](../../kubernetes) # 4. Component Level Architecture
## Table of Contents - [4. Component Level Architecture](#4-component-level-architecture) - [4.1 Introduction](#41-introduction) - [4.2 Kubernetes Node](#42-kubernetes-node) - [4.3 Kubernetes](#43-kubernetes) - [4.4 Container runtimes](#44-container-runtimes) - [4.5 Networking solutions](#45-networking-solutions) - [4.6 Storage components](#46-storage-components) - [4.7 Service meshes](#47-service-meshes) - [4.8 Kubernetes Application package manager](#48-kubernetes-application-package-manager) - [4.9 Kubernetes workloads](#49-kubernetes-workloads) - [4.10 Additional required components](#410-additional-required-components) ## 4.1 Introduction This chapter describes in detail the Kubernetes Reference Architecture in terms of the functional capabilities and how they relate to the Reference Model requirements, i.e. how the infrastructure profiles are determined, documented and delivered. The specifications defined in this chapter will be detailed with unique identifiers, which will follow the pattern: `ra2.Figure 4-1: Kubernetes Reference Architecture
## 4.2 Kubernetes Node This section describes the configuration that will be applied to the physical or virtual machine and an installed Operating System. In order for a Kubernetes Node to be conformant with the Reference Architecture it must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.ch.001`|Huge pages|When hosting workloads matching the Network Intensive profile, it **must** be possible to enable Huge pages (2048KiB and 1048576KiB) within the Kubernetes Node OS, exposing schedulable resources `hugepages-2Mi` and `hugepages-1Gi`.|[infra.com.cfg.004](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| |`ra2.ch.002`|SR-IOV capable NICs|When hosting workloads matching the Network Intensive profile, the physical machines on which the Kubernetes Nodes run **must** be equipped with NICs that are SR-IOV capable.|[e.cap.013](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| |`ra2.ch.003`|SR-IOV Virtual Functions|When hosting workloads matching the Network Intensive profile, SR-IOV virtual functions (VFs) **must** be configured within the Kubernetes Node OS, as the SR-IOV Device Plugin does not manage the creation of these VFs.|[e.cap.013](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| |`ra2.ch.004`|CPU Simultaneous Multi-Threading (SMT)|SMT **must** be enabled in the BIOS on the physical machine on which the Kubernetes Node runs.|[infra.hw.cpu.cfg.004](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| |`ra2.ch.005`|CPU Allocation Ratio - VMs|For Kubernetes nodes running as Virtual Machines, the CPU allocation ratio between vCPU and physical CPU core **must** be 1:1.|[infra.com.cfg.001](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|| |`ra2.ch.006`|CPU Allocation Ratio - Pods|To ensure the CPU allocation ratio between vCPU and physical CPU core is 1:1, the sum of CPU requests and limits by containers in Pod specifications **must** remain less than the allocatable quantity of CPU resources (i.e. `requests.cpu < allocatable.cpu` and `limits.cpu < allocatable.cpu`).|[infra.com.cfg.001](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| |`ra2.ch.007`|IPv6DualStack|To support IPv4/IPv6 dual stack networking, the Kubernetes Node OS **must** support and be allocated routable IPv4 and IPv6 addresses.|[req.inf.ntw.04](./chapter02.md#23-kubernetes-architecture-requirements)|| |`ra2.ch.008`|Physical CPU Quantity|The physical machines on which the Kubernetes Nodes run **must** be equipped with at least 2 physical sockets, each with at least 20 CPU cores.|[infra.hw.cpu.cfg.001](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)Table 4-1: Node Specifications
## 4.3 Kubernetes In order for the Kubernetes components to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.k8s.001`|Kubernetes Conformance|The Kubernetes distribution, product, or installer used in the implementation **must** be listed in the [Kubernetes Distributions and Platforms document](https://docs.google.com/spreadsheets/d/1uF9BoDzzisHSQemXHIKegMhuythuq_GL3N1mlUUK2h0/edit#gid=0) and marked (X) as conformant for the Kubernetes version defined in [README](../README.md#required-versions-of-most-important-components).|[req.gen.cnt.03](./chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| |`ra2.k8s.002`|Highly available etcd|An implementation **must** consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes).|[req.gen.rsl.02 req.gen.avl.01](./chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| |`ra2.k8s.003`|Highly available control plane|An implementation **must** consist of at least one master node per availability zone or fault domain to ensure the high availability and resilience of the Kubernetes control plane services.|[req.gen.rsl.02](./chapter02.md#23-kubernetes-architecture-requirements)Table 4-2: Kubernetes Specifications
## 4.4 Container runtimes |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.crt.001`|Conformance with OCI 1.0 runtime spec|The container runtime **must** be implemented as per the [OCI 1.0](https://github.com/opencontainers/runtime-spec/blob/master/spec.md) (Open Container Initiative 1.0) specification.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| |`ra2.crt.002`|Kubernetes Container Runtime Interface (CRI)|The Kubernetes container runtime **must** be implemented as per the [Kubernetes Container Runtime Interface (CRI)](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)|Table 4-3: Container Runtime Specifications
## 4.5 Networking solutions In order for the networking solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.ntw.001`|Centralised network administration|The networking solution deployed within the implementation **must** be administered through the Kubernetes API using native Kubernetes API resources and objects, or Custom Resources.|[req.inf.ntw.03](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| |`ra2.ntw.002`|Default Pod Network - CNI|The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin for the Default Pod Network, as the alternative (kubenet) does not support cross-node networking or Network Policies.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)Table 4-4: Networking Solution Specifications
## 4.6 Storage components In order for the storage solutions to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.stg.001`| Ephemeral Storage | An implementation must support ephemeral storage, for the unpacked container images to be stored and executed from, as a directory in the filesystem on the worker node on which the container is running.Table 4-6: Storage Solution Specifications
A note on object storage: - This Reference Architecture does not include any specifications for object storage, as this is neither a native Kubernetes object, nor something that is required by CSI drivers. Object storage is an application-level requirement that would ordinarily be provided by a highly scalable service offering rather than being something an individual Kubernetes cluster could offer. > Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest). ## 4.7 Service meshes Application service meshes are not in scope for the architecture. The service mesh is a dedicated infrastructure layer for handling service-to-service communication, and it is recommended to secure service-to-service communications within a cluster and to reduce the attack surface. The benefits of the service mesh framework are described in [5.4.3](./chapter05.md#543-use-transport-layer-security-and-service-mesh). In addition to securing communications, the use of a service mesh extends Kubernetes capabilities regarding observability and reliability. Network service mesh specifications are handled in section [4.5 Networking solutions](#45-networking-solutions). ## 4.8 Kubernetes Application package manager In order for the application package managers to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.pkg.001`|API-based package management| A package manager must use the Kubernetes APIs to manage application artifacts. Cluster-side components such as Tiller are not supported. | [req.int.api.02](./chapter02.md#23-kubernetes-architecture-requirements) ||Table 4-7: Kubernetes Application Package Manager Specifications
## 4.9 Kubernetes workloads In order for the Kubernetes workloads to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| |`ra2.app.001`| [Root](https://github.com/opencontainers/runtime-spec/blob/master/config.md) Parameter Group (OCI Spec) | Specifies the container's root filesystem. | TBD | N/A | |`ra2.app.002`| [Mounts](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) Parameter Group (OCI Spec) | Specifies additional mounts beyond root. | TBD | N/A | |`ra2.app.003`| [Process](https://github.com/opencontainers/runtime-spec/blob/master/config.md#process) Parameter Group (OCI Spec) | Specifies the container process. | TBD | N/A | |`ra2.app.004`| [Hostname](https://github.com/opencontainers/runtime-spec/blob/master/config.md#hostname) Parameter Group (OCI Spec) | Specifies the container's hostname as seen by processes running inside the container. | TBD | N/A | |`ra2.app.005`| [User](https://github.com/opencontainers/runtime-spec/blob/master/config.md#user) Parameter Group (OCI Spec) | User for the process is a platform-specific structure that allows specific control over which user the process runs as. | TBD | N/A | |`ra2.app.006`| Consumption of additional, non-default connection points | The workload must request additional non-default connection points through the use of workload annotations or resource requests and limits within the container spec passed to the Kubernetes API Server. | [req.int.api.01](chapter02.md#23-kubernetes-architecture-requirements) | N/A | |`ra2.app.007`| Host Volumes | Workloads should not use `hostPath` volumes, as [Pods with identical configuration](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) (such as those created from a PodTemplate) may behave differently on different nodes due to different files on the nodes. | [req.kcm.gen.02](chapter02.md#23-kubernetes-architecture-requirements). | N/A | |`ra2.app.008`| Infrastructure dependency | Workloads must not rely on the availability of the master nodes for the successful execution of their functionality (i.e. loss of the master nodes may affect non-functional behaviours such as healing and scaling, but components that are already running will continue to do so without issue). | TBD | N/A | |`ra2.app.009`| Device plugins | Workload descriptors must use the resources advertised by the device plugins to indicate their need for an FPGA, SR-IOV or other acceleration device. | TBD | N/A | |`ra2.app.010`| Node Feature Discovery (NFD)|Workload descriptors must use the labels advertised by [Node Feature Discovery](https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html) to indicate which node software of hardware features they need. | TBD | N/A |Table 4-8: Kubernetes Workload Specifications
## 4.10 Additional required components > This chapter should list any additional components needed to provide the services defined in Chapter 3.2 (e.g., Prometheus)