4. CNF Test Cases and Requirements Traceability¶
4.1. Introduction¶
The scope of this chapter is to identify and list test cases based on requirements defined in Reference Architecture for Kubernetes based cloud infrastructure (RA2). This will serve as traceability between test cases and requirements for Kubernetes platform interoperability.
Note that each requirement may have one or more test cases associated with it.
The Anuket Assured Workload badge requires that workloads are expected to return Must/Must Not results all of the tests/suites identified as priority (see table).
4.2. Selection Criteria¶
Test cases, tools and their dependencies must be open source. The test cases (or test suite with the test case) as well as the environment needed to run the test should be reproducible by any party following publicly available documentation.
Examples of initiatives (having testing tools, test suites, etc) with test cases which could be used include K8s Conformance, K8s e2e, Sonobuoy, Anuket Functest, CNF Conformance.
4.3. Traceability Matrix¶
The following is a Requirements Traceability Matrix (RTM) mapping Test Case, and/or Test Case Coverage, to RM and RA requirements – configuration, deployment, runtime.
4.3.1. Test Case Traceability to RA2 Requirements¶
This section focuses on the test cases covering the requirements in Kubernetes workloads for Kubernetes workloads.
RM/RA Ref |
High-level test definition |
Test name and project |
Priority |
|---|---|---|---|
ra2.app.001 |
Specifies the container’s root filesystem. |
Root Parameter Group (OCI Spec) |
Must |
ra2.app.002 |
Specifies additional mounts beyond root. |
Mounts Parameter Group (OCI Spec) |
Must |
ra2.app.003 |
Specifies the container process. |
Process Parameter Group (OCI Spec) |
Must |
ra2.app.004 |
Specifies the container’s hostname as seen by processes running inside the container. |
Hostname Parameter Group (OCI Spec) |
Must |
ra2.app.005 |
User for the process is a platform-specific structure that allows specific control over which user the process runs as. |
User Parameter Group (OCI Spec) |
Must |
ra2.app.006 |
Consumption of additional, non-default connection points. Any additional non-default connection points must be requested through the use of workload annotations or resource requests and limits within the container spec passed to the Kubernetes API Server. |
Must |
|
ra2.app.007 |
Host Volumes: Workloads should not use hostPath volumes, as Pods with identical configuration (such as those created from a PodTemplate) may behave differently on different nodes due to different files on the nodes. |
Must |
|
ra2.app.008 |
Infrastructure dependency |
Workloads must not rely on the availability of the master nodes for the successful execution of their functionality (i.e. loss of the master nodes may affect non-functional behaviours such as healing and scaling, but components that are already running will continue to do so without issue). |
Must (Not) |
ra2.app.009 |
Device plugins |
Workload descriptors must use the resources advertised by the device plugins to indicate their need for an FPGA, SR-IOV or other acceleration device. |
Must |
ra2.app.010 |
Node Feature Discovery (NFD) |
Workload descriptors must use the labels advertised by Node Feature Discovery to indicate which node software of hardware features they need. |
Must |
ra2.app.011 |
Published helm chart: Helm charts of the CNF must be published into a helm registry and must not be used from local copies. |
Should |
|
ra2.app.012 |
Valid Helm chart: Helm charts of the CNF must be valid and should pass the helm lint validation. |
Should |
|
ra2.app.013 |
Rolling update: Rolling update of the CNF must be possible using Kubernetes deployments. |
Must |
|
ra2.app.014 |
Rolling downgrade: Rolling downgrade of the CNF must be possible using Kubernetes deployments. |
Must |
|
ra2.app.015 |
CNI compatibility: The CNF must use CNI compatible networking plugins. |
Must |
|
ra2.app.016 |
Kubernetes API stability: The CNF must not use any Kubernetes alpha API-s. |
Must (Not) |
|
ra2.app.017 |
CNF resiliency (node drain): CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of a node drain and rescheduling occurs. |
Must (Not) |
|
ra2.app.018 |
CNF resiliency (network latency): CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of network latency up to 2000 ms occurs. |
Must (Not) |
|
ra2.app.019 |
CNF resiliency (pod delete) CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of pod delete occurs. |
Must (not) |
|
ra2.app.020 |
CNF resiliency (pod memory hog): CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of pod memory hog occurs. |
Must (Not) |
|
ra2.app.021 |
CNF resiliency (pod I/O stress): CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of pod I/O stress occurs. |
Must (Not) |
|
ra2.app.022 |
CNF resiliency (pod network corruption): CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of pod network corruption occurs. |
Must (Not) |
|
ra2.app.023 |
CNF resiliency (pod network duplication): CNF must not loose data, must continue to run and its readiness probe outcome must be Success even in case of pod network duplication occurs. |
Must (Not) |
|
ra2.app.024 |
CNF resiliency (pod DNS error): CNF must not lose data, must continue to run and its readiness probe outcome must be Success even in case of pod DNS error occurs. |
N/A |
Must (Not) |
ra2.app.025 |
CNF local storage: CNF must not use local storage. |
Must (Not) |
|
ra2.app.026 |
Liveness probe: All Pods of the CNF must have livenessProbe defined. |
Must |
|
ra2.app.027 |
Readiness probe: All Pods of the CNF must have readinessProbe defined. |
Must |
|
ra2.app.028 |
No access to container daemon sockets: The CNF must not have any of the container daemon sockets (e.g.: /var/run/docker.sock, /var/run/containerd.sock or /var/run/crio.sock) mounted. |
N/A |
Must (Not) |
ra2.app.029 |
No automatic service account mapping: Non specified service accounts must not be automatically mapped. To prevent this the automountServiceAccountToken: false flag must be set in all Pods of the CNF. |
Must (Not) |
|
ra2.app.030 |
No host network access: Host network must not be attached to any of the Pods of the CNF. hostNetwork attribute of the Pod specifications must be False or should not be specified. |
Must (Not) |
|
ra2.app.031 |
Host process namespace separation: Pods of the CNF must not share the host process ID namespace or the host IPC namespace. Pod manifests must not have the hostPID or the hostIPC attribute set to true. |
Must (Not) |
|
ra2.app.032 |
Resource limits: All containers and namespaces of the CNF must have defined resource limits for at least CPU and memory resources. |
Must |
|
ra2.app.033 |
Read only filesystem: All containers of the CNF must have a read only filesystem. The readOnlyRootFilesystem attribute of the Pods in the their securityContext should be set to true. |
Must |
|
ra2.app.034 |
Container image tags: All referred container images in the Pod manifests must be referred by a version tag pointing to a concrete version of the image. latest tag must not be used |
N/A |
Must |
ra2.app.035 |
No hardcoded IP addresses: The CNF must not have any hardcoded IP addresses in its Pod specifications. |
Must (Not) |
|
ra2.app.036 |
No node ports: Service declarations of the CNF must not contain nodePort definition. |
Must (Not) |
|
ra2.app.037 |
Immutable config maps: ConfigMaps used by the CNF must be immutable. |
Must |
|
ra2.app.038 |
Horizontal scaling:Increasing and decreasing of the CNF capacity should be implemented using horizontal scaling. If horizontal scaling is supported, automatic scaling must be possible using Kubernetes Horizontal Pod Autoscale (HPA) feature. |
TBD |
Should |
ra2.app.039 |
CNF image size: The different container images of the CNF should not be bigger than 5GB. |
Should (Not) |
|
ra2.app.040 |
CNF startup time: Startup time of the Pods of a CNF should not be more than 60s where startup time is the time between starting the Pod until the readiness probe outcome is Success. |
Should (Not) |
|
ra2.app.041 |
No privileged mode: None of the Pods of the CNF should run in privileged mode. |
Should (Not) |
|
ra2.app.042 |
No root user: None of the Pods of the CNF should run as a root user. |
Should (Not) |
|
ra2.app.043 |
No privilege escalation: None of the containers of the CNF should allow privilege escalation. |
Should (Not) |
|
ra2.app.044 |
Non-root user: All Pods of the CNF should be able to execute with a non-root user having a non-root group. Both runAsUser and runAsGroup attributes should be set to a greater value than 999. |
Should |
|
ra2.app.045 |
Labels: Pods of the CNF should define at least the following labels: app.kubernetes.io/name, app.kubernetes.io/version and app.kubernetes.io/part-of |
Should |