6. API and Feature Testing requirements¶
6.1. Introduction to API and Feature Testing requirement¶
The CNCF has defined a Kubernetes Testing Special Interest Group [111] to help the community to write and run tests, and to contribute, analyze, and act upon test results. This chapter maps the requirements written in the previous chapters as mandatory Special Interest Group features. It enforces the overall requirements traceability to testing, especially those offered for Kubernetes End-to-End Testing [112]. The Anuket Reference Conformance (RC2) testing matches the features and tests defined here.
6.1.1. Kubernetes feature gate policy¶
Kubernetes Feature gates [113] are a set of key-value pairs that describe the Kubernetes features. The components of the control plane of the Kubernetes Clusters can be run with different Feature Gate settings.
A feature can be in the Alpha, Beta, or General Availability (GA) stage:
Alpha features are disabled by default. Breaking API changes may be expected. They may contain bugs, and support may be dropped.
Beta features are disabled by default. They are well tested, and support will not be dropped, although breaking API changes may happen. As of 1.24, any existing Beta feature will continue to be enabled by default. However, new beta APIs and features will not be enabled by default after Kubernetes 1.24. For more information, see Kubernetes KEP-3136 [114]
GA features are stable. They are always enabled and cannot be disabled.
Only those Kubernetes features can be made mandatory in this Reference Architecture which are GA or were Beta before Kubernetes 1.24.
A list of feature gates is available here [113].
6.1.2. Kubernetes API policy¶
The Kubernetes API [115] supports all operations and communications between components, and external user commands. Everything in the Kubernetes platform is treated as an API object. Different API versions indicate different levels of stability and support. An API can have Alpha, Beta or Stable versions. The version of APIs that are backed by a feature will match the stage of the feature itself (i.e. Alpha, Beta or GA or Stable).
The policy for RA2 to include Kubernetes APIs as mandatory is as follows:
In these Reference Architecture APIs, only those APIs which are in any of the following stages are mandatory:
Stable.
Beta when introduced before Kubernetes version 1.24.
Alpha or Beta when required by RA2 Ch4 specifications, or when included on the list of Mandatory API Groups below.
The Kubernetes API reference is available here [116].
The list of Kubernetes API Groups [117] that are mandatory is as follows:
Group |
Version |
|---|---|
admissionregistration.k8s.io |
v1 |
apiextensions.k8s.io |
v1 |
apiregistration.k8s.io |
v1 |
apps |
v1 |
authentication.k8s.io |
v1 |
authorization.k8s.io |
v1 |
autoscaling |
v1, v2 |
batch |
v1 |
certificates.k8s.io |
v1 |
coordination.k8s.io |
v1 |
core |
v1 |
discovery.k8s.io |
v1 |
events.k8s.io |
v1 |
flowcontrol.apiserver.k8s.io |
v1 |
networking.k8s.io |
v1 |
node.k8s.io |
v1 |
policy |
v1 |
rbac.authorization.k8s.io |
v1 |
scheduling.k8s.io |
v1 |
storage.k8s.io |
v1 |
6.2. API Machinery Special Interest Group [118]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test |
None |
X |
Kubernetes mainstream features |
Feature:ComprehensiveNamespaceDraining |
X |
The deletion of namespaces should always be fast (all 100 namespaces in 150 seconds). |
Feature: CrossNamespacePodAffinity [119] |
The CrossNamespacePodAffinity feature verifies the ResourceQuota with the cross namespace pod affinity scope using scope-selectors. |
|
Feature: PodPriority [119] |
X |
The PodPriority feature verifies the ResourceQuota’s priority class scope against a pod with a different priority class. |
Feature:ScopeSelectors |
X |
Verify ResourceQuota with terminating scopes through scope selectors |
Feature: StorageVersionAPI [120] |
Enable the storage version API. |
|
Feature:WatchList [113] |
Enable support for streaming initial state of objects in watch requests. |
6.3. Apps [121]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test |
None |
X |
Kubernetes mainstream features |
Feature: DaemonSetUpdateSurge [122] |
The Daemon set should surge the pods onto the nodes when the specification is updated and the update strategy is RollingUpdate. |
|
Feature: IndexedJob [123] |
The IndexedJob feature should create pods for an indexed job with completion indexes. |
|
Feature: StatefulSet [124] |
The StatefulSet feature should create a working zookeeper cluster. |
|
Feature:StatefulUpgrade |
The StatefulUpgrade feature should maintain a functioning cluster. |
|
Feature: SuspendJob [125] |
The SuspendJob feature should not create pods when they have been created in a suspended state. |
|
Feature: TaintEviction [126] |
All pods on the unreachable node should be marked as NotReady when the node condition is set to NotReady. All pods should be evicted after eviction timeout has passed. |
|
Feature: TTLAfterFinished [127] |
X |
The job should be deleted once it has finished, after the TTL has elapsed. |
6.4. Auth Special Interest Group [128]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test |
None |
X |
Kubernetes mainstream features |
Feature: BoundServiceAccountTokenVolume [129] |
The ServiceAccount admission controller migration upgrade should maintain a functioning cluster. |
|
Feature:ClusterTrustBundle [113] |
Enable ClusterTrustBundle objects and kubelet integration. |
|
Feature:NodeAuthenticator |
X |
The kubelet’s main port 10250 should reject requests with no credentials. |
Feature:NodeAuthorizer |
X |
Setting existing and non-existent attributes should return with a Forbidden error, not a NotFound error. |
NodeFeature:FSGroup |
X |
ServiceAccounts should set ownership and permission when RunAsUser or FsGroup is present. |
6.5. Cluster Lifecycle Special Interest Group [130]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test |
None |
X |
Kubernetes mainstream features |
Feature:BootstrapTokens |
X |
The BootstrapTokens feature should delete the token secret when the secret has expired. |
6.6. Instrumentation Special Interest Group [131]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test |
None |
X |
Kubernetes mainstream features |
Feature:Elasticsearch |
The Elasticsearch feature should check that the Kibana logging instance is alive. |
|
Feature: StackdriverAcceleratorMonitoring |
Stackdriver Monitoring should have accelerator metrics. |
|
Feature:StackdriverCustomMetrics |
Stackdriver Monitoring should run Custom Metrics - Stackdriver Adapter for the new resource model. |
|
Feature:StackdriverExternalMetrics |
Stackdriver Monitoring should run Custom Metrics - Stackdriver Adapter for external metrics. |
|
Feature:StackdriverMetadataAgent |
Stackdriver Monitoring should run Stackdriver Metadata Agent. |
|
Feature:StackdriverMonitoring |
6.7. Network Special Interest Group [132]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test. |
None |
X |
Kubernetes mainstream features. |
Feature:Example |
The example feature should create a pod that uses DNS. |
|
Feature:Ingress |
The Ingress feature should prevent ingress creation if more than one IngressClass is marked as a default. |
|
Feature: IPv6DualStack [133] |
IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to Pods and Services. IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster from 1.21 onwards, allowing the simultaneous assignment of IPv4 and IPv6 addresses. |
|
Feature:kubemci |
The kubemci feature should create ingress with a preshared certificate. |
|
Feature:KubeProxyDaemonSetMigration |
The upgrade of kube-proxy from static pods to a DaemonSet should maintain a functioning cluster. |
|
Feature:KubeProxyDaemonSetUpgrade |
The upgrade of kube-proxy from static pods to a DaemonSet should maintain a functioning cluster. |
|
Feature:NEG |
The NEG feature should sync the endpoints to NEG. |
|
Feature:NoSNAT |
X |
The NoSNAT feature should be able to send traffic between the Pods without SNAT. |
Feature:Networking-IPv4 |
X |
Networking-IPv4 should provide an IPv4 connection for the containers. |
Feature:Networking-IPv6 |
Networking-IPv6 should provide an IPv6 connection for the containers. |
|
Feature:Networking-Performance |
X |
Measure network responsiveness, latency (both RTT and OWD), and throughput with the iperf2 tool. |
Feature:NetworkPolicy |
NetworkPolicy between the server and the client should enforce a policy to allow traffic only from a different namespace, based on NamespaceSelector. |
|
Feature:PerformanceDNS |
The PerformanceNDS feature should answer DNS queries for a maximum number of services per cluster. |
|
Feature:SCTP |
SCTP should allow the creation of a basic SCTP service with the pod and the endpoints. |
|
Feature:SCTPConnectivity |
The Pods should function for intra-pod communication: sctp. |
|
Feature:ServiceCIDRs [134] |
Track IP address allocations for Service cluster IPs using IPAddress objects. |
6.8. Node Special Interest Group [135]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test. |
None |
X |
Kubernetes mainstream features. |
Feature:DynamicResourceAllocation [113] |
Enables support for resources with custom parameters and a lifecycle that is independent of a Pod. |
|
Feature:Example |
X |
The liveness pods should be automatically restarted. |
Feature: ExperimentalResourceUsageTracking |
Resource tracking for 100 pods per node. |
|
Feature:GPUUpgrade |
The Control Plane node upgrade should not disrupt the GPU Pod. |
|
Feature:InPlacePodVerticalScaling [113] |
Enables in-place Pod vertical scaling. |
|
Feature:NodeLogQuery [113] |
Enables querying logs of node services using the /logs endpoint. |
|
Feature:PodGarbageCollector |
The PodGarbageCollector feature should handle the creation of 1000 pods. |
|
Feature:PodLifecycleSleepAction [113] |
Enables the sleep action in Container lifecycle hooks. |
|
Feature:RegularResourceUsageTracking |
Resource tracking for 0 pods per node. |
|
Feature:SidecarContainers [113] |
Allow setting the restartPolicy of an init container to Always so that the container becomes a sidecar container (restartable init containers). |
|
Feature:UserNamespacesSupport [113] |
Enable user namespace support for Pods. |
|
Feature: ProbeTerminationGracePeriod [136] |
X |
The probing container should override timeoutGracePeriodSeconds when the LivenessProbe field is set. |
NodeFeature: DownwardAPIHugePages [137] |
Downward API tests for huge pages should provide the container’s limits.hugepages-pagesize, and requests.hugepages-pagesize as environmental variables. |
|
NodeFeature: PodReadinessGate [138] |
X |
The Pods should support the pod readiness gates. |
NodeFeature:RuntimeHandler |
The RuntimeClass feature should run a Pod requesting a RuntimeClass with a configured handler. |
|
NodeFeature: Sysctls [139] |
X |
The Sysctls feature should not launch unsafe, but not explicitly enabled sysctls on the node. |
6.9. Scheduling Special Interest Group [140]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test. |
None |
X |
Kubernetes mainstream features. |
Feature:GPUDevicePlugin |
The GPUDevicePlugin feature runs Nvidia GPU Device Plugin tests. |
|
Feature: LocalStorageCapacityIsolation [141] |
X |
The LocalStorageCapacityIsolation feature validates local ephemeral storage resource limits of pods that are allowed to run. |
Feature:Recreate |
The Recreate feature runs Nvidia GPU Device Plugin tests with a recreation. |
6.10. Storage Special Interest Group [142]¶
Labels |
Mandatory |
Description |
|---|---|---|
Conformance |
X |
Kubernetes conformance test. |
None |
X |
Kubernetes mainstream features. |
Feature:ExpandInUsePersistentVolumes |
||
Feature:Flexvolumes |
||
Feature:RecoverVolumeExpansionFailure [113] |
Enables users to edit their PVCs to smaller sizes so as they can recover from previously issued volume expansion failures. |
|
Feature:SELinux |
||
Feature:GKELocalSSD |
||
Feature:VolumeSnapshotDataSource |
||
Feature:Volumes |
X |
|
Feature:vsphere |
||
Feature:Windows |
||
NodeFeature:EphemeralStorage |
X |
|
NodeFeature:FSGroup |
X |