AKS on Azure and has been generally available for years. AKS on Azure Stack HCI is an on-premises implementation of AKS, it runs in customer environments on customer managed hardware. Here’s a list of general similarities and differences between AKS-HCI and AKS.
Features in preview are marked by (*)
| Feature Set | AKS on Azure Stack HCI & AKS on Windows Server | AKS |
| Kubernetes Management Cluster/AKS host | AKS on Azure Stack HCI and Windows Server is a Cluster API based hosted Kubernetes offering. A management Kubernetes cluster is used to manage Kubernetes workload clusters. The management Kubernetes cluster runs in customer datacenters and is managed by the infrastructure administrator. | AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions. |
| Kubernetes Target Cluster (lifecycle operations) | ||
| Cloud Native Computing Foundation (CNCF) certification | Yes | Yes |
| Who manages the cluster? | Managed by you | Managed by you |
| Where is the cluster located? | In your datacenter alongside your AKS hybrid management cluster.
Azure Stack HCI 21H2 Windows Server 2019 Datacenter Windows Server 2022 Datacenter Windows 10/11 IoT Enterprise* |
Azure cloud |
| K8s cluster lifecycle management tools (create, scale, update and delete clusters) | PowerShell (PS)
Windows Admin Center (WAC) Az CLI* Azure Portal* ARM templates* |
Az CLI
Az PowerShell Azure Portal Bicep ARM templates |
| Can you use kubectl and other open-source Kubernetes tools? | Yes | Yes |
| Workload cluster updates | K8s version upgrade through PowerShell or WAC. Initiated by you.
Node OS image update initiated by you; Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated. |
Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions;
OS image patch upgrade; Automatic upgrades; Planned maintenance windows; |
| Kubernetes versions | Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub. | Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions. |
| Can you start/stop K8s clusters to save costs? | Yes, by stopping the underlying failover cluster | Yes |
| Azure Fleet Manager integration | Not yet. | Yes* |
| Terraform support | Not yet. | Yes |
| Node Pools | ||
| Do you support running Linux and Windows node pools in the same cluster? | Yes!
Linux nodes: CBL-Mariner Windows nodes: Windows Server 2019 Datacenter, Windows Server 2022 Datacenter |
Yes.
Linux nodes: Ubuntu 18.04, CBL-Mariner Windows nodes: Windows Server 2019 Datacenter Windows Server 2022 Datacenter |
| What’s your container runtime? | Linux nodes: containerd
Windows nodes: containerd |
Linux nodes: containerd
Windows nodes: containerd |
| Can you scale node pools? | Manually
Cluster autoscaler Vertical pod autoscalar |
Manually
Cluster autoscaler Vertical pod autoscalar |
| Horizontal pod autoscalar | Yes | Yes |
| What about virtual nodes? | ||
| Azure container instance | No | Yes |
| Can you upgrade a node pool? | We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level. | You can perform node pool specific upgrades in an AKS cluster. |
| GPU enabled node pools | Yes* | Yes |
| Azure Container Registry | Yes | Yes |
| KEDA support | Not yet | Yes* |
| Networking | ||
| Who creates and manages the networks? | All networks (for both the management cluster and target K8s clusters) are created and managed by you | By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters |
| What type of network options are supported? | DHCP networks with/without VLAN ID
Static IP networks with/without VLAN ID SDN support for AKS on Azure Stack HCI |
Bring your own Azure virtual network for AKS clusters. |
| Load balancers | HAProxy (default) runs in a separate VM in the target K8s cluster
kubeVIP – runs as a K8s service in the control plane K8s node Bring your own load balancer Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability. You can create multiple instances of a LB (active-passive) for high availability |
Azure load balancer – Basic SKU or Standard SKU
Can also use internal load balancer By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service |
| CNI/Network plugin | Calico (default)
Note: Network policies are covered in the Security and Authentication section. |
Azure CNI
Calico Azure CNI Overlay Bring your own CNI Note: Network policies are covered in the Security and Authentication section. |
| Ingress controllers | No but you can use 3rd party addons – Nginx. 3rd party addons are not supported by Microsoft’s support policy. |
Support for Nginx with web app routing addon. |
| Egress controls | Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies. | You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups. |
| Egress types | Egress types and options depend on your network architecture. | Azure load balancer, managed NAT gateway and user defined routes are the supported egress types. |
| Customize CoreDNS | Allowed | Allowed |
| Service Mesh | Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes.
3rd party addons – Istio, etc. 3rd party addons are not supported by Microsoft’s support policy. |
Open Service Mesh
Marketplace offering available for Istio |
| Storage | ||
| Where is the storage provisioned? | On-premises | Azure Storage.
Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class. |
| What types of persistent volumes are supported? | Read Write Once
Read Write Many |
Read Write Once
Read Write Many |
| Do the storage drivers support Container Storage Interface (CSI)? | Yes | Yes |
| Is dynamic provisioning supported? | Yes | Yes |
| Is volume resizing supported? | Yes | Yes |
| Are volume snapshots supported? | No | Yes |
| Security and Authentication | ||
| How do you access your Kubernetes cluster? | Certificate based kubeconfig (default)
AD based kubeconfig Azure AD and Kubernetes RBAC Azure AD and Azure RBAC* |
Certificate based kubeconfig (default)
Azure AD and Kubernetes RBAC Azure AD and Azure RBAC |
| Network Policies | Yes, we support Calico network policies | Yes, we support Calico and Azure CNI network policies |
| Limit source networks that can access API server | Yes, by using VIP pools. | Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters. |
| Certificate rotation and secrets encryption | Yes | Yes |
| Support for private cluster | Not supported yet | Yes! You can create private AKS clusters |
| Secrets store CSI driver | Yes | Yes |
| Support for disk encryption | Yes, via bitlocker | Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys.
Hosts and locally attached disks can also be encrypted with encryption at host. |
| gMSA v2 support for Windows containers | Yes | Yes |
| Azure Policy | Yes, through Azure Arc enabled K8s | Yes |
| Azure Defender | Yes, through Azure Arc enabled K8s* | Yes |
| Monitoring and Logging | ||
| Collect logs | Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected. | Yes, through Azure Portal, Az CLI, etc |
| Support for Azure Monitor | Yes, through Azure Arc enabled K8s. | Yes |
| 3rd party addons for monitoring and logging | AKS works with Azure managed Prometheus* and Azure managed Grafana* | |
| Subscribe to Azure Event Grid Events | Yes, via Azure Arc enabled Kubernetes* | Yes |
| Develop and run applications | ||
| Azure App service | Yes, through Azure Arc enabled K8s* | Yes |
| Azure Functions | Yes, through Azure Arc enabled K8s* | Yes |
| Azure Logic Apps | Yes, through Azure Arc enabled K8s* | You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS |
| Develop applications using Helm | Yes | Yes |
| Develop applications using Dapr | Yes, through Azure Arc enabled K8s* | Yes |
| DevOps | Azure DevOps via Azure Arc enabled K8s.
GitHub Actions via Azure Arc enabled K8s. GitOps Flux v2 via Azure Arc enabled K8s. 3rd party addon: ArgoCD. 3rd party addons are not supported by Microsoft’s support policy. GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers. |
Azure DevOps
GitHub Actions GitOps Flux v2 |
| Product Pricing | ||
| Product pricing | If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost.
If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free. |
Unlimited free clusters, pay for on-demand compute of the worker nodes.
Paid tier available with uptime SLA, support for 5k nodes. |
| Azure Support | AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support. | AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal. |
| SLA | We do not offer SLAs since AKS-HCI runs in your environment. | Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs. |
This article appears courtesy of Microsoft.