Known Issues and Limitations
This section contains current known issues and limitations with rke2. If you come across issues with rke2 not documented here, please open a new issue here.
Firewalld conflicts with default networking
Firewalld conflicts with RKE2's default Canal (Calico + Flannel) networking stack. To avoid unexpected behavior, firewalld should be disabled on systems running RKE2.
NetworkManager manipulates the routing table for interfaces in the default network namespace where many CNIs, including RKE2's default, create veth pairs for connections to containers. This can interfere with the CNI’s ability to route correctly. As such, if installing RKE2 on a NetworkManager enabled system, it is highly recommended to configure NetworkManager to ignore calico/flannel related network interfaces. In order to do this, create a configuration file called
/etc/NetworkManager/conf.d with the contents:
If you have not yet installed RKE2, a simple
systemctl reload NetworkManager will suffice to install the configuration. If performing this configuration change on a system that already has RKE2 installed, a reboot of the node is necessary to effectively apply the changes.
In some operating systems like RHEL 8.4, NetworkManager includes two extra services called
nm-cloud-setup.timer. These services add a routing table that interfere with the CNI plugin's configuration. Unfortunately, there is no config that can avoid that as explained in the issue. Therefore, if those services exist, they should be disabled.
Before NetworkManager-1.30.0-11.el8_4, the node must also be rebooted after disabling the extra services.
Istio in Selinux Enforcing System Fails by Default
This is due to just-in-time kernel module loading of rke2, which is disallowed under Selinux unless the container is privileged. To allow Istio to run under these conditions, it requires two steps:
- Enable CNI as part of the Istio install. Please note that this feature is still in Alpha state at the time of this writing.
- After the install is complete, there should be
cni-nodepods in a CrashLoopBackoff. Manually edit their daemonset to include
securityContext.privileged: trueon the
This can be performed via a custom overlay as follows:
- apiVersion: "apps/v1"
- path: spec.template.spec.containers.[name:install-cni].securityContext.privileged
For more information regarding exact failures with detailed logs when not following these steps, please see Issue 504.
Control Groups V2
RKE2 v1.19.5+ ships with
containerd v1.4.x or later, hence should run on cgroups v2 capable systems.
Older versions (< 1.19.5) are shipped with containerd 1.3.x fork (with back-ported SELinux commits from 1.4.x) which does not support cgroups v2 and requires a little up-front configuration:
systemd-based system, setting the systemd.unified_cgroup_hierarchy=0
kernel parameter will indicate to systemd that it should run with hybrid (cgroups v1 + v2) support.
Combined with the above, setting the systemd.legacy_systemd_cgroup_controller
kernel parameter will indicate to systemd that it should run with legacy (cgroups v1) support.
As these are kernel command-line arguments they must be set in the system bootloader so that they will be
systemd as PID 1 at
Calico with vxlan encapsulation
Calico hits a kernel bug when using vxlan encapsulation and the checksum offloading of the vxlan interface is on.
The issue is described in the calico project and in
rke2 project. The workaround we are applying is disabling the checksum
offloading by default by applying the value
ChecksumOffloadBroken=true in the calico helm chart.
This issue has been observed in Ubuntu 18.04, Ubuntu 20.04 and openSUSE Leap 15.3
Wicked configures the networking settings of the host based on the sysctl configuration files (e.g. under /etc/sysctl.d/ directory). Even though rke2 is setting parameters such as
/net/ipv4/conf/all/forwarding to 1, that configuration could be reverted by Wicked whenever it reapplies the network configuration (there are several events that result in reapplying the network configuration as well as rcwicked restart during updates). Consequently, it is very important to enable ipv4 (and ipv6 in case of dual-stack) forwarding in sysctl configuration files. For example, it is recommended to create a file with the name
/etc/sysctl.d/90-rke2.conf containing these parameters (ipv6 only needed in case of dual-stack):
Canal and IP exhaustion
There are two possible reasons for this:
iptablesbinary is not installed in the host and there is a pod defining a hostPort. The pod will be given an IP but its creation will fail and Kubernetes will not cease trying to recreate it, consuming one IP every time it tries. Error messages similar to the following will appear in the containerd log. This is the log showing the error:
plugin type="portmap" failed (add): failed to open iptables: exec: "iptables": executable file not found in $PATH
Please install iptables or xtables-nft package to resolve this problem
- By default Canal keeps track of pod IPs by creating a lock file for each IP in
/var/lib/cni/networks/k8s-pod-network. Each IP belongs to a single pod and will be deleted as soon as the pod is removed. However, in the unlikely event that containerd loses track of the running pods, lock files may be leaked and Canal will not be able to reuse those IPs anymore. If this occurs, you may experience IP exhaustion errors, for example:
failed to allocate for range 0: no IP addresses available in range set
There are two ways to resolve this. You can either manually remove unused IPs from that directory or drain the node, run rke2-killall.sh, start the rke2 systemd service and uncordon the node. If you need to undertake any of these actions, please report the problem via GitHub, making sure to specify how it was triggered.
Ingress in CIS Mode
By default, when RKE2 is run with a CIS profile selected by the
profile parameter, it applies network policies that can be restrictive for ingress. This, coupled with the
rke2-ingress-nginx chart having
hostNetwork: false by default, requires users to set network policies of their own to allow access to the ingress URLs. Below is an example networkpolicy that allows ingress to any workload in the namespace it is applied in. See https://kubernetes.io/docs/concepts/services-networking/network-policies/ for more configuration options.
For more information, refer to comments on https://github.com/rancher/rke2/issues/3195.
Upgrading Hardened Clusters from v1.24.x to v1.25.x
Kubernetes removed PodSecurityPolicy from v1.25 in favor of Pod Security Standards. You can read more about PSS in the upstream documentation. For RKE2, there are some manual steps that must be taken if the
profile flag has been set on the nodes.
- On all nodes, update the
cis-1.23, but do not restart or upgrade RKE2 yet.
- Perform the upgrade as normal. If using Automated Upgrades, ensure that the namespace where the
system-upgrade-controllerpod is running in is setup to be privileged in accordance with the Pod Security levels:
# This value must be privileged for the controller to run successfully.
# We are setting these to our _desired_ `enforce` level, but note that these below values can be any of the available options.