Kubernetes Multihomed Pods

For those of your who need more than one network interface for your PODs/Containers in Kubernetes. This document till go over a new extensibility framework called Multus, and a brief example of how I set it up.

What is Multus? Multus is a CNI network plugin for Kubernetes. It allows for attaching multiple network interfaces to pods in Kubernetes. There are alot of good reasons why you would want multiple networks interfaces for your pods, and not alot of good reasons why you shouldn't. Thanks for Intel for open sourcing this great technology!

In the following example, I'll go over a simple multus configuration to add an additional interface (called net1) to your PODs. For this you will need to create the following YAML files:

A simple Multus Setup

They very first thing we need to create is an adminpod. This will be used for customizing our kubernetes hosts that have been selected to run Multus. Essentially, we're dropping in a privledge daemonset pod, that will make modification to the underlying host configuration for Multus, then we'll remove it later. This will save you a few steps for manual changes, and it's a pretty cool thing to do! adminpod.yaml

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: adminpod-ds
  namespace: kube-system
  labels:
    tier: node
    app: adminpod
spec:
  template:
    metadata:
      labels:
        tier: node
        app: adminpod
    spec:
      hostNetwork: true
      nodeSelector:
        role: gateway
      tolerations:
      - operator: Exists
        effect: NoSchedule
      volumes:
      - name: hostdoor
        hostPath:
          path: /
          type: Directory
      containers:
      - name: adminpod
        image: docker:5000/admincontainer:1.0.0.1
        volumeMounts:
        - mountPath: /hostdoor
          name: hostdoor
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        command: ["/bin/bash", "-c", "sleep 2000000000000"]
        securityContext:
          privileged: true

multus-daemonset.yaml

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  version: v1
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            config:
                 type: string
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: multus
rules:
  - apiGroups: ["k8s.cni.cncf.io"]
    resources:
      - '*'
    verbs:
      - '*'
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: multus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
- kind: ServiceAccount
  name: multus
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus
  namespace: kube-system
---
# ------------------------------------------------------
# Currently unused!
# If you wish to customize, mount this in the
# daemonset @ /usr/src/multus-cni/images/70-multus.conf
# ------------------------------------------------------
kind: ConfigMap
apiVersion: v1
metadata:
  name: multus-cni-config
  namespace: kube-system
  labels:
    tier: node
    app: multus
data:
  cni-conf.json: |
    {
      "name": "multus-cni-network",
      "type": "multus",
      "delegates": [
        {
          "type": "weave-net",
          "name": "weave-net.1",
          "delegate": {
            "isDefaultGateway": true
          }
        }
      ],
      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
    }
# -------------- for openshift.
# "delegates": [{
#   "type": "openshift-sdn",
#   "name:" "openshift.1",
#   "masterplugin": true
# }],
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-multus-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: multus
spec:
  template:
    metadata:
      labels:
        tier: node
        app: multus
    spec:
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: multus
      containers:
      - name: kube-multus
        image: nfvpe/multus:latest
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cni
          mountPath: /host/etc/cni/net.d
        - name: cnibin
          mountPath: /host/opt/cni/bin
      volumes:
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /opt/cni/bin
        - name: multus-cfg
          configMap:
            name: multus-cni-config

network-crd.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  version: v1
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            config:
                 type: string

Next we need to setup the network mapping for our new interface. I'm mapping my hosts eth2 interfaces to net1 of the POD. The network assignment I choose is 10.40.40.0/24 which is another private network. macvlan-conf.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth2",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "10.40.40.0/24",
        "rangeStart": "10.40.40.1",
        "rangeEnd": "10.40.40.100",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": ""
      }
    }'

Apply the following YAML configurations to K8S.

kubectl apply -f adminpod.yaml 
kubectl apply -f multus-daemonset.yml
kubectl apply -f network-crd.yaml
kubectl apply -f macvlan-conf.yaml 

Update the hosts you wish to deploy Multus on with this custom CNI configuration. I'm not running flannel, because I like weave-better. You'll need to make sure you are using weave-net.

MCONFIG_FILE=/hostdoor/etc/cni/net.d/1-multus-cni.conf
echo "==> Deploying Weave Configuration for Multus backed Nodes"
json=$'{\n    name: node-cni-network,\n    type: multus,\n    kubeconfig: /etc/cni/net.d/multus.d/multus.kubeconfig,\n    delegates: [{ \n        type: weave-net,\n        hairpinMode: true,\n        masterplugin: true }]\n}\n'

for pod in `kubectl get pod -n kube-system |grep adminpod|awk '{print $1}'`; do
    echo "==> Admin POD: ${pod}"
    kubectl -n kube-system exec ${pod} rm ${MCONFIG_FILE}
    kubectl -n kube-system exec ${pod} touch ${MCONFIG_FILE}
    echo "$json"
    kubectl -n kube-system exec ${pod} -t -- bash -c \
       "echo '{' > ${MCONFIG_FILE}; \
        echo '    \"name\": \"node-cni-network\",' >> ${MCONFIG_FILE}; \
        echo '    \"type\": \"multus\",' >> ${MCONFIG_FILE}; \
        echo '    \"kubeconfig\": \"/etc/cni/net.d/multus.d/multus.kubeconfig\",' >> ${MCONFIG_FILE}; \
        echo '    \"delegates\": [{' >> ${MCONFIG_FILE}; \
        echo '            \"type\": \"weave-net\",' >> ${MCONFIG_FILE}; \
        echo '            \"hairpinMode\": true,'  >> ${MCONFIG_FILE}; \
        echo '            \"masterplugin\": true' >> ${MCONFIG_FILE}; \
        echo '    }]' >> ${MCONFIG_FILE}; \
        echo '}' >> ${MCONFIG_FILE};"
 done

Once this is complete, you should be able to deploy a POD on those hosts running Multus, and they should have a secondary network interface called (net1). Validation

kubectl exec pod -i -t -- bash -il -c ifconfig -a

eth0: flags=4163  mtu 1376
        inet 10.37.0.7  netmask 255.240.0.0  broadcast 10.47.255.255
        inet6 fe80::8cfd:edff:fe83:d575  prefixlen 64  scopeid 0x20
        ether 8e:fd:ed:83:d5:75  txqueuelen 0  (Ethernet)
        RX packets 26493  bytes 8412636 (8.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55418  bytes 11906216 (11.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 711  bytes 163976 (160.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 711  bytes 163976 (160.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net1: flags=4163  mtu 1500
        inet 10.40.40.101  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::8e5:cff:fe57:122e  prefixlen 64  scopeid 0x20
        ether 0a:e5:0c:57:12:2e  txqueuelen 0  (Ethernet)
        RX packets 34138  bytes 8351448 (7.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33548  bytes 8151748 (7.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

---

• ← Previous Post