How to install IoT Edge on Kubernetes

IoT Edge can be installed on Kubernetes by using KubeVirt technology. KubeVirt is an open source, Cloud Native Computing Foundation (CNCF) project that offers a Kubernetes virtualization API and runtime to define and manage virtual machines.

Architecture

Note Description 1 Install KubeVirt Custom Resource Definitions (CRDs) into the Kubernetes cluster. Like the Kubernetes cluster, management and updates to KubeVirt components are outside the purview of IoT Edge. 2️ A KubeVirt VirtualMachine custom resource is used to define a Virtual Machine with required resources and base operating system. A running instance of this resource is created in a Kubernetes Pod using KVM and QEMU technologies. If your Kubernetes node itself is a Virtual Machine, you'll need to enable Nested Virtualization to use KubeVirt. 3️ The environment inside the QEMU container is just like an OS environment. IoT Edge and its dependencies (like the Docker container engine) can be setup using standard installation instructions or a cloud-init script.

Deploying Azure IoT Edge workloads on Kubernetes

Reference from microsoft :- https://learn.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-kubernetes?view=iotedge-1.4

Setup steps As needed, follow the steps to register an IoT Edge device. Take note of the device connection string.

Set up VS Code and tools, associate with IoT Hub from the previous step.

Follow steps, or a subset as needed, to install edge deployment into the cluster.

For simplicity, this tutorial doesn't specify a persistent store for iotedged during install. However, for any serious/PoC deployment, follow the best practice example shown in step 6 of iotedged failure resilience tutorial.

Create K8s namespace

kubectl create ns resources

Install IoT Edge CRD, if not already installed

helm install --repo https://edgek8s.blob.core.windows.net/staging edge-crd edge-kubernetes-crd

Store the device connection string a variable

export connStr=replace-with-device-connection-string-from-step-1

Install the edge workload into the cluster namespace

helm install --repo https://edgek8s.blob.core.windows.net/staging resources-example edge-kubernetes \
  --namespace resources \
  --set "provisioning.deviceConnectionString=$connStr"

In the Visual Studio Code command palette (View menu -> Command Palette...), search for and select Azure IoT Edge: New IoT Edge Solution. Follow the prompts and use the following values to create your solution:

Field Value Select folder Choose the location on your development machine for VS Code to create the solution files. Provide a solution name Enter a descriptive name for your solution or accept the default EdgeSolution. Select module template Choose Empty solution. You'll be making updates to deployment.template.json (see navigation pane on the left) to configure the edgeHub module to use K8s configmaps.

Add the Kubernetes resources in the createOptions section of the edgeHub module in deployment.template.json using Kubernetes extended createOptions feature.

{
  "$schema-template": "2.0.0",
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {}
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
              "createOptions": {}
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
              "createOptions": {
                "HostConfig": {
                  "PortBindings": {
                    "5671/tcp": [{
                      "HostPort": "5671"
                    }],
                    "8883/tcp": [{
                      "HostPort": "8883"
                    }],
                    "443/tcp": [{
                      "HostPort": "443"
                    }]
                  }
                },
+               "k8s-experimental": {
+                 "resources": {
+                   "limits": {
+                     "memory": "128Mi",
+                     "cpu": "500m",
+                     "hardware-vendor.example/foo": 2
+                   },
+                   "requests": {
+                     "memory": "64Mi",
+                     "cpu": "250m",
+                     "hardware-vendor.example/foo": 2
+                   }
+                 }
+               }
              }
            }
          }
        },
        "modules": {}
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {},
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}

Resource requirements API reference has details on allowed values.

🗒

We've used edgeHub as an example here, however you can specify K8s extended createOptions for any module in the edge deployment.

Generate the workload deployment config by right-clicking the deployment.template.json in the left navigation pane and selecting Generate IoT Edge Deployment Manifest. This will generate the minified deployment.amd64.json under the config directory.

Update the configuration for the device by right-clicking deployment.amd64.json and selecting Create Deployment for Single Device. In the displayed list, choose the device created in step 1 to complete the operation.

In a few seconds, you'll see a new edgeHub pod instantiated with the resources defined deployment manifest.

Get pod names

kubectl get pods -n resources

Save edgehub pod name in env var

export ehname=replace-with-edgehub-pod-name

Describe pod spec to see resource requests

kubectl describe pod --namespace=resources $ehname

Cleanup

Cleanup

helm del resources-example -n resources &&
kubectl delete ns resources

Reference :- https://microsoft.github.io/iotedge-k8s-doc/examples/resources.html