Kubernetes for real-time inference

Valohai can push deployments to an existing Kubernetes cluster.

Valohai uses standard Kubernetes APIs to communicate with your Kubernetes cluster, and app.valohai.com (34.248.245.191) should be able to access your cluster’s API Server over HTTPS.

Your cluster can be configured to serve only private deployment endpoints.

Setup instructions

Follow the steps below to configure your cluster for Valohai real-time deployments.

Install ingress-nginx on the cluster

Instructions: https://kubernetes.github.io/ingress-nginx/deploy/

Get the external IP of your ingress-nginx

You’ll need to share this with Valohai.

kubectl -n ingress-nginx get service/ingress-nginx-controller

Create a Kubernetes service account that Valohai will use

kubectl create serviceaccount valohai-deployment

Create a service account token

Note that service account tokens are not created automatically for Kubernetes 1.22 and higher.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: valohai-deployment-token
  namespace: <NAMESPACE HERE>
  annotations:
    kubernetes.io/service-account.name: valohai-deployment
EOF

Find the token name

You’ll need to provide this token back to Valohai.

kubectl get serviceaccounts valohai-deployment -o json
kubectl get secret valohai-deployment-token -o jsonpath='{.data.token}' | base64 --decode

Set up the valohai-deployment-role in Kubernetes

Create a new file valohai-deployment-role.yml with the contents specified below. If you need to limit access to a certain namespace, you can add namespace: <NAMESPACE> under metadata.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: valohai-deployment-role
rules:
  - apiGroups: [""]
    resources: ["events", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods", "pods/log", "services"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["apps", "extensions"]
    resources: ["deployments", "deployments/rollback", "deployments/scale"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["extensions", "networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]

Apply the role with:

kubectl apply -f valohai-deployment-role.yml

Create a rolebinding

kubectl create rolebinding valohai-deployment-binding --role=valohai-deployment-role --serviceaccount=<namespace>:valohai-deployment

Repository access

Make sure your cluster’s nodes can pull from the repository that Valohai is pushing images to.

User Account

This user is required so Valohai can access the cluster and deploy new images to your ECR.

• Create a IAM user valohai-eks-user in your AWS Console.
• Enable Programmatic access and Console access.
• Attach the following existing policies:
  • AmazonEC2ContainerRegistryFullAccess
  • AmazonEKSServicePolicy
• Create a new policy named VH_EKS_USER with the following JSON:

{
    "Version": "2012-10-17",
    "Statement": [
      {
          "Sid": "1",
          "Effect": "Allow",
          "Action": "eks:ListClusters",
          "Resource": "*"
      }
  ]
}

• Back in your Add user tab, click on the refresh button and select the VH_EKS_USER policy.
• Store the access key & secret in a safe place.

Other

You can use standard Docker login (username/password) credentials when pushing to Azure Container Registry, GitLab, Artifactory, Docker Hub, and others. Make sure you create a separate account for Valohai to be able to push to your repository.

Conclusion

You should now have the following values:

• Details of the created cluster. Find these on the cluster’s page on EKS
  • Cluster name
  • AWS region of the cluster
  • API server endpoint
  • Cluster ARN
  • Certificate authority (cluster-certificate-data)
• External IP of the Load Balancer tied to the NGINX Ingress Controller (kubectl get service/ingress-nginx-controller)
• valohai-deployment service accounts token
• If you have an ALB that has a well-trusted cert and points to the Kubernetes API, you’ll need to just provide the ALB address.
• ECR name - Copy the URL you see when creating a new repository in your ECR (for example accountid.dkr.ecr.eu-west-1.amazonaws.com)
• valohai-eks-user access key ID and