Kubernetes for real-time inference

Valohai can push deployments to an existing Kubernetes cluster.

Valohai uses standard Kubernetes APIs to communicate with your Kubernetes cluster, and app.valohai.com (34.248.245.191) should be able to access your cluster’s API Server over HTTPS.

Your cluster can be configured to serve only private deployment endpoints.

Setup instructions

Follow the steps below to configure your cluster for Valohai real-time deployments.

Install ingress-nginx

Install ingress-nginx on the cluster:

https://kubernetes.github.io/ingress-nginx/deploy/

Get the external IP of your ingress-nginx. You’ll need to share this with Valohai.

kubectl -n ingress-nginx get service/ingress-nginx-controller

By default, all of your Valohai endpoints will be accessible from the public internet. You can patch the nginx ConfigMap to whitelist only certain IPs so only they can access the endpoints served by Valohai.

kubectl patch -n ingress-nginx configmap/ingress-nginx-controller --type merge -p '{"data":{"whitelist-source-range": "84.251.7.123/32,84.251.7.124/32"}}'

Kubernetes Service Account

Create a Kubernetes service account that Valohai will use:

kubectl create serviceaccount valohai-deployment

Create a service account token. Note that tokens are not created automatically for Kubernetes 1.22 and higher.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: valohai-deployment-token
  namespace: <NAMESPACE HERE>
  annotations:
    kubernetes.io/service-account.name: valohai-deployment
EOF

Get the token, you’ll need to provide this token back to Valohai.

kubectl get secret valohai-deployment-token -o jsonpath='{.data.token}' | base64 --decode

This will return you the secret name (e.g., valohai-deployment-token-SUFFIX). Use the name to fetch the secret value:

kubectl get secret SECRET-NAME -o json

Setup the valohai-metadata-role in Kubernetes. If you want to limit access to a specific namespace, define it below, otherwise leave it empty.

Create a new file valohai-deployment-role.yml with the following contents:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: valohai-deployment-role
  namespace: <IF THERE IS A NAMESPACE>
rules:
  - apiGroups: [""]
    resources: ["events", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods", "services"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["apps", "extensions"]
    resources: ["deployments", "deployments/rollback", "deployments/scale"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]

Apply the role with:

kubectl apply -f valohai-deployment-role.yml

Create a rolebinding. Replace “default” with your namespace if you defined one when creating your service account.

kubectl create rolebinding valohai-deployment-binding \
    --role=valohai-deployment-role \
    --serviceaccount=default:valohai-deployment

Make sure your cluster’s nodes can pull from the repository that Valohai is pushing images to.

Google Cloud Service Account

The valohai-sa-deployments service account is used by Valohai to manage deployments and images in your Container Registry or Artifact Registry.

In your GCP Project, go to IAM -> Service Accounts and create a new service account for Valohai:

• Type: Service Account
• Name: valohai-sa-deployments
• Role:
  • Service Account Token Creator
  • Storage Admin
  • Kubernetes Engine Developer
• Create Key: JSON

Download the JSON key, as you’ll need to share it with Valohai later.

Other

You can use standard Docker login (username/password) credentials when pushing to Azure Container Registry, GitLab, Artifactory, Docker Hub, and others.

Make sure you create a separate account for Valohai to be able to push to your repository.

Conclusion

You should now have the following values:

• Cluster name
• valohai-deployment service accounts token
• External IP of ingress-nginx (kubectl -n ingress-nginx get service/ingress-nginx-controller)
• Cluster API address and the cluster-certificate-data

If you have an ALB that has a well-trusted cert and points to the Kubernetes API, you’ll need to just provide the ALB address.

Share this information with your Valohai contact using the Vault credentials provided to you.