Example

Create the inference code

To set up online inference, follow these steps and create a new file named predict.py. This script is responsible for:

1. Receiving HTTP POST requests on any path.
2. Reading an image from the POST request.
3. Loading the trained model into memory.
4. Returning the prediction.

from fastapi import FastAPI, File, UploadFile
import tensorflow as tf
import numpy
from PIL import Image
from io import BytesIO

app = FastAPI()

model_path = 'model.h5'
loaded_model = None

@app.post("{full_path:path}")
async def predict(image: UploadFile = File(...)):
    img = Image.open(BytesIO(await image.read()))

    # Resize image and convert to grayscale
    img = img.resize((28, 28)).convert('L')
    img_array = numpy.array(img)

    image_data = numpy.reshape(img_array, (1, 28, 28))

    global loaded_model
    # Check if model is already loaded

    if not loaded_model:
        loaded_model = tf.keras.models.load_model(model_path)

    # Predict with the model
    prediction = loaded_model.predict_classes(image_data)

    return f'Predicted_Digit: {prediction[0]}'

Test locally

You can test the web application locally:

pip install tensorflow==2.5.1 fastapi Pillow python-multipart

uvicorn --debug --reload predict:app

Define an endpoint in valohai.yaml

Add the following endpoint definition in your valohai.yaml:

- endpoint:
    name: digits
    description: predict digits from image inputs
    image: tiangolo/uvicorn-gunicorn-fastapi:python3.7
    server-command: uvicorn predict:app --host 0.0.0.0 --port 8000
    files:
      - name: model
        description: Model output file from TensorFlow
        path: model.h5

Installing Uvicorn in Your Deployment

If your base image doesn’t include Uvicorn, you can install it during the Valohai deployment creation process. Ensure that you have a requirements file named requirements-deployment.txt for your deployment’s dependencies. This file should include Uvicorn and any other necessary packages.

Afterward, update the server-command in your deployment configuration to point to the Uvicorn installation path, such as server-command: ~/.local/bin/uvicorn. This step is crucial to avoid errors related to Uvicorn not being found during deployment.

Add additional requirements

You can add any packages that are not included in the Docker image to the requirements-deployment.txt.

valohai-utils
tensorflow==2.5.1
Pillow
python-multipart

Push your changes to Git

Now push your changes to Git.

git add valohai.yaml
git add predict.py
git commit -m "digit prediction deployment endpoint"
git push

Create a new deployment

• Open your project
• Click on the Fetch repository button to fetch a new commit, if you’ve connected your project to a Git-repository
• Click on your Project’s Deployment tab
• Click on the Create deployment button
• Name your deployment and select where the endpoint will be hosted (by default Valohai.Cloud)
• Click on Create version
• Choose the digits endpoint and select a model.h5 you’ve trained previously.
• Click on Create version

When the status becomes 100% - Available, you can start using your endpoint.

Test endpoint

You can test your deployment endpoints directly from the Valohai web app.

• Open your project
• Click on your Project’s Deployment tab
• Select an existing deployment
• Click on the Test deployment button
• Select your endpoint from the drop-down
• Add a new called image and change the type to File
• Choose an image from your machine (you can use the one below)

Click on the Send request button.