valohai.yaml configuration file¶
Valohai is completely technology agnostic. You can develop in notebooks or scripts in a language and framework of your choice.
Valohai expects that you have a valohai.yaml config file in the root of your repository. The file defines what kind of jobs can be executed inside your project, and the different properties of each job type (these are called steps in Valohai).
You can either write the
valohai.yaml by hand or if you’re using Python you can use the valohai-utils Toolkit to define the configuration file in your code.
Let’s start by bringing just a single job type to Valohai.
Open your existing Python file and define a Valohai step inside it:
import valohai # Prepares a new step called train in valohai.yaml # Execute this Python file to run every time the train step is launched in Valohai valohai.prepare(step="train")
You can now generate a
valohai.yaml configuration file that includes the
train step by running:
vh yaml step myfilename.py
This will create:
valohai.yamlfile with the
trainstep that will run:
pip install -r requirements.txt
your Python script.
If you have an existing
valohai-utilsline will be just added there.
By default, your step will use a standard Python Docker image and install everything from
requirements.txt during runtime.
You could also specify a different Docker image by passing the image name to
import valohai # Prepares a new step called train in valohai.yaml # Execute this Python file to run every time the train step is launched in Valohai # inside a GPU enabled Tensorflow 2.6.1 image. valohai.prepare(step="train", image="tensorflow/tensorflow:2.6.1-gpu")
and then rerun
vh yaml step to update your
vh yaml step myfilename.py
Create a new file called
valohai.yaml and define a Step inside it.
In our example below we’re defining two commands to be ran every time the step called
train is ran. You can remove the pip install command, if you don’t have a need for it.
- step: name: train image: tensorflow/tensorflow:2.6.1-gpu command: - pip install -r requirements.txt - python myfilename.py
You can now upload your local code to Valohai and run the
train step inside the Docker container you just defined.
vh execution run --adhoc train
--adhoc flag allows you to easily upload your local changes to Valohai without going through Git.
Ideally, you’d always upload your changes to Git but sometimes it’s just easier to test small changes using
The other option would be to go through git:
git add . git commit -m "Added Valohai valohai.yaml" git push # Fetch new commits to Valohai vh project fetch # Run a new execution based on the latest Git commit vh exec run train