Live Outputs¶

You use live outputs to upload files mid-execution.

Normally, executions will upload all files stored at /valohai/outputs at the end of the execution, even if the code crashes or is stopped. But in some long running workloads, you might want to save checkpoints or other artifacts mid-execution.

When Valohai detects that a file under /valohai/outputs is marked as read-only, Valohai will remove that file from the directory and upload it right away. All programming languages and shells have a way for marking files read-only e.g. os.chmod in Python.

A simple example:

echo hello >> /valohai/outputs/greeting.txt
chmod 0444 /valohai/outputs/greeting.txt
sleep 30
echo bye >> /valohai/outputs/farewell.txt
# => Generates 2 files:
#    - 'greeting.txt' with 'hello', uploaded right away
#    - 'farewell.txt' with 'bye', uploaded after the execution finishes

Note

If the file permissions change commands are quick, they might not register before the code continues its progress. Because of this, we advice not to generate multiple live outputs with exactly the same name.

That can lead to some funky behaviour.