When creating a pipeline, you can manually select a previously completed execution to use as a node in the new pipeline. This works regardless of whether the past execution was part of a pipeline or a standalone execution. This feature is particularly useful when you want to skip running a node by reusing a prior execution’s results, thus saving both time and computational resources.
You can enable this feature either through the user interface (UI) or programmatically via APIs.
Reusing Executions via the Web Application UI
To reuse a previous execution for a pipeline node:
- In the pipeline graph, select the node where you want to reuse a past execution.
- Use the
Reuse past execution
dropdown menu to select the execution you want to reuse.
Programmatically Reusing Executions
How to use the Valohai APIs
If you’re new to using the Valohai APIs, you can find more information here: Make calls to the Valohai API
You can send reuse_execution_id
in the payload while creating a pipeline instead of a node template. The pipeline would then re-use the past execution as a node execution.
The following example illustrates how to use the pipeline creation API (/API/v0/pipelines/) to create a pipeline that uses previous executions by supplying their IDs.
import os
import valohai
import json
import requests
pipeline_creation_payload = {
"edges": [
{
"source_node": "generate-data",
"source_key": "output.txt",
"source_type": "output",
"target_node": "add-stars",
"target_type": "input",
"target_key": "all_inputs"
},
{
"source_node": "add-stars",
"source_key": "output.txt",
"source_type": "output",
"target_node": "add-random-numbers",
"target_type": "input",
"target_key": "all_inputs"
}
],
"nodes": [
{
"name": "generate-data",
"type": "execution",
"reuse_execution_id": <past_execution_id>,
"on_error": "stop-all"
},
{
"name": "add-stars",
"type": "execution",
"reuse_execution_id": <past_execution_id>,
"on_error": "stop-all"
},
{
"name": "add-random-numbers",
"type": "execution",
"template": {
"environment": <environment_id>,
"commit": <commid_identifier>,
"step": "add-random-numbers",
"image": "python",
"command": "pip install -r requirements.txt\npython ./random_numbers.py {parameters}",
"inputs": {
"all_inputs": []
},
"parameters": {},
"runtime_config": {},
"inherit_environment_variables": true,
"environment_variable_groups": [],
"tags": [],
"time_limit": 0,
"environment_variables": {},
"allow_reuse": false
},
"on_error": "stop-all"
}
],
"project": <project_id>,
"tags": [],
"parameters": {},
"title": "test-reuse-pipeline"
}
response = requests.post(
f'https://app.valohai.com/api/v0/pipelines/',
json=pipeline_creation_payload,
headers={
'Authorization': 'Token ' + os.getenv('VH_TOKEN'),
'Content-Type': 'application/json'
}
)
print(response.status_code)