Where can I find pipeline errors?
A pipeline can fail either because of a node inside a pipeline has completed on an error or because the pipeline definition has caused errors.
Each node in a pipeline is an execution, Task, or deployment. You can either click on the pipeline graph to choose any of the nodes or view their logs for any errors.
You can find pipeline-level errors under the Log tab in the pipeline view.
- Node “name” transitioned to crashed: One of your executions has failed. Check the pipelines executions for an execution that has been completed on an error and check its logs.
- stopping due to 1 incompletable edges: The node couldn’t start because it did not have all the inputs that are required for that step. Review your pipeline edges and make sure you’re passing all the right files.
How can I override default parameters in a pipeline?
You can override the default parameters of a node inside a pipeline using the override attribute.
In the example below we’re creating a pipeline that runs a preprocessing step and three training steps.
The three training nodes are overriding either the default input data, the parameters, or both.
- pipeline:
name: Three-Trainings Pipeline
nodes:
- name: preprocess
type: execution
step: preprocess-dataset
- name: train1
type: execution
step: train-model
override:
inputs:
- name: dataset
- name: train2
type: execution
step: train-model
override:
inputs:
- name: dataset
parameters:
- name: epochs
default: 10
type: integer
- name: learning_rate
default: 0.001
type: float
- name: train3
type: execution
step: train-model
override:
inputs:
- name: dataset
parameters:
- name: epochs
default: 15
type: integer
- name: learning_rate
default: 0.001
type: float
edges:
- [preprocess.output.preprocessed_mnist.npz, train1.input.dataset]
- [preprocess.output.preprocessed_mnist.npz, train2.input.dataset]
- [preprocess.output.preprocessed_mnist.npz, train3.input.dataset]
Troubleshooting a valohai.yaml Parsing Error
When running vh lint on the command line in the local repository, the following error is encountered:
/home/your-user/code/project/valohai.yaml: 1 errors, 0 warnings
error: could not parse YAML: PipelineParameter.__init__() missing 1 required keyword-only argument: 'targets'
How to Fix It:
Check Your valohai.yaml File
Ensure all pipeline parameters have the targets defined properly and double-check your file’s format. YAML files are sensitive to spacing and alignment, and even small mistakes can cause errors.
Look Over Recent Changes
If you’re encountering issues, review any recent changes to the valohai.yaml file that might have been overlooked for mistakes.
Set Up a Check Before Merging Changes
Consider implementing a pre-merge check for the valohai.yaml file using a custom GitHub action. This can help catch errors or inconsistencies before changes are merged. For guidance on implementing a GitHub Action, refer to our documentation here.
Get More Help
If these steps don’t resolve the problem, don’t hesitate to contact our support team for additional assistance.
These steps should help you address any issues with your valohai.yaml file and keep your project running smoothly.
Troubleshooting Pipeline Step Failure and Completion Inconsistency
Sometimes, a step in your pipeline might fail but still appear as Completed. This can be confusing and requires checks to understand and fix the issue.
How to Fix It:
Check the Step Configuration
Closely inspect the failing step’s settings and check for any setup mistakes or missing requirements.
Review the Commands
Examine the commands listed for that step to ensure each is correct and executed in the proper order. By default, the pipeline runs all commands in sequence, even if one fails. To prevent this, add set -e at the start of the step.command list, which forces the step to stop at the first error.
Look at Logs and Outputs
Analyze the step’s logs and outputs to better understand what’s going wrong. Carefully consider the errors and outputs from each command to pinpoint the underlying problem.
Use Documentation and Community Help
Consult the documentation or seek assistance in community forums at docs.valohai.com. You can also reach out to us with any questions at support@valohai.com.
Test Changes Safely
Always make and test changes in a safe environment rather than your primary production setup. This approach allows you to address potential issues without impacting your main projects.