Top 5 Q&A Nuggets: Pipeline in the pipeline, Metadata and More 💼🌐🚀"

Question 1: Hey, I ran into an issue. When I run the executions from the terminal using commands like 'vh execution run train --adhoc,' Valohai generate a name for my execution like 'ch'?

I didn’t set any name, and I'm curious about how this happens.

Answer:

👨‍💻 Let's delve into those flags for 'vh execution run': '--watch,' '--adhoc,' and '--title.' Now, each of these has its shortform counterparts: '--watch' can be shortened to '-wa,' '--adhoc' to '-a,' and '--title' to '-t.'

In the scenario you mentioned, you ran 'vh execution run train --adhoc -watch.' What happened here is that Valohai interpreted the typoed '-watch' as if it were using the shortform format. It took '-wa' as '--watch' and 't' as '--title.' So, 'ch' magically became the title of the execution because Valohai used these components creatively to generate the execution name. 😄👀

It's almost like Valohai has a secret sense of humor – turning typos into comedy gold! 🤣

Question 2: Can we call a pipeline from a pipeline?

Answer:
Hey there! 👋 Absolutely, it's possible! To make this happen, you'll want to kick things off with an API call that starts a new pipeline right from your current one. Check out this example of how you can create a pipeline using the API: Link to Example.

Now, when it comes to making that API call, you've got a couple of options to choose from:

1. You can go with the PipelineCreateFromTemplate method. It's a breeze to use, but keep in mind, it runs the new pipeline with all the default settings, so you can't tweak the inputs or parameters.
2. Or, you can fire up an API call directly from the Valohai UI.

To make all these API calls play nice, you'll need to set up an environment variable with your Valohai API token. It's as easy as pie:

• Go on over to your profile ➡ Authentication ➡ Manage Tokens ➡ Give birth to a shiny new token ➡ Don't forget to save it! 🙌

Question 3: Hello Valohai. Can I combine inputs from different sources under one input in my valohai.yaml?

Answer:
Hey there! Absolutely, you can do that!

To achieve it, simply define your inputs in the valohai.yaml like this:

- name: dataset
      default:
      - s3://valohai-academy-files/tutorials/valohai101/2020.csv
      - azure://valohaiacademyfiles/tutorials/valohai101/2021.csv
      optional: false

You can also set it from the UI when creating the execution add the inputs there.

Question 4: Can I create aliases programmatically, or is it only possible through the UI?

Answer:
Yes, you can do that!.

The documentation related to alias creation can be found here.

In a nutshellyou will need to create a .metadata.json sidecar file that you attach to your output files. E.g. if your output file is mymodel.h5 you would do something like this

# Create a sidecar file for Valohai
# Tell Valohai to update the dataset-a-latest alias to point to this file
metadata = {
    "valohai.alias": "model-latest", # creates or updates a Valohai data alias to point to this output file
}

metadata_path = valohai.outputs().path('mymodel.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
    json.dump(metadata, outfile)

Question 5: Hello 👋 Is it possible to retrieve the pipeline node from execution data, the same way it is possible to retrieve the pipeline from one of its node?

Thanks!

Answer:

Hey there! Sure thing! Here's how you can do it:

In the UI: Just click on the '=<pipeline-id>' in the execution table to open the pipeline. You can also filter the executions table based on the pipeline id.

With the API: The ExecutionRetrieve endpoint provides information about the pipeline that the execution was a part of. Use this: https://app.valohai.com/api/v0/executions/<exec-id>/ 😊👍