Top 5 Q&A Nuggets on API, Pipelines, and More! 🌐🔍

Hey team! 🌟 I've compiled a list of top 5 questions to foster some collective wisdom-sharing and enhance our knowledge pool. Let's dive into these queries together and unravel some valuable insights! 🚀 Feel free to chime in and share your expertise. Let the knowledge-sharing party begin! 🎉💡

Question 1: Hello team, I've been experimenting with various environments and observed that the execution was in the queue for over 40 minutes without running. Can you provide insights into the possible reasons for this? Thank you!

Oh, the mysteries of the queue! 😅

There could be a couple of reasons for this.

Maybe cloud platform is taking a little breather and the machine is not available at that moment, or you might need a bit more quota for that machine.

Don't worry, though!

The organisation admin can play detective by checking for any messages on why the machine isn't feeling like scaling up. Just head to Manage <Organization> -> Environments -> Pick the Environment -> Give the 'Status' a check. 🕵️‍♂️✨

Question 2: Can we launch a pipeline with the API using the default parameters/ inputs (and therefore not specify them in the payload again)? Thanks!

For sure! There's an API endpoint called PipelineCreateFromTemplate, and it does just what you're looking for. It creates the pipeline based on the default values, saving you from the hassle of specifying parameters or inputs in the payload. 🚀

However, it's important to note that with the PipelineCreateFromTemplate endpoint, you can't modify those parameters or input values directly through this call. If you find yourself needing to tweak them, you'll have to make another API call. Rest assured, we're considering creating an option to address this concern! 😅

So, for the PipelineCreateFromTemplate endpoint, you only need to include these in your payload:

{ "project": "<project-uuid>", "commit": "<commit-hash>", "name": "<pipeline-name-from-valohai.yaml>"}.

Everything else gets its magic from the valohai.yaml and the default values set there. No need to stress about defining parameter or input values when using this endpoint! 🎉

Question 3: Hey team! Quick question: Could you refresh my memory on the CLEAR CACHE environment variable in VLH? Is it CLEAR_CACHE=1?

Sure thing! You might want to use VH_CLEAN, it actually has a few buddies. 🤓 Check out the different env variables here:

• VH_NO_DATA_CACHE: Ignores pre-existing cached data.
• VH_NO_IMAGE_CACHE: Forces a fresh pull of Docker images from the source.
• VH_CLEAN: Takes charge of cleaning Docker images and cached data before and after execution.
• VH_TMPFS: Manages whether to write /tmp operations on disk or in memory (set to false for disk).
• VH_SHM_SIZE: Boosts the container's shared memory directory size. 🚀

Question 4: Hey there! Is it possible to retrieve the execution from the S3 URI of an execution output file?

Absolutely! You can fetch all the outputs by sending a GET request to https://app.valohai.com/api/v0/data/. Once you get the results, you can search for the specific S3 URI you're interested in.

If you happen to know the project, you can narrow down the search by adding ?project=<project-id>, so you use

https://app.valohai.com/api/v0/data/?project={project_id}

By the way, if you've got the datum-id handy, you can directly grab the information using https://app.valohai.com/api/v0/data/ 🚀

https://app.valohai.com/api/v0/data/<datum-id>. 

Question 5: Hey there! 🌞 I need to trigger a valohai pipeline with a different data input once per day. I don’t want to do this manually by updating the valohai.yml file manually. Is my best option to use the Valohai API, or is there another approach I can consider?

The API might be your best option.

While that might be your smoothest ride, if your new data conveniently shows up daily in the same folder of your S3 bucket (consistent path), you're in luck! Set up a trigger in Valohai for an automated daily execution.

Here's your itinerary: Project 'Settings' -> Triggers -> Create Trigger -> Conditions (Choose Scheduled/Cron) -> Set it to Daily -> Actions (Select 'Run Execution' or 'Run Pipeline').

But if your data likes to play hide and seek, API calls with new parameters for the data path every day is your trusty alternative. Happy automating! 🤖✨