Updates 2022

Patch Notes - 2022-12-22

New Features

Dataset version aliases can now be updated programmatically with output metadata.
Datasets can now also be created from task outputs.

UX Improvements

The datasets view is now available both in a project context and stand-alone, for a less jarring navigation experience.
Data browser views now support presets.
Triggers can now be deleted with the click of a button, instead of with the tippy-tap-tipety-tap of a programmer writing a script against our API.
We now prohibit creating useless require-approval pipeline actions.
Deleted executions no longer contribute to project property keys (metadata and parameters), so no more accuray littering your views forever!
You can now disable word-wrapping for event log views.
The execution table's title column's width is now stored (locally) so you don't have to resize it every time you navigate around.
You can now add a note (comment) to an execution or task right while creating one.
You can no longer rename projects that have active deployment versions, so as not to break every single URL pointing to them.

Performance Improvements

Executions and tasks should start a bit faster, since environment scaling is deferred.
We made the app small-talk a little less (like a good Finn should), which should make it a bit faster.
Environment scaling should occur more snappily all in all.

Bugfixes

When using wildcards and the suffix keep-directories mode, patterns such as foo/a*.jpg with foo/acat.jpg and foo/adog.jpg will no longer result in cat.jpg and dog.jpg, but acat.jpg and adog.jpg as expected.
Validation for datum alias names is now unified between the UI and the API. Oopsy...
You'll now actually know why you can't disable a default environment, not just get a dreaded "oopsy woopsy" message.
Event export CSVs that would contain null bytes no longer contain null bytes, which is useful as it makes it possible to actually export them. Nice!
Certain race conditions around execution finishing status propagation should be now gone. Atomicity is hard, but we're trying.

Patch Notes - 2022-11-09

New features 🚀

You can now elect for your Azure AD authenticated users to have accounts automatically created. Contact our support to enable this for your domain.
You can now copy Notebook Executions.
You can now restart a task's failed executions (via the API).
The data browser search bar now supports wildcards. * You can now tag pipelines after they have been created, too.
For the API users amogus, you can also now multi-tag executions, tasks, data, and pipelines.
You can now create environment variable groups for your organization, and use them via the API. The GUI will follow soon.

Performance improvements 🚄

API calls for executions that are running but have printed no metadata are now much faster.
In general, API calls that need to talk to queue machines are now much faster.

Bug fixes 🐞

Certain instance types' price histories didn't work. They now do.
The Team Access Grant editor now works again.
Pipeline execution nodes' steps' environment overrides are now properly taken into account in the UI.
Input parameter interpolation now works correctly in multi-batch tasks.
Certain requests for property keys could fail when there were none. No more.
Empty pipelines are now considered complete.

Patch Notes - 2022-10-19

Bugfix: Fix autorestart check attempting to read None stop reason
- Feature: Show execution metadata in datums table
- Feature: Auto-login to notebook executions
- Bugfix: More robust notebook execution token passing
- Bugfix: Show commits first 8 chars instead of the last in the project executions table

Patch Notes 2022-10-18

The pipeline list now also supports saved view presets.
- You can now build new dataset versions by copying extant versions.
- The dashboard view is a bit smarter.
- Organization-level environment variables are now propagated to executions and deployment versions when Inherit is on.

Performance improvements

We took a critical eye to the number of network roundtrips being made to check for execution statuses (hat tip to Sentry's new Performance features!) and got rid of some of them.
Input data change alerts are now disabled by default due to low usage.
The dashboard and the user menu load faster.

Bugfixes

Your browser won't suggest translating the dashboard's fun, quirky, multilingual greeting text anymore.
It's now easier to spot errors in the new fancy login screen.

Patch Notes 2022-10-06

New features

You're about to witness the strength of PROJECT PINK SHARK. Let us know if something looks off! We think it's pretty spiffy though.
You can now use dataset version creation metadata to also create new datasets. Just name a new dataset and thy will be done.
You can now set parameters on the pipeline level and propagate them to nodes.

Bugfixes

Certain highly concurrent situations should no longer cause even more highly concurrent situations.
More than 100 environments are now loaded in various environment-loading views.
The "This datum has been purged" error message you'd get when copying an execution will no longer cause an infinite loop of error messages.

Performance improvements

Pipelines with task-output edges will now be executed much faster.
Executions with lots of datum inputs were given a good ol' speed-up too.
The Project List API is faster with large projects.
Input and dataset ingestion is faster. You'll notice this as a small systemic speedup.

Patch Note 2022-09-28

We've had a good number of improvements.

We upgraded internal bits and pieces related to our deployment Kubernetes version support. The minimum Kubernetes cluster version is now 1.19.
You can now filter executions by date.
Scatter plots' points are now scaled to a sane linear range.
Datum listing APIs should be much faster now.
Something's brewing in notebook world. Stay tuned!
Project Pink Shark is still brewing too, but you've maybe seen hints of it here on the flipside.
Problems creating dataset versions via executions are now shown as execution alerts.

Patch Note 2022-09-15

Bugfixes & Performance Improvements

Commits from external stores can now be downloaded and you don't get a cryptic "empty response" protocol error.
Restarted executions' inputs no longer get a duplicate_restartprefix.
The dashboard's project popovers appear where they should, not elsewhere.
The execution outputs list is now much, much faster, because we're not fetching data that nobody needs.

Patch Note 2022-09-07

New features

You can now set a root path for each Store. This lets you use a shared storage account, bucket, data lake, data bog, data swamp, or similar without Valohai thinking it owns the whole thing.
The root path only affects future storage operations. Existing data is not moved.
A new Manual Search task type was added. You can define full parameter sets to run a Task over.
You can now hide old or unnecessary commits from commit selection dropdowns. Find the new Manage Commits view in Project > Settings > Repository.

Enhancements

The user experience for creating new API tokens is now much nicer.
Parameter inputs that would be overridden by incoming pipeline edges are now shown disabled.
The tabs in the Project Settings view no longer occasionally have a secondary duplicate title.
You can now filter Datums based on whether they were uploaded, or created by executions.
Dataset-related APIs are now faster and more robust.
Deployment log filtering now takes timezone selection into account.

Bugfixes

Certain old runtime statistics could be misparsed. No more.
Certain pipelines that could work just fine when created could fail when used via triggers. No more.
Datasets and aliases couldn't be referred to using IDs only. No more.
Deployment image building wasn't aware of variable YAML paths. No more.

Patch Note 2022-08-23

Bugfixes

Interrupted spot executions' outputs didn't always get schlepped over correctly. They now do.
Task nodes with optional parameters without default values now work more reliably.
Copying executions works more reliably.
Jobs that fail to enqueue for 50 times will now error out.
Jupyter executions launched from certain environments now work properly.

Patch Note 2022-08-15

New features

Interrupted spot executions get their previous outputs as inputs under at _restart.
This allows easily to proceed with the work after an unexpected spot restart.

UI improvements

A lot of small user interface improvements e.g. to the usage view.

Bugfixes

You can now properly delete projects with exceptionally long names.
GCP preemptible executions are properly restarted if the resources are reclaimed midrun.
Fixed that multi-organization users might be blocked from accessing any environments.
Improved code commit resolution when working with multiple Valohai YAML files inside a single project.

Patch Note 2022-08-03

Oh boy, have we got a doozy for you! While most of the Valohai crew was enjoying a well-earned Finnish summer over July, a couple of us held down the fort and cranked out a bunch of features and bugfixes for all y'all.

New features

You can now view the queue and scaling status for your execution's or task's environment in a nifty pop-over card.
Let us know if there are more details that you'd like to see in the card and we'll look into it!
Spot/pre-emptible executions can now be automatically restarted if they get preempted.
A future iteration will also automagically carry over any output state from the preempted execution, but for now you'll have to start from scratch.
You'll now get a notification when you're added to a team or to collaborate on a project.

Task UI improvements

The Task detail table got a face lift. It's now more concise and readable.
You can now see the price estimate for an entire task in the Task detail table. There's naturally also an API endpoint for that.
Tasks can now be tagged.

Other UI improvements

We yeeted the secondary "project navigation bar" out and tucked its content into the top navigation bar. UX SCIENCE!!!
Nearly all disabled UI elements now have a nifty tooltip telling you why they're disabled.
You can now create deployment nodes in the create pipeline view (if that feature is enabled for your account, anyway).
You can now save local drafts of pipelines in the create pipeline view, if you're lucky enough to be allowed to do so.
You can now view the API call for creating a deployment version. Automation!
The "Create Project" Ownership dropdown is now a lot clearer. Optgroups are a thing!
Links in the execution event log are now clickable. Wow!

API enhancements

You can now filter executions by their creation time range.
You can now set the status_detail field of a running execution via the API.
In the near future, we'll also surface this to the execution itself, so it's one valohai-utils call away!

Performance improvements

The datum browser got a bit faster.
When changing filters or sorting in various tables, we now sneakily cancel any outstanding requests to the server. Less requests = less lag, or that's the theory anyway.
The execution property keys endpoint learned a thing or two about caching.
Retrieving metadata for multiple executions at once is now much faster.
Purging an execution's outputs is now somewhat faster. Would that cloud storage providers had an API for all-or-nothing deletion of multiple files...

Bugfixes

The Usage view got some love:
Date filtering was fixed. Turns out end dates aren't the same thing as start dates.
Aggregations were occasionally mishandled. We replaced the "mis" with "correct", so now they're correcthandled, which isn't a word, but it's a good thing.
Relative time popovers now show the date too, not just the time. Although, of course we all know that executions run in the witching hour tend to have the best accuracy metrics.
The datum browser no longer loads data twice to start with. That was silly to do, wasn't it?
Copying tasks or executions with inputs that referred to datums could occasionally break the UI. This should no longer happen.
The quite defunct forms in the Browsable API were obliterated.
Legacy execution realtime stats are now properly formatted. No more 300,000 CPUs for you, sorry!
Deployments will no longer crash and burn due to environment variables that Kubernetes just can't deal with.
Scaling status is no longer shown for environments that can't scale.
Non-billed environments are no longer considered for execution credit check.

Patch Note 2022-06-27

Improvements

Allow sorting executions by task tags and show those tags in the UI.
Datum and datum alias user interfaces have been given multiple improvements related to deletion and showing that status.
Expand task input fields for string parameters. String inputs can sometimes get reaaaally long for a good reason.

Bugfixes

Fix metadata rendering on when an execution has metadata with multiple different types.
It could crash the interface on some occasions if an unexpected type was encountered.
Fix projects with spaces in their name.
Project names haven't been allowed spaces for a while and our old migration didn't catch all of them. This might've caused errors when operating on those projects.
Skip calculating execution metadata statistics on arrays and objects.
Showed useless [Object]s on the user interface.

Patch Note 2022-06-15

Speed improvements

Google Storage URL resolution is now twice as fast. Nice.

UX improvements

An informative message was added to the datum alias table when it would otherwise be empty.
Executions now have a neat link to repository settings when no commits are available.
You can now elect to purge all of a project's data when trashing it.
Datum aliases now have a "Copy URL" dropdown action.

Bugfix... improvements

Execution modification times (mtime for you API fiends) are now properly updated upon status changes.
Optional choice inputs now correctly have an empty choice option in the UI.
Certain not-found and permission-related errors now give more helpful error messages.
The browser back button now works better in certain tabbed views.

Patch Note 2022-06-08

UI improvements

The data browser's sidebar can now be collapsed.
You can now zoom further away in the pipeline views. This is quite useful for large pipelines.

Bugfixes

Some files that were previously unpreviewable are now previewable. It was as easy as removing an "un", who knew!
Commits using certain custom YAML paths were unretrievable. No longer! We removed "un" from there too.
The dashboard's chat button now actually works.

Patch Note 2022-06-01

Hey, happy June everyone! We've been working hard to make the Valohai platform a little better... and there's big data-handling improvements coming up!

UI Improvements

We now have all new gauges for the real-time machine statistics (CPU, memory, GPU, etc.). They're quite fancy!
Durations and times are now more easily mouse-overable.
You no longer need to confirm that, yes, you do want to download this single datum.

Bugfixes

Certain corner cases regarding lazily created grid search task executions have been sanded out. They're very smooth now.

Patch Note 2022-05-18

New features

Valohai now estimates the CO2 emissions of executions run on EC2. This is powered by Teads Engineering's dataset and our very own Wattson library.
Distributed tasks are now a thing! If you want to try them out, reach out to our support.
You can now specify registry credentials to be used with any Docker Hub image, public or private. This is so you don't trip on the Docker Hub rate limit.

Improvements

The task details table has been tidied for more readability.
You can now tag multiple executions in one fell swoop in the Executions table.
You can now download the SSH key parts when generating them – no need to manually copy-paste things!
Help is now more readily available on your dashboard. Never fear!
Column selection popovers got a de-uglification.
You can now see deployment, deployment version, and endpoint IDs in the respective views.

Bugfixes

Tags are now copied along with pipelines.
Stores can now be soft-deleted more reliably.
Project names are now validated a little more strictly. Or, in other words, like they should have been all along.
Certain cancelled-job states are now handled with more grace.
Certain execution data didn't get copied when a task created new executions. It does now.

Patch Note 2022-05-05

New features

Grid search tasks no longer create all of the executions beforehand, for efficiency and performance reasons.
Grid search tasks can be much larger than before.
You also won't be saddled with a ton of never-started executions if you decide to stop a task early.
The task status information in the frontend was also enhanced to account for "not-created-yet" executions.
The Pipeline detail view gained a new Outputs tab.
You can now store and recall view presets (filters, columns) in the Executions table.
This feature is in beta; please reach out to our support to test it out.

UX improvements

The "Image" column is now available in the executions list view.
Many instances of the back button not doing what the back button should (i.e. go back) have been fixed.
You can now select a time range in the Usage Explorer.
Deployments can now be sorted by enabled status.
You can now generate SSH key pairs directly in the browser.
You can now filter executions, tasks, and pipelines by multiple creators.
It's now easier to navigate to executions from the comparison view.
Table columns can now be reordered.

Bugfixes

Transient errors connecting to queue machines are now handled better and won't cause executions to immediately error out.
A Bayesian task no longer creates more executions than it should if the batch size is larger than the execution limit.
You can now properly unset the commit and log stores for a project.
Pipeline copying was improved. It will be further improved. Stay tuned.
Certain corner cases of pipeline stepping not doing what it should might be fixed now.
Logarithmic variant parameters could occasionally divide by zero. This has been fixed, and as such this no longer causes a catastrophic collapse of the known universe.
The pipeline detail view could occasionally cause a bunch of extraneous requests to be made. It's been reined in.

Patch Note 2022-04-13

Bugfixes – User interface

Flag parameters are now correctly reset when copying executions.
We now use the production build of React. Who knew that you really need to correctly set NODE_ENV...

Bugfixes – Infrastructure

Private installations behind proxies are now better supported.
Whenever possible to infer, cache memory is no longer counted towards the memory usage of an execution.
Temporary connection errors should no longer cause inputs to be missing from an execution.
Certain high-log-message-output situations should now be handled better.
Executions should finish faster now.
Certain degenerate task parameter cases no longer cause Weirdness and Bad Things, but an Error instead.
Some timeout conditions are now handled better.
These patch note bullet points are now in some cases vaguer than they had been before.

Patch Note 2022-03-16

Worker improvements

datum_id is now available in the input metadata files on the worker.
Image pull progress gets now reported.

UX improvements

You can now set the maximum number of decimals shown in the Executions table's metadata columns, in case auto-determination doesn't quite cut it.

Bugfixes

The Deployments overview view looked silly. It no longer does.
Copying executions with files whose locations had been resolved from wildcards using keep-directories now retain those locations.

Patch Note 2022-03-09

🌻

Technology preview: monorepository

You no longer need be confined to a single valohai.yaml file, but can choose between multiple!
Please contact our support to enable monorepositories for your account.

UX improvements

Commits now appear under all of their related refnames, not only an arbitrary single one.
When a repository fetch fails, it's now much easier to get to configure your repository better.
The project dropdown menu in the top right corner was made faster.
The notification bell dropdown was emwidened, to fit more and bigger words.
If there is a problem retrieving events for an ongoing execution, there will be a marker on the Log tab to that effect.
When deleting a pipeline, its executions and tasks are similarly archived away.
The effective enablement status for deployment versions and endpoints is now more obvious.
Organizations' teams can now be renamed.
Invalid cron expressions in triggers are now caught early. No more wondering why a trigger doesn't fire because it's set for next year!

API improvements

The fetch API call now supports setting a ref for a particular fetch.

Under-the-hood improvements

Some rust was added where it was still conspicuously absent, to make things faster.

Bugfixes

The Create Pipeline view occasionally had a smooshed layout. We unsmooshed it to memorate the occasion.
The preview for text datums no longer keeps scrolling up annoyingly, because we applied a proprietary secret sauce de-annoying algorithm to it.
Certain technically invalid commands involving multiple parameters no longer crash and burn.
The Team Quotas page is again accessible without having to fiddle with your browser URL bar by hand.
A "No repository" label had been misaligned. We nudged it back into place.
When fetching a commit configuration fails, you no longer see the steps from an unrelated commit. That was weird.
S3 store locations are no longer validated for non-S3, but S3-compatible stores.
Organization-owned single project stores can now be converted to organization-level shared stores.

Patch Note 2022-02-24

New features

We're now using Zendesk for customer support and bug reporting. Fun!
You can now select whether a pipeline node erroring out should cause the whole pipeline to do so, or if it should continue its merry way.

Bugfixes

The Datum API was made faster.
The Execution view is more robust now.

Patch Note 2022-02-16

Happy Valentine's Day! We've been too busy working on stuff to say that earlier, but better late than never!

New features

You can now copy existing pipelines into new ones. This is beta, and available at special request. Let us know.
Organization admins can now tweak their environments' prices.

UX polish

Metadata and parameter columns now try to figure out an optimal rounding for their contents to avoid misleading rounding.
Metadata charts are more responsive on small screens.
You can now see which project environment variables are inherited when the option to do so is enabled.
The Create Pipeline view also works better on narrow screens these days.
For in pipelines executions, breadcrumb correct the navigation is order now. Unlike that sentence.
You can now see a preview of the command line when creating an execution.
You can now easily select all nodes to reuse when editing or copying pipelines.

Bugfixes

Concurrent pipeline and task creation should now cause less trouble. Hey... deja vu there...
Ad-hoc ingestion is now faster and more robust.
The Task view no longer makes a huge barrage of initial requests. Oops!

Patch Note 2022-01-31

Happy Monday! It's very nearly February now!

API improvements

The execution creator ID (not just the username) is now available in the API.

UI polish

All the main table views are now always w i d e
The email address management view got a bit of nail polish.
The dashboard view no longer shows stray slashes while loading data.
Datum previews can now be popped out into modals, for a little more space.
When searching for a datum, more options are now shown.
The user experience for custom Docker repositories is a little more informative.

Bugfixes

An "invitations" link in the organization settings was fixed to be significantly less broken.
The pipeline creation view no longer just remains blank if there are no commits.
Query strings in the deployment tester view now work once more.
Concurrent pipeline and execution creation should now cause less trouble.
When searching for a datum, starting your search with an execution # now works more reliably.
Tasks in stopping states no longer occasionally inadvertently revert to not being stopping.

Patch Note 2022-01-18

UX Enhancements

The first column of the Executions table is now resizable. Wow!
We no longer attempt to automagically load the datum preview for large files.
You can click anywhere on a datum row to open its preview pane.

API Enhancements

Objects belonging to archived projects are no longer listed.
You can now find executions not belonging to a task or pipeline by passing e.g. ?task=null.

Bugfixes

The frontend of the project general settings view now takes into consideration the read-onliness of a project.
The billing views work correctly once more.
Starting new executions and especially large tasks should be a lot faster since we now avoid doing extra work. Smart!

Patch Note 2022-01-12

New features

Datum tables now have a new, nifty split view for viewing datum details!
Datums can be mass-downloaded and mass-purged.

Improvements

Datum aliases can now be up to 300 characters.
Where possible, DELETE API requests return the ID of the freshly deleted object.
Notification channels can now be named.

Bugfixes

Starting and stopping tasks and executions is now a bit faster.
Using the pre-yuletide tag parameter for filtering datums now works again.

Patch Note 2022-01-05

Happy first patch note of 2022! Here's to hoping for a brighter year.

New features

Executions, pipelines, and tasks now track where they're copied from.
You can filter the executions, pipelines, and tasks tables to include or exclude copied objects.

Improvements

The pipeline creation view got a bit of a facelift in preparation for something even grander that's, well, pun absolutely intended, in the pipeline.

Bugfixes

Execution finalization should now be more robust.
Tasks and executions now stop a bit more snappishly.

PreviousUpdates 2023 NextUpdates 2021

Last updated 21 hours ago

Was this helpful?