Updates 2017
Patch Note 2017-12-10
Make it more comfortable to select machine type on the command-line client.
Add credit system to the billing.
Improve accuracy of execution duration calculations.
Developer Comment: Canceled executions might sometimes get zero duration, thus become ignored for billing. Now they'll get a proper duration.
Empty lines on hyperparameter optimization parameters are now ignored.
Developer Comment: These empty lines caused crashing executions because of a null input if the user code didn't handle them correctly.
Patch Note 2017-11-23
Add execution tagging system.
Allow users to download files through web client data browser.
Show which machine type was used for an execution in the web client.
Uploading won't overwrite an already existing S3 files.
Developer Comment: Previously, overwrite of files could happen if executions uploaded two files with the same name to a custom S3 store.
Patch Note 2017-11-06
Add GitHub App integration to streamline onboarding of new users.
Allow users to delete executions and uploaded files through the web client.
Using custom Docker image will now start on the first try.
Developer Comment: Our Docker image caching locked specific operating system files that our worker software was required to use. This quirk caused the first run sometimes to crash in a mysterious error but rerunning solved the issue. But this locking logic has now been fixed.
Patch Note 2017-10-18
Disallow copying of ad-hoc executions.
Developer Comment: This is 100% ideological change as we had this working before disabling it. One-off executions shouldn't be copyable, the machine learning pipeline should we well defined and version controlled if it is meant for production.
Improve error reporting on git repository fetch errors for support.
Load long executions logs without lagging the web client.
Developer Comment: So loading all the longs synchronously was a bad idea to start with. When we have clients that log more than 50k lines of output, you just cannot render all at once. Well, fixed.
Patch Note 2017-10-02
Release the example project for R.
Limit how many previous commit versions to show in execution create page dropdown.
Developer Comment: We had projects that were so long running that the number of distinct commits made it hard to find the version you are looking for. Final fix for this is still in the roadmap.
Avoid executions from entering eternal stopping limbo.
Developer Comment: When workers lost connection to the master node while user instructed the execution to be stopped, it would start spinning down when the connection was resumed but never actually finish shutting down. This has been fixed.
Patch Note 2017-09-13
Add new "Stopping" status for executions that take a long time to shut down.
Allow project admins to set a default store, so it doesn't have to be defined before every execution and task.
Patch Note 2017-09-04
Allow hyperparameter optimization of flag parameter type.
Custom stores can now be uploaded to.
Limit how many log events are shown in the web client.
Developer Comment: One of our users were logging more than 20k lines per minute, so we needed a way to limit how much is shown on the web client.
Resign uploaded input file credentials before queueing an execution that uses then.
Developer Comment: This caused long-lived files not to be able to be downloaded by workers.
Patch Note 2017-08-22
Valohai platform is now out of beta!
Add store system to allow customized download endpoints.
Default store is still Valohai managed S3 bucket but now you can customize it.
Rework the usage flow how to define existing files as inputs to executions.
Patch Note 2017-08-10
Add user profile page.
Rework how real-time statistics are displayed.
Developer Comment: Now with more spacing!
Patch Note 2017-07-27
Add support for non-GPU instances.
Developer Comment: CPU or memory heavy machines can frequently be more useful for feature extraction and other non-deep learning training steps of machine learning.
Improve how fast execution status updates.
Allow collaborators to see input and output files of projects they have been invited.
Developer Comment: We were a bit too harsh when checking permissions to see project files. Better safe than sorry!
Patch Note 2017-07-10
Allow users to browse, download and manage their S3 output files on the web client.
Allow users to use output and manually uploaded files as input without opening an S3 object to the public.
Patch Note 2017-06-27
Allow uploading input files to S3 stores with the web client.
Allow short hash for commit selection in API and the command-line client.
Implement per-user quota to restrict how much a single user can run in the shared clusters.
Developer Comment: We had free-tier users that took all the computational power in our cluster so we had to restrict it so everybody can get a share without slowing down their workflow.
Patch Note 2017-06-14
Add full repository control to the REST API.
Add more details about executions to the REST API.
Add support for subdirectories in output upload.
Input parameter 0 (zero) will now work correctly.
Patch Note 2017-05-25
Add support for writing notes on executions.
Show links to documentation all around the web client so you can learn more about a specific feature of the system.
Record real-time statistics like CPU or GPU so you can view the final values after completion.
Non-US locale decimal point will now work with hyperparameter range definitions.
Patch Note 2017-05-11
Show total worker machine memory on real-time statistics.
Now clicking usage number will take you to detail spending page.
Allow importing tutorial projects even if you have completed the onboarding.
Patch Note 2017-04-27
Add pagination support for task executions list page.
Show more detailed resource usage by project and by hardware type used for billing.
Make worker infrastructure clean up cached content more rigorously.
Developer Comment: We were caching stuff for a too long time which caused some of our storages to get full because of unnecessary files.
Navigate straight to the execution logs screen after execution creation.
Developer Comment: One of our users requested this small tweak, and we agree that it saves a lot of times if you are working on the executions a couple of hours per day.
Patch Note 2017-04-10
Improve project to repository linking user flow.
Now repository is automatically synced when changed.
We can automatically detect which type of git repository we are working with from the repository URL.
Improve AWS cluster downscaling as workloads can drop from hundreds of parallel executions to just zero in mere minutes.
Developer Comment: Downscaling happened too fast after workload spike. Frequently users just review the results and start a new batch right after the previous one.
Show more obvious error when the command-line client is used to start executions on a project without a linked repository.
Patch Note 2017-03-29
Release Valohai documentation website.
Show billing estimate for the cloud resource usage.
Patch Note 2017-03-01
Add hyperparameter optimization and tasks.
Developer Comment: Allows manual grid search by listing parameter variants you want to try.
Allow project collaborators also see and use on-premises hardware.
Add local directory mount as an input method for on-premises workers.
Patch Note 2017-02-22
Web client graphics are now served from a CDN for better speed.
Stop canceled executions more gracefully.
Developer Comment: The running process will be sent SIGINT before the actual force shutdown.
Patch Note 2017-02-20
Allow users to select which type of AWS instances to run their execution on, not just g2.
Stopped executions should not be picked again by Valohai workers.
Developer Comment: Worker nodes could be a bit too eager to work and took canceled work from work queues.
Patch Note 2017-02-14
Make our AWS server clusters scale according to workloads we are having.
Improve layout of the execution real-time statistics.
Rework how browsers render execution logs; now with more color!
Patch Note 2017-02-10
Add support for project collaborators.
New parameter type: flag.
Now visitor will stay on the same execution screen tab when refreshing.
Show real-time statistics of executions such as CPU and memory.
Patch Note 2017-02-02
Move default hosting of output files from the master server to S3.
Allow users to cancel already started executions.
Allow users to copy previous executions.
Add support for optional execution inputs and parameters.
Add support for multiple commands in a single execution.
Patch Note 2017-01-18
Parameters passed to the worker nodes are now properly quoted.
Developer Comment: This caused some invalid commands to be generated when using string parameters with spaces.
Patch Note 2017-01-17
Integrate Valohai YAML standard.
Support for Valohai YAML steps, Docker image, command, inputs, and parameters.
Add metadata support by reading JSON from standard output.
Last updated
Was this helpful?