Polyaxon
Contents
Installing and Configuring Polyaxon CLI
Installing the CLI:
pip[3] install polyaxon-cli
Keep in mind that the location, were polyaxon is installed, must be on your $PATH. For the previous command polyaxon can e.g. be installed in $HOME/.local/bin (which is usually not on $PATH).
Configuring the CLI (so you can access the server backend on perses):
polyaxon config set --host=perses.informatik.privat --port=31811
Login in to Polyaxon
Before you can log in to Polyaxon you will need an account. If you don't have an account yet, send an EMail to naether.markus@gmail.com .
polyaxon login --username=<USERNAME> [--password=<PASSWORD>]
Using Polyaxon
Creating new project
First, you can create a new project from any place within your filesystem:
polyaxon project create --name=<my-project> --description='My polyaxon project.'
Within a Terminal go to the directory of you project and initialize the previously created project:
polyaxon init <my-project>
Through this command an empty file polyaxonfile.yml will be generated, which can then be edited further to reflect the training of your project. For some small examples take a look at the examples section of this article.
Configuring Project
A simple example of a project description. Keep in mind to set the kind value correctly (experiment for a single experiment, group for set of experiments).
---
version: 1
kind: experiment
build:
image: tensorflow/tensorflow:latest-py3
build_steps:
- pip3 install --no-cache-dir -U polyaxon-client
run:
cmd: python3 main.pyA good documentation and overview of the whole syntax and all sections are available through the official documentation of polyaxon: https://docs.polyaxon.com/references/polyaxonfile-yaml-specification/
Uploading and running the Project
In order to start the training process you must upload the source and tell polyaxon which polyaxon file should be used for execution (the default is polyaxonfile.yml).
Either use two separate steps:
polyaxon upload polyaxon run [-f <DESCRIPTION_FILE.yml>]
Or combine both steps into one:
polyaxon run -u [-f <DESCRIPTION_FILE.yml>]
Observation of running task
Get a list of all experiments:
polyaxon project experiments
Inspect the resources and logs of a specific experiment: Resources:
polyaxon experiment -xp <ID> get
Logs:
polyaxon experiment -xp <ID> logs [--past]
More information of of the CLI just go to the well documented reference of polyaxon: https://docs.polyaxon.com/references/polyaxon-cli/
Retrieving Data
Currently we have two different data input directories that can be used:
/data/1/: This path directly maps to titan:/local/hdd/exports/data/ and is the normal path you should use.
/data/local/: This path maps to [rubur/flavus]:/local/ssd/ should be used for real big datasets with dozens of GB of data.
/data/datasets/: This path maps to /nfs/datasets/. Keep in mind that you will only have read access to those directories.
/data/students/: This path maps to /nfs/students/. Keep in mind that you will only have read access to those directories.
Saving Data
In order to later use e.g. your trained model you have to save it somewhere. When using Polyaxon you can directly use the build in get_outputs_path() method.
from polyaxon_client.tracking import get_outputs_path
By using the path provided by get_outputs_path() for saving your data you will be able to download the files that are saved within this path by using e.g.:
polyaxon experiment -xp <ID> outputs
Web-Dashboard
Some of the data can also be viewed via the Web-Dashboard which is either directly accessible via: http://perses.informatik.privat:31811
Or by calling:
polyaxon dashboard
Examples
For a small example one can download the following repository:
git clone https://ad-git.informatik.uni-freiburg.de/ad/polyaxon_demo.git git clone git@ad-git.informatik.uni-freiburg.de:ad/polyaxon_demo.git
Polyaxon Quick Start
There is now a small Quick Start in the official polyaxon documentation available: https://docs.polyaxon.com/concepts/quick-start/
Polyaxon Integrations
For a complete list of possible integrations take a look at the official integration page. For your purposes one of the following integrations will be the most useful ones:
Jupyter Notebook (Don't forget to always destroy them if you don't need them anymore!)