Polyaxon

Installing and Configuring Polyaxon CLI

Installing the CLI:

pip[3] install polyaxon-cli

• Keep in mind that the location, were polyaxon is installed, must be on your $PATH. For the previous command polyaxon can e.g. be installed in $HOME/.local/bin (which is usually not on $PATH).

Configuring the CLI (so you can access the server backend on perses):

polyaxon config set --host=perses.informatik.privat --port=31811

Login in to Polyaxon

Before you can log in to Polyaxon you will need an account. If you don't have an account yet, send an EMail to naether.markus@gmail.com .

polyaxon login --username=<USERNAME> [--password=<PASSWORD>]

Using Polyaxon

Creating new project

First, you can create a new project from any place within your filesystem:

polyaxon project create --name=<my-project> --description='My polyaxon project.'

Within a Terminal go to the directory of you project and initialize the previously created project:

polyaxon init <my-project>

Through this command an empty file polyaxonfile.yml will be generated, which can then be edited further to reflect the training of your project. For some small examples take a look at the examples section of this article.

Configuring Project

A simple example of a project description. Keep in mind to set the kind value correctly (experiment for a single experiment, group for set of experiments).

---
version: 1

kind: experiment

build:
  image: tensorflow/tensorflow:latest-py3
  build_steps:
    - pip3 install --no-cache-dir -U polyaxon-client

run:
  cmd: python3 main.py

A good documentation and overview of the whole syntax and all sections are available through the official documentation of polyaxon: https://docs.polyaxon.com/references/polyaxonfile-yaml-specification/

Uploading and running the Project

In order to start the training process you must upload the source and tell polyaxon which polyaxon file should be used for execution (the default is polyaxonfile.yml).

Either use two separate steps:

polyaxon upload
polyaxon run [-f <DESCRIPTION_FILE.yml>]

Or combine both steps into one:

polyaxon run -u [-f <DESCRIPTION_FILE.yml>]

Observation of running task

Get a list of all experiments:

polyaxon project experiments

Inspect the resources and logs of a specific experiment: Resources:

polyaxon experiment -xp <ID> get

Logs:

polyaxon experiment -xp <ID> logs [--past]

More information of of the CLI just go to the well documented reference of polyaxon: https://docs.polyaxon.com/references/polyaxon-cli/

Retrieving Data

Currently we have two different data input directories that can be used:

• /data/1/: This path directly maps to titan:/local/hdd/exports/data/ and is the normal path you should use.

• /data/local/: This path maps to [rubur/flavus]:/local/ssd/ should be used for real big datasets with dozens of GB of data.

• /data/datasets/: This path maps to /nfs/datasets/. Keep in mind that you will only have read access to those directories.

• /data/students/: This path maps to /nfs/students/. Keep in mind that you will only have read access to those directories.

Saving Data

In order to later use e.g. your trained model you have to save it somewhere. When using Polyaxon you can directly use the build in get_outputs_path() method.

from polyaxon_client.tracking import get_outputs_path

By using the path provided by get_outputs_path() for saving your data you will be able to download the files that are saved within this path by using e.g.:

polyaxon experiment -xp <ID> outputs

Web-Dashboard

Some of the data can also be viewed via the Web-Dashboard which is either directly accessible via: http://perses.informatik.privat:31811

Or by calling:

polyaxon dashboard

Examples

For a small example one can download the following repository:

git clone https://ad-git.informatik.uni-freiburg.de/ad/polyaxon_demo.git
git clone git@ad-git.informatik.uni-freiburg.de:ad/polyaxon_demo.git

Polyaxon Quick Start

There is now a small Quick Start in the official polyaxon documentation available: https://docs.polyaxon.com/concepts/quick-start/

Polyaxon Integrations

For a complete list of possible integrations take a look at the official integration page. For your purposes one of the following integrations will be the most useful ones:

• Tensorflow

• PyTorch

• Keras

• Jupyter Notebook (Don't forget to always destroy them if you don't need them anymore!)