Differences between revisions 1 and 10 (spanning 9 versions)

Polyaxon

Contents

Polyaxon

Installing and Configuring Polyaxon CLI

Installing the CLI:

pip[3] install polyaxon-cli

Keep in mind that the location, were polyaxon is installed, must be on your $PATH. For the previous command polyaxon can e.g. be installed in $HOME/.local/bin (which is usually not on $PATH).

Configuring the CLI (so you can access the server backend on perses):

polyaxon config set --host=10.8.152.97 --http_port=31811 --ws_port=31812

Login in to Polyaxon

Before you can log in to Polyaxon you will need an account. If you don't have an account yet, send an EMail to naether.markus@gmail.com .

polyaxon login --username=<USERNAME> [--password=<PASSWORD>]

Using Polyaxon

Creating new project

First, you can create a new project from any place within your filesystem:

polyaxon project create --name=<my-project> --description='My polyaxon project.'

Within a Terminal go to the directory of you project and initialize the previously created project:

polyaxon init <my-project>

Through this command an empty file polyaxonfile.yml will be generated, which can then be edited further to reflect the training of your project. For some small examples take a look at the examples section of this article.

Configuring Project

A simple example of a project description. Keep in mind to set the kind value correctly (experiment for a single experiment, group for set of experiments).

---
version: 1

kind: experiment

build:
  image: tensorflow/tensorflow:latest-py3
  build_steps:
    - pip3 install --no-cache-dir -U polyaxon-client

run:
  cmd: python3 main.py

A good documentation and overview of the whole syntax and all sections are available through the official documentation of polyaxon: https://docs.polyaxon.com/polyaxonfile_specification/sections/

Uploading and running the Project

In order to start the training process you must upload the source and tell polyaxon which polyaxon file should be used for execution (the default is polyaxonfile.yml).

Either use to seperate steps:

polyaxon upload
polyaxon run [-f <DESCRIPTION_FILE.yml>

Or combine both steps into one:

polyaxon run -u [-f <DESCRIPTION_FILE.yml>]

Observation of running task

Get a list of all experiments:

polyaxon project experiments

Inspect the resources and logs of a specific experiment: Resources:

polyaxon experiment -xp <ID> get

Logs:

polyaxon experiment -xp <ID> logs [--past]

More information of of the CLI just go to the well documented reference of polyaxon: https://docs.polyaxon.com/polyaxon_cli/introduction/

Retrieving Data

Currently we have two different data input directories that can be used:

/data/1/: This path directly maps to titan:/local/hdd/exports/data/ and is the normal path you should use.
/data/local/: This path maps to [rubur/flavus]:/local/ssd/ should be used for real big datasets with dozens of GB of data.

Saving Data

In order to later use e.g. your trained model you have to save it somewhere. When using Polyaxon you can directly use the build in get_outputs_path() method. ` from polyaxon_client.tracking import get_outputs_path `

By using the path provided by get_outputs_path() for saving your data you will be able to download the files that are saved within this path by using e.g.: ` polyaxon experiment -xp <ID> outputs `

Web-Dashboard

Some of the data can also be viewed via the Web-Dashboard which is either directly accessible via: http://10.8.152.97:31811

Or by calling:

polyaxon dashboard

Examples

For a small example one can download the following repository:

git clone https://ad-git.informatik.uni-freiburg.de/ad/polyaxon_demo.git
git clone git@ad-git.informatik.uni-freiburg.de:ad/polyaxon_demo.git

Polyaxon Quick Start

There is now a small Quick Start in the official polyaxon documentation available: https://docs.polyaxon.com/quick_start/

-  ⇤ ← Revision 1 as of 2019-01-18 10:57:32 → 
  Size: 3015
  Editor: Markus Näther
  Comment:
+   ← Revision 10 as of 2019-02-13 09:14:12 → ⇥
  Size: 4220
  Editor: Markus Näther
  Comment: Added section Retrieving Data
-Deletions are marked like this.
+Additions are marked like this.
 Line 2:
-Line 6:
+Line 5:
+Installing the CLI:
 Line 7:
-Installing the CLI:
-Line 9:
+Line 8:
-pip install polyaxon-cli
+pip[3] install polyaxon-cli
-Line 11:
+Line 10:
-Configuring the CLI ( so you can access the server backend on perses ):
+ Keep in mind that the location, were polyaxon is installed, must be on your {{{$PATH}}}. For the previous command polyaxon can e.g. be installed in {{{$HOME/.local/bin}}} (which is usually not on {{{$PATH}}}).

Configuring the CLI (so you can access the server backend on perses):
-Line 13:
+Line 15:
-polyaxon config set --host=10.8.152.97 --http_port=31811  --ws_port=31812
+polyaxon config set --host=10.8.152.97 --http_port=31811 --ws_port=31812
-Line 15:
+Line 17:
-Line 17:
+Line 18:
 Line 23:
+== Using Polyaxon ==
=== Creating new project ===
First, you can create a new project from any place within your filesystem:
-Line 24:
+Line 27:
-== Using Polyaxon ==

=== Creating new project ===

First, you can create a new project from any place within your filesystem:
-Line 32:
+Line 30:
+Within a Terminal go to the directory of you project and initialize the previously created project:
-Line 33:
+Line 32:
-Within a Terminal go to the directory of you project and initialize the previously created project:
-Line 38:
+Line 35:
-Line 42:
+Line 38:
+A simple example of a project description. Keep in mind to set the {{{kind}}} value correctly (experiment for a single experiment, group for set of experiments).
-Line 43:
+Line 40:
-A simple example of a project description. Keep in mind to set the {{{kind}}} value correctly (experiment for a single experiment, group for set of experiments).
-Line 56:
+Line 52:
   cmd: python3 main.py
-Line 58:
+Line 54:
-A good documentation and overview of the whole syntax and all sections are available through the official documentation of polyaxon:
[[https://docs.polyaxon.com/polyaxonfile_specification/sections/]]
+A good documentation and overview of the whole syntax and all sections are available through the official documentation of polyaxon: https://docs.polyaxon.com/polyaxonfile_specification/sections/
-Line 64:
+Line 57:
-Line 68:
+Line 60:
-Line 72:
+Line 65:
+Or combine both steps into one:
-Line 73:
+Line 67:
-Or combine both steps into one:
-Line 77:
+Line 70:
+=== Observation of running task ===
Get a list of all experiments:
-Line 78:
+Line 73:
-=== Observation of running task ===

Get a list of all experiments:
-Line 85:
+Line 76:
+Inspect the resources and logs of a specific experiment: Resources:
-Line 86:
+Line 78:
-Inspect the resources and logs of a specific experiment:
Resources:
-Line 91:
+Line 81:
+Logs:
-Line 92:
+Line 83:
-Logs:
-Line 96:
+Line 86:
+More information of of the CLI just go to the well documented reference of polyaxon: https://docs.polyaxon.com/polyaxon_cli/introduction/
-Line 97:
+Line 88:
-More information of of the CLI just go to the well documented reference of polyaxon:
[[https://docs.polyaxon.com/polyaxon_cli/introduction/]]
+=== Retrieving Data ===
-Line 100:
+Line 90:
+Currently we have two different data input directories that can be used:

 * '''/data/1/''': This path directly maps to ```titan:/local/hdd/exports/data/``` and is the normal path you should use.
 * '''/data/local/''': This path maps to ```[rubur/flavus]:/local/ssd/``` should be used for real big datasets with dozens of GB of data.

=== Saving Data ===

In order to later use e.g. your trained model you have to save it somewhere. When using Polyaxon you can directly use the build in ```get_outputs_path()``` method. 
```
from polyaxon_client.tracking import get_outputs_path
```

By using the path provided by ```get_outputs_path()``` for saving your data you will be able to download the files that are saved within this path by using e.g.:
```
polyaxon experiment -xp <ID> outputs
```

=== Web-Dashboard ===
Some of the data can also be viewed via the Web-Dashboard which is either directly accessible via: http://10.8.152.97:31811

Or by calling:

{{{
polyaxon dashboard
}}}
-Line 101:
+Line 116:
+For a small example one can download the following repository:
-Line 102:
+Line 118:
-For a small example one can download the following repository:
-Line 107:
+Line 122:
-Line 109:
+Line 123:
-There is now a small ''Quick Start'' in the official polyaxon documentation available: [[https://docs.polyaxon.com/quick_start/]]
+There is now a small ''Quick Start'' in the official polyaxon documentation available: https://docs.polyaxon.com/quick_start/