1706
Comment:
|
6899
instance
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
#acl Niklas Schnelle:read,write All:read |
|
Line 3: | Line 5: |
!GitHub repository: https://github.com/elmar-haussmann/aqqu . | <<TableOfContents(3)>> |
Line 5: | Line 7: |
== Aqqu instance == | == Description == |
Line 7: | Line 9: |
2016-06-30: runs under http://metropolis.informatik.uni-freiburg.de:5455 | Question answering from Freebase as described in the [[http://ad-publications.informatik.uni-freiburg.de/CIKM_freebase_qa_BH_2015.pdf|CIKM 2015]] publication. The code below also contains some improvements (neural network, performance) that came after the publication. The public code also contains a README that describes how to download, install, train and run the system. Below describes how to setup the demo (for which the code is not public). |
Line 9: | Line 11: |
TODO: how to start it. | == Code == |
Line 11: | Line 13: |
== Virtuoso instance == | The main repository is public on !GitHub: |
Line 13: | Line 15: |
2016-06-30: Virtuoso instance for Aqqu runs unter http://metropolis.informatik.uni-freiburg.de:9000/sparql . | https://github.com/ad-freiburg/aqqu |
Line 15: | Line 17: |
Start it as follows (kill old process manually): | There is also an internal repository that should be kept in sync with the !GitHub one but may contain branches for internal testing/conferences: https://ad-git.informatik.uni-freiburg.de/ad/Aqqu The Web UI is currently only available internally. It should probably made public too: https://ad-git.informatik.uni-freiburg.de/ad/aqqu-webserver Previously Elmar kept an internal Aqqu version on !BitBucket this is very old and stale though: https://bitbucket.org/elmar-haussmann/aqqu . Similarly there is an old !BitBucket repository for the Web UI: https://bitbucket.org/elmar-haussmann/aqqu-webserver . == Instances == The recommended way to set up an Aqqu instances is described in the [[https://github.com/ad-freiburg/aqqu/blob/master/README.md|README]] on !GitHub and uses `nvidia-docker` (or `docker` when not using a GPU). The below instances were set up be me (Niklas) so they might need a `sudo -u schnelle -i` before to get the right permissions for the repositories. This method is used on the following '''Aqqu''' instances: * On '''titan''' ([[http://titan.informatik.privat:8300]]) * Using the QLever '''freebase-aqqu'' instance listed [[Projects/QLever#Instances|here]]: * Under `/local/ssd2/aqqu` * On '''vulcano''' ([[http://vulcano.informatik.privat:8090]]): * First trained on the `sqtrain` dataset then on `wqsptrain` and `free917train` * Under `/local/raid/ad/schnelle/aqqu` * Using the Virtuoso SPARQL Engine `docker-compose` setup * `docker-compose up -d` in `/local/raid/ad/schnelle/virtuoso-compose` * [[https://github.com/ad-freiburg/virtuoso-compose]] These instances can be used with the following '''Aqqu Web UI''' instances * On '''titan''' ([[http://titan.informatik.privat:5454]], http://titan.informatik.privat:5455 ) * Under `/local/ssd2/aqqu_webui/` * Launched with a single `docker-compose up -d --scale webui=2` == Old Aqqu instance (via Elmar's home) == Start as follows on metropolis: {{{ ssh metropolis sudo su haussmae cd /home/haussmae/demos/aqqu-demo source venv/bin/activate # was: activate aqqu PYTHONPATH=$(pwd):$PYTHONPATH python webserver/translation_webserver.py }}} === Virtuoso instance (using Docker Compose) === 2018-04-02: Virtuoso instance for Aqqu runs at http://vulcano.informatik.privat:9000/sparql this is proxied to http://aqqu.informatik.uni-freiburg.de/sparql/ First make sure you have Docker + Docker Compose set up as described here: https://docs.docker.com/compose/install/ also make sure that your user is in the *docker group* If you are setting up a new instance first check out the repository. If this is an existing instance just switch to its checked out virtuoso-compose (e.g. at vulcano:/local/raid/ad/schnelle/virtuoso-compose) {{{ git clone https://ad-git.informatik.uni-freiburg.de/ad/virtuoso-compose.git cd virtuoso-compose }}} Then start virtuoso with varnish as a caching proxy as follow (the resulting server listens on port 9000). Note that this is the same command whether you set up initially or manually launch the instance {{{ docker-compose up -d }}} To view the log run {{{ docker-compose logs -f }}} === Virtuoso instance (via Elmars home) === Start virtuoso as follows. Server listens on port 8999. (Alternatively follow the instructions from scratch instructions below) {{{ ssh metropolis sudo su haussmae cd /home/haussmae/keyword-translation make start-virtuoso-mini-cai }}} Start the HTTP proxy as follows (kill old process manually, listens on port 9000): |
Line 24: | Line 114: |
=== Parser === Aqqu uses a parser to get part-of-speech tags of query words. The parser is accessed via HTTP Api calls. To start the parser server: {{{ ssh metropolis sudo su haussmae cd /home/haussmae/keyword-translation make start-parser }}} The port is configured in the corenlp-frontent/build.xml. The API can be accessed like this: http://metropolis.informatik.uni-freiburg.de:4000/parse/?text=This%20is%20a%20test%20sentence. === Run the new Aqqu version (with NN) === Start on titan (requires GPU): {{{ ssh titan sudo su haussmae cd /home/haussmae/aqqu-bitbucket source activate aqqu PYTHONPATH=$(pwd):$PYTHONPATH python webserver/translation_webserver.py }}} Runs on port 5454 on titan now. However, titan is not available from outside the uni network. To start a port-forwarding from metropolis: {{{ ssh metropolis sudo su haussmae cd /home/haussmae/temp/nc/python-port-forwardt python2 port-forward.py }}} The service is now available on metropolis:5454 === How to update (any) Virtuoso with custom data === |
|
Line 29: | Line 161: |
grant execute on SPARQL_INSERT_DICT_CONTENT to "SPARQL”; | grant execute on SPARQL_INSERT_DICT_CONTENT to "SPARQL"; |
Line 31: | Line 163: |
grant execute on SPARQL_DELETE_DICT_CONTENT to "SPARQL”; | grant execute on SPARQL_DELETE_DICT_CONTENT to "SPARQL"; |
Line 58: | Line 190: |
== Data == All of the required data to run Aqqu is part of the materials (see above). It is located in the ''data'' subfolder. The scripts to create this data are part of the (old) keyword-translation repository: [[https://bitbucket.org/onekonek/keyword-translation]] |
Aqqu
Contents
Description
Question answering from Freebase as described in the CIKM 2015 publication. The code below also contains some improvements (neural network, performance) that came after the publication. The public code also contains a README that describes how to download, install, train and run the system. Below describes how to setup the demo (for which the code is not public).
Code
The main repository is public on GitHub:
https://github.com/ad-freiburg/aqqu
There is also an internal repository that should be kept in sync with the GitHub one but may contain branches for internal testing/conferences:
https://ad-git.informatik.uni-freiburg.de/ad/Aqqu
The Web UI is currently only available internally. It should probably made public too:
https://ad-git.informatik.uni-freiburg.de/ad/aqqu-webserver
Previously Elmar kept an internal Aqqu version on BitBucket this is very old and stale though:
https://bitbucket.org/elmar-haussmann/aqqu .
Similarly there is an old BitBucket repository for the Web UI:
https://bitbucket.org/elmar-haussmann/aqqu-webserver .
Instances
The recommended way to set up an Aqqu instances is described in the README on GitHub and uses nvidia-docker (or docker when not using a GPU).
The below instances were set up be me (Niklas) so they might need a sudo -u schnelle -i before to get the right permissions for the repositories.
This method is used on the following Aqqu instances:
On titan (http://titan.informatik.privat:8300)
Using the QLever freebase-aqqu instance listed here:
Under /local/ssd2/aqqu
On vulcano (http://vulcano.informatik.privat:8090):
First trained on the sqtrain dataset then on wqsptrain and free917train
Under /local/raid/ad/schnelle/aqqu
Using the Virtuoso SPARQL Engine docker-compose setup
docker-compose up -d in /local/raid/ad/schnelle/virtuoso-compose
These instances can be used with the following Aqqu Web UI instances On Under /local/ssd2/aqqu_webui/ Launched with a single docker-compose up -d --scale webui=2
Start as follows on metropolis:
2018-04-02: Virtuoso instance for Aqqu runs at http://vulcano.informatik.privat:9000/sparql this is proxied to http://aqqu.informatik.uni-freiburg.de/sparql/ First make sure you have Docker + Docker Compose set up as described here: https://docs.docker.com/compose/install/ also make sure that your user is in the *docker group* If you are setting up a new instance first check out the repository. If this is an existing instance just switch to its checked out virtuoso-compose (e.g. at vulcano:/local/raid/ad/schnelle/virtuoso-compose) Then start virtuoso with varnish as a caching proxy as follow (the resulting server listens on port 9000). Note that this is the same command whether you set up initially or manually launch the instance To view the log run
Start virtuoso as follows. Server listens on port 8999. (Alternatively follow the instructions from scratch instructions below) Start the HTTP proxy as follows (kill old process manually, listens on port 9000):
Aqqu uses a parser to get part-of-speech tags of query words. The parser is accessed via HTTP Api calls. To start the parser server: The port is configured in the corenlp-frontent/build.xml. The API can be accessed like this: http://metropolis.informatik.uni-freiburg.de:4000/parse/?text=This%20is%20a%20test%20sentence.
Start on titan (requires GPU): Runs on port 5454 on titan now. However, titan is not available from outside the uni network. To start a port-forwarding from metropolis: The service is now available on metropolis:5454
Grant access rights via the ISQL tool as follows: Complex example SPARQL query from "Programmieren in C++, SS 2016, Ü10 (all action or animation movie with their release date, genre, director, production company, and rating):
All of the required data to run Aqqu is part of the materials (see above). It is located in the data subfolder. The scripts to create this data are part of the (old) keyword-translation repository: https://bitbucket.org/onekonek/keyword-translation
titan (http://titan.informatik.privat:5454, http://titan.informatik.privat:5455 ) Old Aqqu instance (via Elmar's home)
ssh metropolis
sudo su haussmae
cd /home/haussmae/demos/aqqu-demo
source venv/bin/activate # was: activate aqqu
PYTHONPATH=$(pwd):$PYTHONPATH python webserver/translation_webserver.py
Virtuoso instance (using Docker Compose)
git clone https://ad-git.informatik.uni-freiburg.de/ad/virtuoso-compose.git
cd virtuoso-compose
docker-compose up -d
docker-compose logs -f
Virtuoso instance (via Elmars home)
ssh metropolis
sudo su haussmae
cd /home/haussmae/keyword-translation
make start-virtuoso-mini-cai
ssh metropolis
sudo su haussmae
cd /home/haussmae/keyword-translation
make start-varnish
Parser
ssh metropolis
sudo su haussmae
cd /home/haussmae/keyword-translation
make start-parser
Run the new Aqqu version (with NN)
ssh titan
sudo su haussmae
cd /home/haussmae/aqqu-bitbucket
source activate aqqu
PYTHONPATH=$(pwd):$PYTHONPATH python webserver/translation_webserver.py
ssh metropolis
sudo su haussmae
cd /home/haussmae/temp/nc/python-port-forwardt
python2 port-forward.py
How to update (any) Virtuoso with custom data
data/virtuoso/install/bin/isql localhost:1112 dba dba
grant execute on SPARQL_INSERT_DICT_CONTENT to SPARQL_UPDATE;
grant execute on SPARQL_INSERT_DICT_CONTENT to "SPARQL";
grant execute on SPARQL_DELETE_DICT_CONTENT to SPARQL_UPDATE;
grant execute on SPARQL_DELETE_DICT_CONTENT to "SPARQL";
PREFIX fb: <http://rdf.freebase.com/ns/>
SELECT DISTINCT ?fn, ?y, ?gn, ?dn, ?pn, ?rn where {
?f fb:type.object.type fb:film.film .
?f fb:film.film.initial_release_date ?y .
?f fb:film.film.genre ?g .
?f fb:film.film.directed_by ?d .
?f fb:film.film.production_companies ?p .
?f fb:film.film.rating ?r .
?f fb:type.object.name ?fn .
?g fb:type.object.name ?gn .
?d fb:type.object.name ?dn .
?p fb:type.object.name ?pn .
?r fb:type.object.name ?rn
FILTER(lang(?fn)='en')
FILTER(lang(?gn)='en')
FILTER(lang(?dn)='en')
FILTER(lang(?pn)='en')
FILTER(lang(?rn)='en')
FILTER(?gn='Action Film'@en OR ?gn='Animation'@en)
}
Data