Size: 2947
Comment:
|
Size: 3007
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 22: | Line 22: |
{{{ | |
Line 24: | Line 24: |
}}} | |
Line 26: | Line 27: |
{{{ | |
Line 29: | Line 30: |
}}} | |
Line 35: | Line 37: |
{{{ make deploy-broker }}} |
|
Line 36: | Line 41: |
make deploy-broker | |
Line 39: | Line 43: |
{{{ | |
Line 41: | Line 45: |
}}} | |
Line 43: | Line 48: |
{{{ | |
Line 45: | Line 50: |
}}} | |
Line 50: | Line 56: |
{{{ | |
Line 52: | Line 58: |
}}} | |
Line 55: | Line 62: |
{{{ | |
Line 58: | Line 65: |
}}} | |
Line 64: | Line 72: |
Broccoli
Code
Code is in https://ad-websvn.informatik.uni-freiburg.de/broccoli/. The Code for CSD is in the subfolder: https://ad-websvn.informatik.uni-freiburg.de/broccoli/nlp/
Current version
Built by Björn beginning of August 2016 (TODO: copy to elba), Wikipedia version from August 2016 (2.8B postings), latest Freebase dump (freebase-rdf-latest, 372M statements extracted a la Freebase Easy).
Building an Index
First, obtain a Wikipedia XML and a Freebase dump (usually stored in /nfs/raid5/broccoli/...) and make sure these correct files are referenced in broccoli/Makefile and broccoli/freebase/Makefile. Give a proper name to your index using the variable DBTAIL in broccoli/Makefile
Create ontology.txt
If you use an existing RDF3X DB (as usually the case), make sure it is referenced correctly in broccoli/freebase/Makefile and only run (inside the broccoli folder):
make get-freebase-ontology
Otherwise run:
make -C freebase/ build-db make get-freebase-ontology
Create cas0.zip
Make sure all paths are set correctly in paths.mak
on FILICUDI!! run (in the broccoli fodler)
make deploy-broker
Then, on the server you want to have the reader and writer, run
make deploy-reader
To get things going run the following on as many PC's (and servers) as possible
make deploy-senna
Create a broccoli index
Make sure to copy/move or reference the cas0.zip you built in broccoli/Makefile then run
make build-txt build-index
Start the server
make start PORT=<PORT> == Image Cache ==
To add individual images (for demos, needs access to raid so it can write to cache folder, and img has to have a file extension for convert to work, tested on filicudi, does not work on stromboli because code requires Python version >= 3.3):
python3 ~/broccoli/img-hack/image_to_cache.py --mid <MID> --img 'http://...'
HiWi project Kai Haase: see Google Doc.
TODO: Florians code has a mechanism for removing outdated images, which also removes images in the cache which now return a 404 not found (which effectively removes all images from the cache after the shutdown of the Freebase API). This should be corrected. Here is the guilty piece of code from https://ad-websvn.informatik.uni-freiburg.de/broccoli/freebase-imgsvc/fbthumbsvc.php:
// If no image could be found (404 error) create a 404 cache file for the // current id, return a 404 error and end the script. if ($return_status_code == 404) { // If there still was an expired cache file then remove it now! if ($cachefile_exists) { unlink($cachefile_path); } // Create a 404 cache file for the current id. touch($cachefile_path . '_404'); returnMissingError(); }