Broccoli

Code

Code is in https://ad-websvn.informatik.uni-freiburg.de/broccoli/. The Code for CSD is in the subfolder: https://ad-websvn.informatik.uni-freiburg.de/broccoli/nlp/

Current version

Built by Björn beginning of August 2016 (TODO: copy to elba), Wikipedia version from August 2016 (2.8B postings), latest Freebase dump (freebase-rdf-latest, 372M statements extracted a la Freebase Easy).

Building an Index

First, obtain a Wikipedia XML and a Freebase dump (usually stored in /nfs/raid5/broccoli/...) and make sure these correct files are referenced in broccoli/Makefile and broccoli/freebase/Makefile. Give a proper name to your index using the variable DBTAIL in broccoli/Makefile

Create ontology.txt

If you use an existing RDF3X DB (as usually the case), make sure it is referenced correctly in broccoli/freebase/Makefile and only run (inside the broccoli folder):

    make get-freebase-ontology

Otherwise run:

    make -C freebase/ build-db
    make get-freebase-ontology

Create cas0.zip

Make sure all paths are set correctly in paths.mak

on FILICUDI!! run (in the broccoli fodler)

    make deploy-broker

Then, on the server you want to have the reader and writer, run

    make deploy-reader

To get things going run the following on as many PC's (and servers) as possible

    make deploy-senna

Create a broccoli index

Make sure to copy/move or reference the cas0.zip you built in broccoli/Makefile then run

    make build-txt build-index

Start the server

    make start PORT=<PORT>
== Image Cache ==

To add individual images (for demos, needs access to raid so it can write to cache folder, and img has to have a file extension for convert to work, tested on filicudi, does not work on stromboli because code requires Python version >= 3.3):

python3 ~/broccoli/img-hack/image_to_cache.py --mid <MID> --img 'http://...'

HiWi project Kai Haase: see Google Doc.

TODO: Florians code has a mechanism for removing outdated images, which also removes images in the cache which now return a 404 not found (which effectively removes all images from the cache after the shutdown of the Freebase API). This should be corrected. Here is the guilty piece of code from https://ad-websvn.informatik.uni-freiburg.de/broccoli/freebase-imgsvc/fbthumbsvc.php:

// If no image could be found (404 error) create a 404 cache file for the
// current id, return a 404 error and end the script.
if ($return_status_code == 404)
{
  // If there still was an expired cache file then remove it now!
  if ($cachefile_exists)
  {
      unlink($cachefile_path);
  }

  // Create a 404 cache file for the current id.
  touch($cachefile_path . '_404');
 
  returnMissingError();
}