Size: 5079
Comment:
|
Size: 5336
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 8: | Line 8: |
Follow these steps to checkout the !CompleteSearch code from our SVN, compile it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>. | Follow these steps to checkout the !CompleteSearch code from our SVN, compile it, build an index, run a server for that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>. |
Line 13: | Line 13: |
svn checkout http://ad-svn.informatik.uni-freiburg.de/completesearch/codebase |
svn checkout https://ad-svn.informatik.uni-freiburg.de/completesearch/codebase |
Line 23: | Line 22: |
sudo apt-get install libsparsehash-dev # Google HashMap (http://code.google.com/p/google-sparsehash) | sudo apt-get install libsparsehash-dev # Google Hash Map (http://code.google.com/p/google-sparsehash) |
Line 30: | Line 29: |
Edit the Makefile and adjust ''CS_CODE_DIR'' and ''STXXL_CONFIG'', then do: |
|
Line 34: | Line 35: |
This will build three binaries: * buildIndex * buildDocsDB * startCompletionServer |
This will build three binaries: ''buildIndex'', ''buildDocsDB'', ''startCompletionServer''. |
Line 43: | Line 40: |
=== 2. Input (to be produced by a suitable parser) === | === 2. Parse === |
Line 45: | Line 42: |
'''Input 1:''' a file ''<base-name>.docs'', with lines of the form | Use our generic XML Parser (see ''codebase/parser/XmlParserNewExampleMain.cpp'' for an example usage) or our generic CSV Parser (''codebase/parser/CsvParserMain'') or your own parser to produce the following two intermediate files. 1. The file ''<base-name>.docs'', with lines of the form |
Line 48: | Line 49: |
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document> |
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document> |
Line 52: | Line 52: |
This file must be sorted so that ''sort -c -k1,1n'' does not complain. Here is a simple example (the multi-spaces are all TABs) |
This file must be sorted so that ''sort -c -k1,1n'' does not complain. Here is a simple example (the multi-spaces are all TABs) |
Line 62: | Line 61: |
'''Ínput 1:''' a file ''<base-name>.words'', with lines of the form | 2. The file ''<base-name>.words'', with lines of the form |
Line 68: | Line 67: |
This file must be sorted such that ''sort -c -k1,1 -k2,2n -k4,4n'' does not complain. Here is a simple example, matching the example above (again multi-spaces are all TABs): |
This file must be sorted such that ''sort -c -k1,1 -k2,2n -k4,4n'' does not complain. Here is a simple example, matching the example above (again multi-spaces are all TABs): |
Line 97: | Line 95: |
It enables fast processing of the powerful query language offered by CompleteSearch (including | It enables fast processing of the powerful query language offered by !CompleteSearch (including |
Line 114: | Line 112: |
This produces the (binar) file ''<base-name>.docs.DB'' which provides an efficient mapping | This produces the (binary) file ''<base-name>.docs.DB'' which provides an efficient mapping |
Line 136: | Line 134: |
curl "http://localhost:8888/?q=die*&h=1&c=3" | curl "http://localhost:8888/?q=doc*&h=1&c=3" |
CompleteSearch
Quick Intro
Follow these steps to checkout the CompleteSearch code from our SVN, compile it, build an index, run a server for that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>.
0. Get source code
svn checkout https://ad-svn.informatik.uni-freiburg.de/completesearch/codebase Username: [ask us] Password: [ask us]
Third-party code you need to install:
sudo apt-get install libboost-dev # Boost (http://www.boost.org) sudo apt-get install libsparsehash-dev # Google Hash Map (http://code.google.com/p/google-sparsehash) sudo apt-get install libgtest-dev # Google Test (http://code.google.com/p/googletest) sudo apt-get install libstxxl-dev # STXXL (http://stxxl.sourceforge.net)
1. Compile
Edit the Makefile and adjust CS_CODE_DIR and STXXL_CONFIG, then do:
make build-all
This will build three binaries: buildIndex, buildDocsDB, startCompletionServer.
If you call any of these binaries without parameters you will get usage info with all the available options.
2. Parse
Use our generic XML Parser (see codebase/parser/XmlParserNewExampleMain.cpp for an example usage) or our generic CSV Parser (codebase/parser/CsvParserMain) or your own parser to produce the following two intermediate files.
1. The file <base-name>.docs, with lines of the form
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document>
This file must be sorted so that sort -c -k1,1n does not complain. Here is a simple example (the multi-spaces are all TABs)
1 u:http://some.url.wherever/foo t:First document H: This is a stupid document. 2 u:http://some.url.wherever/bar t:Second document H: This is a boring document.
2. The file <base-name>.words, with lines of the form
<word><TAB><doc id><TAB><score><TAB><position>
This file must be sorted such that sort -c -k1,1 -k2,2n -k4,4n does not complain. Here is a simple example, matching the example above (again multi-spaces are all TABs):
a 1 1 5 a 2 1 5 boring 2 1 6 document 1 2 2 document 1 1 7 document 2 2 2 document 2 1 7 first 1 1 1 is 1 1 4 is 2 1 4 second 2 1 1 stupid 1 1 6 this 1 1 3 this 2 1 3
3. Build the word index
buildIndex HYB <base-name>.words
This produces the (binary) index file <base-name>.hybrid. It enables fast processing of the powerful query language offered by CompleteSearch (including full-text search, prefix search and completion, synonym search, error-tolerant search, etc.).
buildIndex also produces the file <base-name>.vocabulary that provides the mapping from word ids to words. This is an ASCII file, you can just look at it.
Note that by default, the HYB index is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). TODO: say something about this here, it's actually quite easy.
4. Build the doc index
buildDocsDB <name>.docs
This produces the (binary) file <base-name>.docs.DB which provides an efficient mapping from doc ids to documents. This is needed if you want to show excerpts / snippets from documents matching the query (which is almost always the case).
5. Start server
startCompletionServer -Z <base-name>.hybrid
This starts the server. If you run it without argument, it prints usage information and shows you the (very many) command line options. The -Z argument lets the server run in the foreground, and output everything to the console, which is convenient for testing. The default mode is to run as a background process and write all output to a log file.
6. Queries
The server listens on the port you specified in step 6 (8888 by default), and speaks HTTP. For example:
curl "http://localhost:8888/?q=doc*&h=1&c=3"
This will return the result as an XML, which should be self-explanatory.
Here is the list of parameters which you may pass along with the query (q=...)
- h : number of hits
- c : number of completions (of last query word, if you put a * behind it)
- f : send hits starting from this one (default: 0)
- en : number of excerpts per hit
- er : excerpt radius (number of words to the left and right of matching words)
- rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending)
- rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above)
- s : how to aggregate scores (expert option, ignore for the moment)
- format : one of xml, json, jsonp. Return result in that format.
More detailed information
Here is the link to the old Wiki from the MPII. This contains lots of detailed information, but most of this is really for developers of the code. For building applications, the above should be enough.