|
Size: 3281
Comment:
|
Size: 3715
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 9: | Line 9: |
| Follow these steps to checkout the CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to <bast@informatik.uni-freiburg.de>. | Follow these steps to checkout the !CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>. |
| Line 11: | Line 11: |
| 0. Get source code | === 0. Get source code === |
| Line 13: | Line 13: |
| {{{ | |
| Line 17: | Line 18: |
| }}} | |
| Line 18: | Line 20: |
| 1. Compile | === 1. Compile === |
| Line 20: | Line 22: |
| {{{ | |
| Line 21: | Line 24: |
| }}} | |
| Line 24: | Line 28: |
| buildIndex buildDocsDB startCompletionServer |
* buildIndex * buildDocsDB * startCompletionServer |
| Line 31: | Line 35: |
| 2. Input (to be produced by a suitable parser) | === 2. Input (to be produced by a suitable parser) === |
| Line 33: | Line 37: |
| a <name>.words file, with lines of the form | Ínput 1: a file ''<base-name>.words'', with lines of the form |
| Line 35: | Line 39: |
| {{{ | |
| Line 36: | Line 41: |
| }}} | |
| Line 37: | Line 43: |
| Must be sorted so that sort -c -k1,1 -k2,2n -k4,4n does not complain. | This file must be sorted such that ''sort -c -k1,1 -k2,2n -k4,4n'' does not complain. |
| Line 39: | Line 45: |
| And a <name>.docs file, with lines of the form | Input 2: a file ''<base-name>.docs'', with lines of the form |
| Line 44: | Line 50: |
| Must be sorted so that sort -c -k1,1n does not complain. | This file must be sorted so that ''sort -c -k1,1n'' does not complain. |
| Line 49: | Line 55: |
| 3. Build the word index | === 3. Build the word index === |
| Line 51: | Line 57: |
| buildIndex HYB <name>.words | {{{ buildIndex HYB <base-name>.words }}} |
| Line 53: | Line 61: |
| This produces the main index file <name>.hybrid needed for prefix search (this is a binary file). It also produces the file <name>.vocabulary, that provides the mapping from word ids to words (it's an ascii file, you can just look at it). |
This produces the (binary) index file ''<base-name>.hybrid''. It enables fast processing of the powerful query language offered by CompleteSearch (including full-text search, prefix search and completion, synonym search, error-tolerant search, etc.). |
| Line 58: | Line 66: |
| Note that by default, HYB is built with block of fixed sizes. It is more | ''buildIndex'' also produces the file ''<base-name>.vocabulary'' that provides the mapping from word ids to words. This is an ASCII file, you can just look at it. Note that by default, the HYB index is built with block of fixed sizes. It is more |
| Line 60: | Line 71: |
| option). Let's talk about this more when efficiency becomes an issue for you. |
option). TODO: say something about this here, it's actually quite easy. |
| Line 63: | Line 73: |
| 4. Build the doc index | === 4. Build the doc index === |
| Line 65: | Line 75: |
| {{{ | |
| Line 66: | Line 77: |
| }}} | |
| Line 67: | Line 79: |
| This produces the file <name>.docs.DB which provides efficient mapping from doc ids to documents. Needed if you want to show excerpts/snippets from documents matching the query. |
This produces the (binar) file ''<base-name>.docs.DB'' which provides an efficient mapping from doc ids to documents. This is needed if you want to show excerpts / snippets from documents matching the query (which is almost always the case). |
| Line 71: | Line 83: |
| 5. Start server | === 5. Start server === |
CompleteSearch
Old Wiki from the MPII (lots of detailed / internal information)
Quick Intro
Follow these steps to checkout the CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>.
0. Get source code
svn checkout http://vulcano.informatik.uni-freiburg.de/svn/completesearch/codebase Username: [ask us] Password: [ask us]
1. Compile
make all
This will build three binaries:
- buildIndex
- buildDocsDB
- startCompletionServer
If you call any of these binaries without parameters you will get usage info with all the available options.
2. Input (to be produced by a suitable parser)
Ínput 1: a file <base-name>.words, with lines of the form
<word><TAB><doc id><TAB><score><TAB><position>
This file must be sorted such that sort -c -k1,1 -k2,2n -k4,4n does not complain.
Input 2: a file <base-name>.docs, with lines of the form
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document>
This file must be sorted so that sort -c -k1,1n does not complain.
You find a very simple example under http://www.mpi-inf.mpg.de/~bast/topsecret/example.tgz
3. Build the word index
buildIndex HYB <base-name>.words
This produces the (binary) index file <base-name>.hybrid. It enables fast processing of the powerful query language offered by CompleteSearch (including full-text search, prefix search and completion, synonym search, error-tolerant search, etc.).
buildIndex also produces the file <base-name>.vocabulary that provides the mapping from word ids to words. This is an ASCII file, you can just look at it.
Note that by default, the HYB index is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). TODO: say something about this here, it's actually quite easy.
4. Build the doc index
buildDocsDB <name>.docs
This produces the (binar) file <base-name>.docs.DB which provides an efficient mapping from doc ids to documents. This is needed if you want to show excerpts / snippets from documents matching the query (which is almost always the case).
5. Start server
startCompletionServer -Z <name>.hybrid
This starts the server. If you run it without argument, it prints usage information. The -Z argument lets the server run in the foreground, and output everything to the console, which is convenient for testing.
6. Queries
The server listens on the port you specified in step 6 (8888 by default), and speaks HTTP. For example:
curl "http://localhost:8888/?q=die*&h=1&c=3"
This will return the result as an XML, which should be self-explanatory.
Here is the list of parameters which you may pass along with the query (q=...)
h : number of hits c : number of completions (of last query word, if you put a * behind it) f : send hits starting from this one (default: 0) en : number of excerpts per hit er : size of excerpt rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending) rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above) s : how to aggregate scores (expert option, ignore for the moment)
Jede Menge Doku auf http://search.mpi-inf.mpg.de/wiki/CompleteSearch