2721
Comment:
|
3281
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
== Documentation (users) == | #acl All:read |
Line 3: | Line 3: |
[wiki:Self:completesearch/Installation Installation Guide] | = CompleteSearch = |
Line 5: | Line 5: |
[wiki:Self:completesearch/Local Set up the system on a local machine (notebook)] | [[MpiiWiki|Old Wiki from the MPII]] (lots of detailed / internal information) |
Line 7: | Line 7: |
[wiki:Self:completesearch/IndexBuilding Index Building: Tools, Formats, etc.] | == Quick Intro == |
Line 9: | Line 9: |
[wiki:Self:completesearch/DocumentFormats Document Formats: .docs, .words, .vocabulary, etc.] | Follow these steps to checkout the CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to <bast@informatik.uni-freiburg.de>. |
Line 11: | Line 11: |
[wiki:Self:completesearch/CharacterEncoding UTF-8 support for PHP (php_mbstring extension)] | 0. Get source code |
Line 13: | Line 13: |
[wiki:Self:completesearch/PHPErrorReporting Error reporting / handling with PHP] | svn checkout http://vulcano.informatik.uni-freiburg.de/svn/completesearch/codebase Username: [ask us] Password: [ask us] |
Line 15: | Line 18: |
[wiki:Self:completesearch/Php Problems with PHP] | 1. Compile |
Line 17: | Line 20: |
[wiki:Self:completesearch/NotesToJavascript Problem including Javascript from external servers] | make all |
Line 19: | Line 22: |
[wiki:Self:completesearch/CSSBoxModel The CSS Box Model] | This will build three binaries: |
Line 21: | Line 24: |
buildIndex buildDocsDB startCompletionServer |
|
Line 22: | Line 28: |
== Documentation (developers) == | If you call any of these binaries without parameters you will get usage info with all the available options. |
Line 24: | Line 31: |
[wiki:Self:completesearch/SourceCodeOverview Source code overview] | 2. Input (to be produced by a suitable parser) |
Line 26: | Line 33: |
[http://search.mpi-inf.mpg.de/ir-www/doc Documentation produced by Doxygen] | a <name>.words file, with lines of the form |
Line 28: | Line 35: |
[wiki:Self:CodingConventions Coding Conventions] and [wiki:Self:completesearch/DesignConventions OO and C++ Design Conventions] | <word><TAB><doc id><TAB><score><TAB><position> |
Line 30: | Line 37: |
[wiki:Self:completesearch/ProblemsCompiling Typical problems compiling] | Must be sorted so that sort -c -k1,1 -k2,2n -k4,4n does not complain. |
Line 32: | Line 39: |
[wiki:Self:completesearch/Debugging Debugging HowTo] | And a <name>.docs file, with lines of the form |
Line 34: | Line 41: |
[wiki:Self:completesearch/GNUBuildSystem Building with autoconf/automake] and [wiki:Self:completesearch/CMakeBuildSystem Building with CMake] | <doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document> |
Line 36: | Line 44: |
[wiki:Self:completesearch/MinGW Compiling under MinGW] | Must be sorted so that sort -c -k1,1n does not complain. |
Line 38: | Line 46: |
[wiki:Self:completesearch/ExcerptGenerator Excerpt Generator requirements] | You find a very simple example under http://www.mpi-inf.mpg.de/~bast/topsecret/example.tgz |
Line 40: | Line 49: |
[wiki:Self:completesearch/ExcerptGenerator/ThesisTopic Description for a Master's Thesis on Excerpt Generation] [wiki:Self:Projects/ExcerptGeneration Excerpt Generation Project Overview] | 3. Build the word index |
Line 42: | Line 51: |
[wiki:Self:completesearch/Templates Template peculiarities in the CompleteSearch code] | buildIndex HYB <name>.words |
Line 44: | Line 53: |
[wiki:Self:completesearch/CVSHistory CVS history (TODO: what's this; is it still used?)] | This produces the main index file <name>.hybrid needed for prefix search (this is a binary file). It also produces the file <name>.vocabulary, that provides the mapping from word ids to words (it's an ascii file, you can just look at it). |
Line 46: | Line 58: |
[wiki:Self:completesearch/SyslogdAndLogrotate Using syslogd and logrotate to manage the server's log files] | Note that by default, HYB is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). Let's talk about this more when efficiency becomes an issue for you. |
Line 48: | Line 63: |
[wiki:Self:completesearch/UsefulLiterature (Secondary) literature that might be useful] | 4. Build the doc index |
Line 50: | Line 65: |
[wiki:Self:completesearch/FirefoxBugAbsentCursor Missing cursor on form fields (Firefox bug)] | buildDocsDB <name>.docs |
Line 52: | Line 67: |
[wiki:Self:completesearch/QuerySpecification Specification of queries (in German only)] | This produces the file <name>.docs.DB which provides efficient mapping from doc ids to documents. Needed if you want to show excerpts/snippets from documents matching the query. |
Line 54: | Line 71: |
[wiki:Self:completesearch/InitializationOfCompleteSearch The Initialization of CompleteSearch (in German only)] | 5. Start server |
Line 56: | Line 73: |
startCompletionServer -Z <name>.hybrid | |
Line 57: | Line 75: |
== HowTos == | This starts the server. If you run it without argument, it prints usage information. The -Z argument lets the server run in the foreground, and output everything to the console, which is convenient for testing. |
Line 59: | Line 79: |
[wiki:Self:completesearch/SeleniumRC Testing with SeleniumRC] | 6. Queries |
Line 61: | Line 81: |
[wiki:Self:completesearch/ModPhpStartetExe Have a click on a link start a local application] | The server listens on the port you specified in step 6 (8888 by default), and speaks HTTP. For example: |
Line 63: | Line 84: |
[wiki:Self:completesearch/Examples Example programs etc.] | curl "http://localhost:8888/?q=die*&h=1&c=3" |
Line 65: | Line 86: |
This will return the result as an XML, which should be self-explanatory. | |
Line 66: | Line 88: |
== ToDos == | Here is the list of parameters which you may pass along with the query (q=...) |
Line 68: | Line 91: |
[wiki:Self:completesearch/TODO TODO list] | h : number of hits c : number of completions (of last query word, if you put a * behind it) f : send hits starting from this one (default: 0) en : number of excerpts per hit er : size of excerpt rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending) rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above) s : how to aggregate scores (expert option, ignore for the moment) |
Line 70: | Line 102: |
[wiki:Self:NewFeatures New Features that would be nice to have] | Jede Menge Doku auf http://search.mpi-inf.mpg.de/wiki/CompleteSearch |
CompleteSearch
Old Wiki from the MPII (lots of detailed / internal information)
Quick Intro
Follow these steps to checkout the CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to <bast@informatik.uni-freiburg.de>.
0. Get source code
svn checkout http://vulcano.informatik.uni-freiburg.de/svn/completesearch/codebase Username: [ask us] Password: [ask us]
1. Compile
make all
This will build three binaries:
buildIndex buildDocsDB startCompletionServer
If you call any of these binaries without parameters you will get usage info with all the available options.
2. Input (to be produced by a suitable parser)
a <name>.words file, with lines of the form
<word><TAB><doc id><TAB><score><TAB><position>
Must be sorted so that sort -c -k1,1 -k2,2n -k4,4n does not complain.
And a <name>.docs file, with lines of the form
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document>
Must be sorted so that sort -c -k1,1n does not complain.
You find a very simple example under http://www.mpi-inf.mpg.de/~bast/topsecret/example.tgz
3. Build the word index
buildIndex HYB <name>.words
This produces the main index file <name>.hybrid needed for prefix search (this is a binary file). It also produces the file <name>.vocabulary, that provides the mapping from word ids to words (it's an ascii file, you can just look at it).
Note that by default, HYB is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). Let's talk about this more when efficiency becomes an issue for you.
4. Build the doc index
buildDocsDB <name>.docs
This produces the file <name>.docs.DB which provides efficient mapping from doc ids to documents. Needed if you want to show excerpts/snippets from documents matching the query.
5. Start server
startCompletionServer -Z <name>.hybrid
This starts the server. If you run it without argument, it prints usage information. The -Z argument lets the server run in the foreground, and output everything to the console, which is convenient for testing.
6. Queries
The server listens on the port you specified in step 6 (8888 by default), and speaks HTTP. For example:
curl "http://localhost:8888/?q=die*&h=1&c=3"
This will return the result as an XML, which should be self-explanatory.
Here is the list of parameters which you may pass along with the query (q=...)
h : number of hits c : number of completions (of last query word, if you put a * behind it) f : send hits starting from this one (default: 0) en : number of excerpts per hit er : size of excerpt rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending) rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above) s : how to aggregate scores (expert option, ignore for the moment)
Jede Menge Doku auf http://search.mpi-inf.mpg.de/wiki/CompleteSearch