2978
Comment: Tips for writing portable code
|
4435
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
== Documentation (users) == | #acl All:read |
Line 3: | Line 3: |
[wiki:Self:completesearch/Installation Installation Guide] | = CompleteSearch = |
Line 5: | Line 5: |
[wiki:Self:completesearch/Local Set up the system on a local machine (notebook)] | [[MpiiWiki|Old Wiki from the MPII]] (lots of detailed / internal information) |
Line 7: | Line 7: |
[wiki:Self:completesearch/IndexBuilding Index Building: Tools, Formats, etc.] | == Quick Intro == |
Line 9: | Line 9: |
[wiki:Self:completesearch/DocumentFormats Document Formats: .docs, .words, .vocabulary, etc.] | Follow these steps to checkout the !CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>. |
Line 11: | Line 11: |
[wiki:Self:completesearch/CharacterEncoding UTF-8 support for PHP (php_mbstring extension)] | === 0. Get source code === |
Line 13: | Line 13: |
[wiki:Self:completesearch/PHPErrorReporting Error reporting / handling with PHP] | {{{ svn checkout http://ad-svn.informatik.uni-freiburg.de/completesearch/codebase Username: [ask us] Password: [ask us] }}} |
Line 15: | Line 20: |
[wiki:Self:completesearch/Php Problems with PHP] | === 1. Compile === |
Line 17: | Line 22: |
[wiki:Self:completesearch/Glossary Glossary of important terms in search engine technology] | {{{ make all }}} |
Line 19: | Line 26: |
== Documentation (developers) == | This will build three binaries: |
Line 21: | Line 28: |
[wiki:Self:completesearch/SourceCodeOverview Source code overview] | * buildIndex * buildDocsDB * startCompletionServer |
Line 23: | Line 32: |
[http://search.mpi-inf.mpg.de/ir-www/doc Documentation produced by Doxygen] | If you call any of these binaries without parameters you will get usage info with all the available options. |
Line 25: | Line 35: |
[wiki:Self:CodingConventions Coding Conventions] and [wiki:Self:completesearch/DesignConventions OO and C++ Design Conventions] | === 2. Input (to be produced by a suitable parser) === |
Line 27: | Line 37: |
[wiki:Self:completesearch/ProblemsCompiling Typical problems compiling] | '''Input 1:''' a file ''<base-name>.docs'', with lines of the form |
Line 29: | Line 39: |
[wiki:Self:completesearch/Debugging Debugging HowTo] | {{{ <doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document> }}} |
Line 31: | Line 44: |
[wiki:Self:completesearch/GNUBuildSystem Building with autoconf/automake] and [wiki:Self:completesearch/CMakeBuildSystem Building with CMake] | This file must be sorted so that ''sort -c -k1,1n'' does not complain. |
Line 33: | Line 46: |
[wiki:Self:completesearch/MinGW Compiling under MinGW] | Here is a simple example (the multi-spaces are all TABs) |
Line 35: | Line 48: |
[wiki:Self:completesearch/ExcerptGenerator Excerpt Generator requirements] [wiki:Self:completesearch/ExcerptGenerator/ThesisTopic Description for a Master's Thesis on Excerpt Generation] [wiki:Self:Projects/ExcerptGeneration Excerpt Generation Project Overview] [wiki:Self:completesearch/Templates Template peculiarities in the CompleteSearch code] [wiki:Self:completesearch/CVSHistory CVS history (TODO: what's this; is it still used?)] [wiki:Self:completesearch/SyslogdAndLogrotate Using syslogd and logrotate to manage the server's log files] [wiki:Self:completesearch/UsefulLiterature (Secondary) literature that might be useful] [wiki:Self:completesearch/FirefoxBugAbsentCursor Missing cursor on form fields (Firefox bug)] [wiki:Self:completesearch/NotesToJavascript Problem including Javascript from external servers] [wiki:Self:completesearch/CSSBoxModel The CSS Box Model] [wiki:Self:completesearch/QuerySpecification Specification of queries (in German only)] [wiki:Self:completesearch/InitializationOfCompleteSearch The Initialization of CompleteSearch (in German only)] [wiki:Self:completesearch/WritingPortableCode Tips for writing portable code] == HowTos == [wiki:Self:completesearch/SeleniumRC Testing with SeleniumRC] [wiki:Self:completesearch/ModPhpStartetExe Have a click on a link start a local application] [wiki:Self:completesearch/Examples Example programs etc.] [wiki:Self:completesearch/GettingStartedWithDocBook Getting Started With DocBook] |
{{{ 1 u:http://some.url.wherever/foo t:First document H: This is a stupid document. 2 u:http://some.url.wherever/bar t:Second document H: This is a boring document. }}} |
Line 70: | Line 54: |
== ToDos == | '''Ínput 1:''' a file ''<base-name>.words'', with lines of the form |
Line 72: | Line 56: |
[wiki:Self:completesearch/TODO TODO list] | {{{ <word><TAB><doc id><TAB><score><TAB><position> }}} |
Line 74: | Line 60: |
[wiki:Self:NewFeatures New Features that would be nice to have] | This file must be sorted such that ''sort -c -k1,1 -k2,2n -k4,4n'' does not complain. Here is a simple example, matching the example above (again multi-spaces are all TABs): {{{ a 1 1 5 a 2 1 5 boring 2 1 6 document 1 2 2 document 1 1 7 document 2 2 2 document 2 1 7 first 1 1 1 is 1 1 4 is 2 1 4 second 2 1 1 stupid 1 1 6 this 1 1 3 this 2 1 3 }}} === 3. Build the word index === {{{ buildIndex HYB <base-name>.words }}} This produces the (binary) index file ''<base-name>.hybrid''. It enables fast processing of the powerful query language offered by CompleteSearch (including full-text search, prefix search and completion, synonym search, error-tolerant search, etc.). ''buildIndex'' also produces the file ''<base-name>.vocabulary'' that provides the mapping from word ids to words. This is an ASCII file, you can just look at it. Note that by default, the HYB index is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). TODO: say something about this here, it's actually quite easy. === 4. Build the doc index === {{{ buildDocsDB <name>.docs }}} This produces the (binar) file ''<base-name>.docs.DB'' which provides an efficient mapping from doc ids to documents. This is needed if you want to show excerpts / snippets from documents matching the query (which is almost always the case). === 5. Start server === {{{ startCompletionServer -Z <base-name>.hybrid }}} This starts the server. If you run it without argument, it prints usage information and shows you the (very many) command line options. The ''-Z'' argument lets the server run in the foreground, and output everything to the console, which is convenient for testing. The default mode is to run as a background process and write all output to a log file. === 6. Queries === The server listens on the port you specified in step 6 (''8888'' by default), and speaks ''HTTP''. For example: {{{ curl "http://localhost:8888/?q=die*&h=1&c=3" }}} This will return the result as an XML, which should be self-explanatory. Here is the list of parameters which you may pass along with the query (q=...) * h : number of hits * c : number of completions (of last query word, if you put a * behind it) * f : send hits starting from this one (default: 0) * en : number of excerpts per hit * er : size of excerpt * rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending) * rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above) * s : how to aggregate scores (expert option, ignore for the moment) * format : one of xml, json, jsonp. Return result in that format. |
CompleteSearch
Old Wiki from the MPII (lots of detailed / internal information)
Quick Intro
Follow these steps to checkout the CompleteSearch code from our SVN, build it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>.
0. Get source code
svn checkout http://ad-svn.informatik.uni-freiburg.de/completesearch/codebase Username: [ask us] Password: [ask us]
1. Compile
make all
This will build three binaries:
- buildIndex
- buildDocsDB
- startCompletionServer
If you call any of these binaries without parameters you will get usage info with all the available options.
2. Input (to be produced by a suitable parser)
Input 1: a file <base-name>.docs, with lines of the form
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw text of document>
This file must be sorted so that sort -c -k1,1n does not complain.
Here is a simple example (the multi-spaces are all TABs)
1 u:http://some.url.wherever/foo t:First document H: This is a stupid document. 2 u:http://some.url.wherever/bar t:Second document H: This is a boring document.
Ínput 1: a file <base-name>.words, with lines of the form
<word><TAB><doc id><TAB><score><TAB><position>
This file must be sorted such that sort -c -k1,1 -k2,2n -k4,4n does not complain.
Here is a simple example, matching the example above (again multi-spaces are all TABs):
a 1 1 5 a 2 1 5 boring 2 1 6 document 1 2 2 document 1 1 7 document 2 2 2 document 2 1 7 first 1 1 1 is 1 1 4 is 2 1 4 second 2 1 1 stupid 1 1 6 this 1 1 3 this 2 1 3
3. Build the word index
buildIndex HYB <base-name>.words
This produces the (binary) index file <base-name>.hybrid. It enables fast processing of the powerful query language offered by CompleteSearch (including full-text search, prefix search and completion, synonym search, error-tolerant search, etc.).
buildIndex also produces the file <base-name>.vocabulary that provides the mapping from word ids to words. This is an ASCII file, you can just look at it.
Note that by default, the HYB index is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). TODO: say something about this here, it's actually quite easy.
4. Build the doc index
buildDocsDB <name>.docs
This produces the (binar) file <base-name>.docs.DB which provides an efficient mapping from doc ids to documents. This is needed if you want to show excerpts / snippets from documents matching the query (which is almost always the case).
5. Start server
startCompletionServer -Z <base-name>.hybrid
This starts the server. If you run it without argument, it prints usage information and shows you the (very many) command line options. The -Z argument lets the server run in the foreground, and output everything to the console, which is convenient for testing. The default mode is to run as a background process and write all output to a log file.
6. Queries
The server listens on the port you specified in step 6 (8888 by default), and speaks HTTP. For example:
curl "http://localhost:8888/?q=die*&h=1&c=3"
This will return the result as an XML, which should be self-explanatory.
Here is the list of parameters which you may pass along with the query (q=...)
- h : number of hits
- c : number of completions (of last query word, if you put a * behind it)
- f : send hits starting from this one (default: 0)
- en : number of excerpts per hit
- er : size of excerpt
- rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending)
- rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above)
- s : how to aggregate scores (expert option, ignore for the moment)
- format : one of xml, json, jsonp. Return result in that format.