= Table of Contents = [[TableOfContents]] '''Components:''' Complete``Search has three components '''1. Index Building''' Tools for building, from a collection of documents, the index files required for fast query processing '''2. Completion Server:''' This is a (multi-threaded) C++ program that listens to queries on a specified port, and answers with the help of the previously built index. '''3. Web Interface:''' [PHP code (apache) + Java``Script (client), TODO: explain] In the following, it is (briefly) explained how to install and work with each of these components. [[Anchor(indexbuilding)]] == 1 Index building == === 1.1 Parsing === {{{ svn checkout file:///KM/ir/SVN/xmlparser vim .parse.cpp g++ -o .parse .parse.cpp xmlparser/XmlParser.o .parse .xml }}} This should produce a .words file and a .docs file. Actually it doesn't matter how they are produced, as long as they adhere to the format [TODO: explain]. [''Joachim: The XML input format for the parser is described on CompleteSearch/DocumentFormats. Copy the file exampleParser.cpp to .parser.cpp, adjust the makefile and look at the code. The most important method is getItem("elementname") with which you can extract the contents of an and produce 1 line in the .words file per word contained in this element. If the XML input contains several consecutive s, you can iterate over these by getItems("elementname").''] [''Joachim: If you do not specify the name of the .words and .docs file on the command line, two files named collection.docs and collection.words_unsorted will be created. So you might want to call your parser like this: `$ RollingStones.parse RollingStones.xml RollingStones.docs RollingStones.words_unsorted` ''] === 1.2 Sorting === {{{ sort -k1,1 -k2,2n -k4,4n .words_unsorted > .words sort -k1,1n .docs_unsorted > .docs }}} In case the .words and .docs file are not sorted acoording to the format specifications [TODO: link], this would be a way to sort them. If the files were produced with the help of xmlparser above (or by any parser which just outputs things in the order it encounters them), sorting is necessary. === 1.3 Words Index (.hybrid and .vocabulary) === [''Joachim: For the following to work, you first have to checkout the `autocompletion` folder: `$ svn checkout file:///KM/ir/SVN/autocompletion` ''] {{{ make buildIndex buildIndex HYB .words }}} [''Joachim: The .hybrid file is the actual index. .vocabulary serves to translate word -> word ID. And I do not understand the next two sentences.''] For good efficiency it is actually a bit more tricky (good choice block boundaries is important). TODO: take this burden from the user, and incorporate it into buildIndex === 1.4 Docs Index (.docs.DB) === {{{ make buildDocsDB buildDocsDB .docs }}} Run without arguments for detailed usage information about command line options etc. == 2. Completion Server (C++) == === 2.1 Get it === {{{ svn checkout file:///KM/ir/SVN/autocompletion }}} === 2.2 Compile it === {{{ make startCompletionServer }}} === 2.3 Run it === {{{ startCompletionServer -p -l .hybrid }}} call without arguments for detailed usage information requires three files: .hybrid (the HYB index in some binary format), .vocabulary (list of distinct words in sorted order), .docs.db (url + title + raw text of the documents) See Section [#indexbuilding Index building] on how to produce these files == 3. Web Interface == === 3.1 Get it === {{{ svn checkout file:///KM/ir/SVN/autocomplete-php }}} contains three components: 3.1.1 application-independent code: the ''autocomplete'' directory 3.1.2 application-specific code: the files ''index.php'', ''autocomplete.css'', ''autocomplete_config.php'' 3.1.3 web-server configuration details: the file ''STILL_HAS_TO_BE_WRITTEN.txt'' === 3.2 Configure it === {{{ mkdir cp index.php autocomplete.css autocomplete_config.php }}} should be a directory served by your web server. The search engine will then be accessible unter http:///.../ {{{ vim autocomplete_config.php }}} The config file is just a list of variable assignements. It is well-documented and clearly says [TODO: not yet] which parts you must adapt (few), and which you can just leave to their default values (you can always come back and change them later). {{{ vim }}} Adapt it as explained in the web-server configuration file from 2.1.3 above. You must have Apache + PHP installed (PHP4 or PHP5 are both fine). [TODO: give details on how to install it. For Debian, it's a simple package ''libapache2-mod-php4''. For windows there is an installer ''[TODO]''.] == Installing under Windows == === Getting the source code === This section tells you how to install a subproject such as {{{autocompletion}}}. Install the [http://tortoisesvn.tigris.org/ Tortoise] SVN client. Make sure that you can login with your normal user account via ssh to your SVN server machine ''without getting any error message printed to STDERR''. (If this happens, Tortoise will terminate the connection immediately.) Open Windows Explorer, press the right mouse button, and select "SVN Checkout". (Tortoise is fully integrated into Explorer. If the "SVN Checkout" menu does not appear after the right click, go to any non-special directory such as "My Documents".) As the URL of the repository, enter {{{ svn+ssh://ziegler@contact.mpi-sb.mpg.de/KM/ir/SVN/autocompletion }}} wherein you have to substitute {{{ziegler}}} for your login name on contact and {{{autocompletion}}} for the subproject that you want to checkout. As the checkout directory, give the path to the directory that you want the subproject to be checked out to; this path should end with the name of the subproject, for example, {{{ D:\MyProjects\autocompletion }}} You will notice that Tortoise asks you 3 times for your password. Later, when you do your daily work with Tortoise, you will notice that this happens on ''each and every action that accesses the remote repository''. If this is annoying to you (and it surely will be), you can follow the instructions given in the [http://tortoisesvn.net/ssh_howto TortoiseSVN SSH Howto]. See also this [http://subversion.tigris.org/faq.html#ssh-auth-cache FAQ]: "When I access a repository using svn+ssh, my password is not cached in ~/.subversion/auth/. How do I avoid having to type it so often?" A short summary of this procedure: * On any Unix host, cd to your ~/.ssh * {{{$ ssh-keygen -b 1024 -t dsa -f mykey}}} * {{{$ cat mykey.pub >> authorized_keys}}} * Transfer the file {{{mykey}}} to your windows machine into a directory only readable by you, and remove it on the Unix host * On your windows machine, install the programs puttygen and pageant from [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html here] * Open puttygen, load the file {{{mykey}}}, and save the generated key as file {{{mykey.ppk}}} * Open pageant. It will be running in the system tray. Go to "Add Key", enter {{{mykey.ppk}}} * Start working with Tortoise without having to enter any password * You must keep pageant running while working with Tortoise To perform any of the usual SVN client actions on a file, right click the filename in Explorer and select the TortoiseSVN submenu.