Differences between revisions 11 and 159 (spanning 148 versions)
Revision 11 as of 2007-05-28 15:08:27
Size: 1661
Editor: dslb-084-058-237-229
Comment:
Revision 159 as of 2012-01-25 21:40:48
Size: 4611
Editor: Hannah Bast
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= About character encoding (28May07 Markus) = #acl All:read
Line 3: Line 3:
CompletionSearch supports ISO-8859-1 and the multibyte character encoding UTF-8.
UTF-8 is the default encoding with the following consequences:

 * The $AC->settings->encoding is 'utf-8' unless overriden in autocomplete_config.php
 * The texts in text.php are saved as UTF-8
 * The css file uses '@charset "utf-8";'
 * We use mb_strtolower (instead of strtolower) with parameter $AC->settings->encoding to enable UTF-8

We do the following depending on the defined encoding:
 * We UTF-8 encode $AC->settings->capitals if $AC->settings->encoding is UTF-8
 * In ajax.php we UTF-8 encode the query string if $AC->settings->encoding is UTF-8 and the charset of content_type is not UTF-8 (means the request is sent as a non-UTF-8 type)
 * We set the page encoding of index.php, options.php and change_options.php according to $AC->settings->encoding (<meta http-equiv="content-type" content="text/html;charset=<?php echo $AC->settings->encoding; ?>">
)
 * Texts from text.php are UTF-8 decoded by $AC->get_text() if $AC->settings->encoding is ISO-8859-1
 * We url encode the javascript code in function javascript_rhs (in generate_javascript.php) if $AC->settings->encoding is not UTF-8 (this is not necessary if utf-8 is used)
= CompleteSearch =
Line 20: Line 6:
== Quick Intro ==
Line 21: Line 8:
== The PHP Apache extension php_mbstring == Follow these steps to checkout the !CompleteSearch code from our SVN, compile it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>.
Line 23: Line 10:
The use of the mb_strtolower function (and other mb_ functions) requires the extension php_mbstring in php.ini: === 0. Get source code ===
Line 26: Line 13:
In windows:
extension=php_mbstring.dll

or in linux:
extension=php_mbstring.so
svn checkout
http://ad-svn.informatik.uni-freiburg.de/completesearch/codebase
Username: [ask us]
Password: [ask us]
Line 33: Line 19:
(On geek, the mb_... functions were available by default, on Markus' laptop the line above had to be added.) === 1. Compile ===

{{{
make all
}}}

This will build three binaries:

 * buildIndex
 * buildDocsDB
 * startCompletionServer

If you call any of these binaries without parameters you will get usage
info with all the available options.

=== 2. Input (to be produced by a suitable parser) ===

'''Input 1:''' a file ''<base-name>.docs'', with lines of the form

{{{
<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw
text of document>
}}}

This file must be sorted so that ''sort -c -k1,1n'' does not complain.

Here is a simple example (the multi-spaces are all TABs)

{{{
1 u:http://some.url.wherever/foo t:First document H: This is a stupid document.
2 u:http://some.url.wherever/bar t:Second document H: This is a boring document.
}}}


'''Ínput 1:''' a file ''<base-name>.words'', with lines of the form

{{{
<word><TAB><doc id><TAB><score><TAB><position>
}}}

This file must be sorted such that ''sort -c -k1,1 -k2,2n -k4,4n'' does not complain.

Here is a simple example, matching the example above (again multi-spaces are all TABs):

{{{
a 1 1 5
a 2 1 5
boring 2 1 6
document 1 2 2
document 1 1 7
document 2 2 2
document 2 1 7
first 1 1 1
is 1 1 4
is 2 1 4
second 2 1 1
stupid 1 1 6
this 1 1 3
this 2 1 3
}}}


=== 3. Build the word index ===

{{{
buildIndex HYB <base-name>.words
}}}

This produces the (binary) index file ''<base-name>.hybrid''.
It enables fast processing of the powerful query language offered by CompleteSearch (including
full-text search, prefix search and completion, synonym search, error-tolerant search,
etc.).

''buildIndex'' also produces the file ''<base-name>.vocabulary'' that provides
the mapping from word ids to words. This is an ASCII file, you can just look at it.

Note that by default, the HYB index is built with block of fixed sizes. It is more
efficient though to pass it an explicit list of block boundaries (-B
option). TODO: say something about this here, it's actually quite easy.

=== 4. Build the doc index ===

{{{
buildDocsDB <name>.docs
}}}

This produces the (binar) file ''<base-name>.docs.DB'' which provides an efficient mapping
from doc ids to documents. This is needed if you want to show excerpts / snippets
from documents matching the query (which is almost always the case).

=== 5. Start server ===

{{{
startCompletionServer -Z <base-name>.hybrid
}}}

This starts the server. If you run it without argument, it prints usage
information and shows you the (very many) command line options. The ''-Z''
argument lets the server run in the foreground, and
output everything to the console, which is convenient for testing. The default
mode is to run as a background process and write all output to a log file.

=== 6. Queries ===

The server listens on the port you specified in step 6 (''8888'' by
default), and speaks ''HTTP''. For example:

{{{
curl "http://localhost:8888/?q=die*&h=1&c=3"
}}}

This will return the result as an XML, which should be self-explanatory.

Here is the list of parameters which you may pass along with the query
(q=...)

 * h : number of hits
 * c : number of completions (of last query word, if you put a * behind it)
 * f : send hits starting from this one (default: 0)
 * en : number of excerpts per hit
 * er : size of excerpt
 * rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending)
 * rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above)
 * s : how to aggregate scores (expert option, ignore for the moment)
 * format : one of xml, json, jsonp. Return result in that format.

== More detailed information ==

Here is the link to the [[MpiiWiki|old Wiki from the MPII]]. This contains lots of detailed information, but most of this is really for developers of the code. For building applications, the above should be enough.

CompleteSearch

Quick Intro

Follow these steps to checkout the CompleteSearch code from our SVN, compile it, build an index, run a server on that index, and ask queries to that server via HTTP. Don't be afraid, it's easy. If you have questions, send an email to Hannah Bast <bast@informatik.uni-freiburg.de>.

0. Get source code

svn checkout
http://ad-svn.informatik.uni-freiburg.de/completesearch/codebase
Username: [ask us]
Password: [ask us]

1. Compile

make all

This will build three binaries:

  • buildIndex
  • buildDocsDB
  • startCompletionServer

If you call any of these binaries without parameters you will get usage info with all the available options.

2. Input (to be produced by a suitable parser)

Input 1: a file <base-name>.docs, with lines of the form

<doc id><TAB>u:<url of document><TAB>t:<title of document><TAB>H:<raw
text of document>

This file must be sorted so that sort -c -k1,1n does not complain.

Here is a simple example (the multi-spaces are all TABs)

1       u:http://some.url.wherever/foo  t:First document        H: This is a stupid document.
2       u:http://some.url.wherever/bar  t:Second document       H: This is a boring document.

Ínput 1: a file <base-name>.words, with lines of the form

<word><TAB><doc id><TAB><score><TAB><position>

This file must be sorted such that sort -c -k1,1 -k2,2n -k4,4n does not complain.

Here is a simple example, matching the example above (again multi-spaces are all TABs):

a         1      1      5
a         2      1      5
boring    2      1      6
document  1      2      2
document  1      1      7
document  2      2      2
document  2      1      7
first     1      1      1
is        1      1      4
is        2      1      4
second    2      1      1
stupid    1      1      6
this      1      1      3
this      2      1      3

3. Build the word index

buildIndex HYB <base-name>.words

This produces the (binary) index file <base-name>.hybrid. It enables fast processing of the powerful query language offered by CompleteSearch (including full-text search, prefix search and completion, synonym search, error-tolerant search, etc.).

buildIndex also produces the file <base-name>.vocabulary that provides the mapping from word ids to words. This is an ASCII file, you can just look at it.

Note that by default, the HYB index is built with block of fixed sizes. It is more efficient though to pass it an explicit list of block boundaries (-B option). TODO: say something about this here, it's actually quite easy.

4. Build the doc index

buildDocsDB <name>.docs

This produces the (binar) file <base-name>.docs.DB which provides an efficient mapping from doc ids to documents. This is needed if you want to show excerpts / snippets from documents matching the query (which is almost always the case).

5. Start server

startCompletionServer -Z <base-name>.hybrid

This starts the server. If you run it without argument, it prints usage information and shows you the (very many) command line options. The -Z argument lets the server run in the foreground, and output everything to the console, which is convenient for testing. The default mode is to run as a background process and write all output to a log file.

6. Queries

The server listens on the port you specified in step 6 (8888 by default), and speaks HTTP. For example:

curl "http://localhost:8888/?q=die*&h=1&c=3"

This will return the result as an XML, which should be self-explanatory.

Here is the list of parameters which you may pass along with the query (q=...)

  • h : number of hits
  • c : number of completions (of last query word, if you put a * behind it)
  • f : send hits starting from this one (default: 0)
  • en : number of excerpts per hit
  • er : size of excerpt
  • rd : how to rank the documents (0 = by score, 1 = by doc id, 2 = by word id, append a or d for ascending or descending)
  • rw : how to rank the words (0 = by score, 1 = by doc count, 2 = by occurrence count, 3 = by word id, 4 = by doc id, append a or d as above)
  • s : how to aggregate scores (expert option, ignore for the moment)
  • format : one of xml, json, jsonp. Return result in that format.

More detailed information

Here is the link to the old Wiki from the MPII. This contains lots of detailed information, but most of this is really for developers of the code. For building applications, the above should be enough.

CompleteSearch: FrontPage (last edited 2017-03-19 13:30:19 by Hannah Bast)