|
Size: 1032
Comment: Comments about section ".docs"
|
Size: 1088
Comment: Comments about section ".docs"
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 18: | Line 18: |
| How do we compute the doc id? What's the difference to the line number? How do we compute the title? What chars are allowed? |
|
| Line 19: | Line 23: |
How do we comoute the doc id? What's the difference to the line number? |
The raw data
"The raw data may be anything: [...] It is the job of the parser to process the raw data and produce the following files". If the raw data really may be anything, then the parser must be able to parse anything. This is quite impossible.
<collection_name>.vocabulary
<word written in ASCII>: What to do with words that have non-ASCII chars in the input document?
Terms "word" and "non-word": Please give an exact definition.
<collection_name>.docs (TODO: rename to <collection_name>.docs_ascii)
I've put the TODO onto the TODO page of the wiki. OK? Shall we elminate it here?
How do we compute the doc id? What's the difference to the line number?
How do we compute the title? What chars are allowed?
H: Is this the textual content of a page, that is, with all (HTML) markup removed? Or is it guaranteed that <page> elements in the input have no further subelements? I think it should be specified in the section on the parser what exactly it has to do to produce H. Put differently: what is the format of H?