| <?xml version="1.0" encoding="UTF-8"?> |
| |
| <document> |
| <properties> |
| <title>Plan for enhancements to Lucene</title> |
| <authors> |
| <person email="acoliver@apache.org" name="Andrew C. Oliver" id="AO"/> |
| </authors> |
| </properties> |
| <body> |
| |
| <section name="Purpose"> |
| <p> |
| The purpose of this document is to outline plans for |
| making <a href="http://jakarta.apache.org/lucene"> |
| Jakarta Lucene</a> work as a more general drop-in |
| component. It makes the assumption that this is an |
| objective for the Lucene user and development community. |
| </p> |
| <p> |
| The best reference is <a href="http://www.htdig.org"> |
| htDig</a>, though it is not quite as sophisticated as |
| Lucene, it has a number of features that make it |
| desirable. It however is a traditional c-compiled app |
| which makes it somewhat unpleasant to install on some |
| platforms (like Solaris!). |
| </p> |
| <p> |
| This plan is being submitted to the Lucene developer |
| community for an initial reaction, advice, feedback and |
| consent. Following this it will be submitted to the |
| Lucene user community for support. Although, I'm (Andy |
| Oliver) capable of providing these enhancements by |
| myself, I'd of course prefer to work on them in concert |
| with others. |
| </p> |
| <p> |
| While I'm outlaying a fairly large feature set, these can |
| be implemented incrementally of course (and are probably |
| best if done that way). |
| </p> |
| </section> |
| |
| <section name="Goal and Objectives"> |
| <p> |
| The goal is to provide features to Lucene that allow it |
| to be used as a drop-in search engine. It should provide |
| many of the features of projects like <a |
| href="http://www.htdig.org">htDig</a> while surpassing |
| them with unique Lucene features and capabilities such as |
| easy installation on and java-supporting platform, |
| and support for document fields and field searches. And |
| of course, <a href="http://apache.org/LICENSE"> |
| a pragmatic software license</a>. |
| </p> |
| <p> |
| To reach this goal we'll implement code to support the |
| following objectives that augment but do not replace |
| the current Lucene feature set. |
| </p> |
| <ul> |
| <li> |
| Document Location Independence - meaning mapping |
| real contexts to runtime contexts. |
| Essentially, if the document is at |
| /var/www/htdocs/mydoc.html, I probably want it |
| indexed as |
| http://www.bigevilmegacorp.com/mydoc.html. |
| </li> |
| <li> |
| Standard methods of creating central indicies - |
| file system indexing is probably less useful in |
| many environments than is *remote* indexing (for |
| instance http). I would suggest that most folks |
| would prefer that general functionality be |
| supported by Lucene instead of having to write |
| code for every indexing project. Obviously, if |
| what they are doing is *special* they'll have to |
| code, but general document indexing across |
| web servers would not qualify. |
| </li> |
| <li> |
| Document interpretation abstraction - currently |
| one must handle document object construction via |
| custom code. A standard interface for plugging |
| in format handlers should be supported. |
| </li> |
| <li> |
| Mime and file-extension to document |
| interpretation mapping. |
| </li> |
| </ul> |
| </section> |
| <section name="Crawlers"> |
| <p> |
| Crawlers are data source executable code. They crawl a file |
| system, ftp site, web site, etc. to create the index. |
| These standard crawlers may not make ALL of Lucene's |
| functionality available, though they should be able to |
| make most of it available through configuration. |
| </p> |
| <!--<section name="AbstractIndexer">--> |
| <p> |
| <b> Abstract Crawler </b> |
| </p> |
| <p> |
| The AbstractCrawler is basically the parent for all |
| Crawler classes. It provides implementation for the |
| following functions/properties: |
| </p> |
| <ul> |
| <li> |
| index path - where to write the index. |
| </li> |
| <li> |
| cui - create or update the index |
| </li> |
| <li> |
| root context - the start of the pathname |
| that should be replaced by the |
| replace with property or dropped |
| entirely. Example: /opt/tomcat/webapps |
| </li> |
| <li> |
| replace with - when specified replaces |
| the root context. Example: |
| http://jakarta.apache.org. |
| </li> |
| <li> |
| replacement type - the type of |
| replace with path: relative, URL or |
| path. |
| </li> |
| <li> |
| location - the location to start |
| indexing at. |
| </li> |
| <li> |
| doctypes - only index documents with |
| these doctypes. If not specified all |
| registered mime-types are used. |
| Example: "xml,doc,html" |
| </li> |
| <li> |
| recursive - if not specified is turned |
| off. |
| </li> |
| <li> |
| level - optional level of directory or |
| links to traverse. By default is |
| assumed to be infinite. Recursive must |
| be turned on or this is ignored. Range: |
| 0 - Long.MAX_VALUE. |
| </li> |
| <li> |
| SleeptimeBetweenCalls - can be used to |
| avoid flooding a machine with too many |
| requests |
| </li> |
| <li> |
| RequestTimeout - kill the crawler |
| request after the specified period of |
| inactivity. |
| </li> |
| <li> |
| IncludeFilter - include only items |
| matching filter. (can occur multiple |
| times) |
| </li> |
| <li> |
| ExcludeFilter - exclude only items |
| matching filter. (can occur multiple |
| times) |
| </li> |
| <li> |
| ExpandOnly - use but do not index items |
| that match this pattern (regex?) (can |
| occur multiple times) |
| </li> |
| <li> |
| NoExpand - Index but do not follow the |
| links in items that match this pattern |
| (regex?) (can occur multiple times) |
| </li> |
| <li> |
| MaxItems - stops indexing after x |
| documents have been indexed. |
| </li> |
| <li> |
| MaxMegs - stops indexing after x megs |
| have been indexed.. (should this be in |
| specific crawlers?) |
| </li> |
| <li> |
| properties - in addition to the settings |
| (probably from the command line) read |
| this properties file and get them from |
| it. Command line options override |
| the properties file in the case of |
| duplicates. There should also be an |
| environment variable or VM parameter to |
| set this. |
| </li> |
| </ul> |
| <!--</section>--> |
| <!--<s2 title="FileSystemIndexer">--> |
| <p> |
| <b>FileSystemCrawler</b> |
| </p> |
| <p> |
| This should extend the AbstractCrawler and |
| support any additional options required for a |
| file system index. |
| </p> |
| <!--</s2>--> |
| <!--<s2 title="HTTPIndexer">--> |
| <p> |
| <b>HTTP Crawler </b> |
| </p> |
| <p> |
| Supports the AbstractCrawler options as well as: |
| </p> |
| <ul> |
| <li> |
| span hosts - Whether to span hosts or not, |
| by default this should be no. |
| </li> |
| <li> |
| restrict domains - (ignored if span |
| hosts is not enabled). Whether all |
| spanned hosts must be in the same domain |
| (default is off). |
| </li> |
| <li> |
| try directories - Whether to attempt |
| directory listings or not (so if you |
| recurse and go to |
| /nextcontext/index.html this option says |
| to also try /nextcontext to get the dir |
| listing) |
| </li> |
| <li> |
| map extensions - |
| (always/default/never/fallback). Whether |
| to always use extension mapping, by |
| default (fallback to mime type), NEVER |
| or fallback if mime is not available |
| (default). |
| </li> |
| <li> |
| ignore robots - ignore robots.txt, on or |
| off (default - off) |
| </li> |
| </ul> |
| <!-- </s2> --> |
| </section> |
| |
| <section name="MIMEMap"> |
| <p> |
| A configurable registry of document types, their |
| description, an identifier, mime-type and file |
| extension. This should map both MIME -> factory |
| and extension -> factory. |
| </p> |
| <p> |
| This might be configured at compile time or by a |
| properties file, etc. For example: |
| </p> |
| <table> |
| <tr> |
| <td>Description</td> |
| <td>Identifier</td> |
| <td>Extensions</td> |
| <td>MimeType</td> |
| <td>DocumentFactory</td> |
| </tr> |
| <tr> |
| <td>"Word Document"</td> |
| <td>"doc"</td> |
| <td>"doc"</td> |
| <td>"vnd.application/ms-word"</td> |
| <td>POIWordDocumentFactory</td> |
| </tr> |
| <tr> |
| <td>"HTML Document"</td> |
| <td>"html"</td> |
| <td>"html,htm"</td> |
| <td></td> |
| <td>HTMLDocumentFactory</td> |
| </tr> |
| </table> |
| </section> |
| <section name="DocumentFactory"> |
| <p> |
| An interface for classes which create document objects |
| for particular file types. Examples: |
| HTMLDocumentFactory, DOCDocumentFactory, |
| XLSDocumentFactory, XML DocumentFactory. |
| </p> |
| </section> |
| <section name="FieldMapping classes"> |
| <p> |
| A class that maps standard fields from the |
| DocumentFactories into *fields* in the Document objects |
| they create. I suggest that a regular expression system |
| or xpath might be the most universal way to do this. |
| For instance if perhaps I had an XML factory that |
| represented XML elements as fields, I could map content |
| from particular fields to their fields or suppress them |
| entirely. We could even make this configurable. |
| </p> |
| <p> |
| |
| for example: |
| </p> |
| <ul> |
| <li> |
| htmldoc.properties |
| </li> |
| <li> |
| suppress=* |
| </li> |
| <li> |
| author=content:g/author\:\ ........................................./ |
| </li> |
| <li> |
| author.suppress=false |
| </li> |
| <li> |
| title=content:g/title\:\ ........................................./ |
| </li> |
| <li> |
| title.suppress=false |
| </li> |
| </ul> |
| <p> |
| In this example we map html documents such that all |
| fields are suppressed but author and title. We map |
| author and title to anything in the content matching |
| author: (and x characters). Okay my regular expresions |
| suck but hopefully you get the idea. |
| </p> |
| </section> |
| <section name="Final Thoughts"> |
| <p> |
| We might also consider eliminating the DocumentFactory |
| entirely by making an AbstractDocument from which the |
| current document object would inherit from. I |
| experimented with this locally, and it was a relatively |
| minor code change and there was of course no difference |
| in performance. The Document Factory classes would |
| instead be instances of various subclasses of |
| AbstractDocument. |
| </p> |
| <p> |
| My inspiration for this is HTDig (http://www.htdig.org/). |
| While this goes slightly beyond what HTDig provides by |
| providing field mapping (where HTDIG is just interested |
| in Strings/numbers wherever they are found), it provides |
| at least what I would need to use this as a drop-in for |
| most places I contract at (with the obvious exception of |
| a default set of content handlers which would of course |
| develop naturally over time). |
| </p> |
| <p> |
| I am able to certainly contribute to this effort if the |
| development community is open to it. I'd suggest we do |
| it iteratively in stages and not aim for all of this at |
| once (for instance leave out the field mapping at first). |
| </p> |
| <p> |
| |
| Anyhow, please give me some feedback, counter |
| suggestions, let me know if I'm way off base or out of |
| line, etc. -Andy |
| </p> |
| </section> |
| |
| </body> |
| </document> |
