blob: 832c571dd6759bd0d52766ea766e42aab04980aa [file] [log] [blame]
Apache OpenNLP ${pom.version}
===============================
Building from the Source Distribution
-------------------------------------
At least Maven 3.0.0 is required for building.
To build everything go into the opennlp directory and run the following command:
mvn clean install
The results of the build will be placed in:
opennlp-distr/target/apache-opennlp-[version]-bin.tar-gz (or .zip)
What is in Similarity component in Apache OpenNLP ${pom.version}
---------------------------------------
SIMILARITY COMPONENT of OpenNLP
1. Introduction
This component does text relevance assessment. It takes two portions of texts (phrases, sentences, paragraphs) and returns a similarity score.
Similarity component can be used on top of search to improve relevance, computing similarity score between a question and all search results (snippets).
Also, this component is useful for web mining of images, videos, forums, blogs, and other media with textual descriptions. Such applications as content generation
and filtering meaningless speech recognition results are included in the sample applications of this component.
Relevance assessment is based on machine learning of syntactic parse trees (constituency trees, http://en.wikipedia.org/wiki/Parse_tree).
The similarity score is calculated as the size of all maximal common subtrees for sentences from a pair of texts (
www.aaai.org/ocs/index.php/WS/AAAIW11/paper/download/3971/4187, www.aaai.org/ocs/index.php/FLAIRS/FLAIRS11/paper/download/2573/3018,
www.aaai.org/ocs/index.php/SSS/SSS10/paper/download/1146/1448).
The objective of Similarity component is to give an application engineer as tool for text relevance which can be used as a black box, no need to understand
computational linguistics or machine learning.
2. Installation
Please refer to OpenNLP installation instructions
3. First use case of Similarity component: search
To start with this component, please refer to SearchResultsProcessorTest.java in package opennlp.tools.similarity.apps
public void testSearchOrder() runs web search using Bing API and improves search relevance.
Look at the code of
public List<HitBase> runSearch(String query)
and then at
private BingResponse calculateMatchScoreResortHits(BingResponse resp, String searchQuery)
which gets search results from Bing and re-ranks them based on computed similarity score.
The main entry to Similarity component is
SentencePairMatchResult matchRes = sm.assessRelevance(snapshot, searchQuery);
where we pass the search query and the snapshot and obtain the similarity assessment structure which includes the similarity score.
To run this test you need to obtain search API key from Bing at www.bing.com/developers/s/APIBasics.html and specify it in public class BingQueryRunner in
protected static final String APP_ID.
4. Solving a unique problem: content generation
To demonstrate the usability of Similarity component to tackle a problem which is hard to solve without a linguistic-based technology,
we introduce a content generation component:
RelatedSentenceFinder.java
The entry point here is the function call
hits = f.generateContentAbout("Albert Einstein");
which writes a biography of Albert Einstein by finding sentences on the web about various kinds of his activities (such as 'born', 'graduate', 'invented' etc.).
The key here is to compute similarity between the seed expression like "Albert Einstein invented relativity theory" and search result like
"Albert Einstein College of Medicine | Medical Education | Biomedical ...
www.einstein.yu.edu/Albert Einstein College of Medicine is one of the nation's premier institutions for medical education, ..."
and filter out irrelevant search results.
This is done in function
public HitBase augmentWithMinedSentencesAndVerifyRelevance(HitBase item, String originalSentence,
List<String> sentsAll)
SentencePairMatchResult matchRes = sm.assessRelevance(pageSentence + " " + title, originalSentence);
You can consult the results in gen.txt, where an essay on Einstein bio is written.
These are examples of generated articles, given the article title
http://www.allvoices.com/contributed-news/9423860/content/81937916-ichie-sings-jazz-blues-contemporary-tunes
http://www.allvoices.com/contributed-news/9415063-britney-spears-femme-fatale-in-north-sf-bay-area
5. Solving a high-importance problem: filtering out meaningless speech recognition results.
Speech recognitions SDKs usually produce a number of phrases as results, such as
"remember to buy milk tomorrow from trader joes",
"remember to buy milk tomorrow from 3 to jones"
One can see that the former is meaningful, and the latter is meaningless (although similar in terms of how it is pronounced).
We use web mining and Similarity component to detect a meaningful option (a mistake caused by trying to interpret meaningless
request by a query understanding system such as Siri for iPhone can be costly).
SpeechRecognitionResultsProcessor.java does the job:
public List<SentenceMeaningfullnessScore> runSearchAndScoreMeaningfulness(List<String> sents)
re-ranks the phrases in the order of decrease of meaningfulness.
6. Similarity component internals
in the package opennlp.tools.textsimilarity.chunker2matcher
ParserChunker2MatcherProcessor.java does parsing of two portions of text and matching the resultant parse trees to assess similarity between
these portions of text.
To run ParserChunker2MatcherProcessor
private static String MODEL_DIR = "resources/models";
needs to be specified
The key function
public SentencePairMatchResult assessRelevance(String para1, String para2)
takes two portions of text and does similarity assessment by finding the set of all maximum common subtrees
of the set of parse trees for each portion of text
It splits paragraphs into sentences, parses them, obtained chunking information and produces grouped phrases (noun, evrn, prepositional etc.):
public synchronized List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(String para)
and then attempts to find common subtrees:
in ParseTreeMatcherDeterministic.java
List<List<ParseTreeChunk>> res = md.matchTwoSentencesGroupedChunksDeterministic(sent1GrpLst, sent2GrpLst)
Phrase matching functionality is in package opennlp.tools.textsimilarity;
ParseTreeMatcherDeterministic.java:
Here's the key matching function which takes two phrases, aligns them and finds a set of maximum common sub-phrase
public List<ParseTreeChunk> generalizeTwoGroupedPhrasesDeterministic
7. Package structure
opennlp.tools.similarity.apps : 3 main applications
opennlp.tools.similarity.apps.utils: utilities for above applications
opennlp.tools.textsimilarity.chunker2matcher: parser which converts text into a form for matching parse trees
opennlp.tools.textsimilarity: parse tree matching functionality
Requirements
------------
Java 1.5 is required to run OpenNLP
Maven 3.0.0 is required for building it
Known OSGi Issues
------------
In an OSGi environment the following things are not supported:
- The coreference resolution component
- The ability to load a user provided feature generator class
Note
----
The current API contains still many deprecated methods, these
will be removed in one of our next releases, please
migrate to our new API.