merge from bgalitsky's own git repo
diff --git a/opennlp-similarity/README b/opennlp-similarity/README
deleted file mode 100644
index b535487..0000000
--- a/opennlp-similarity/README
+++ /dev/null
@@ -1,138 +0,0 @@
-Apache OpenNLP ${pom.version}
-===============================
-
-
-Building from the Source Distribution
--------------------------------------
-
-At least Maven 3.0.0 is required for building.
-
-To build everything go into the opennlp directory and run the following command:
- mvn clean install
-
-The results of the build will be placed in:
- opennlp-distr/target/apache-opennlp-[version]-bin.tar-gz (or .zip)
-
-What is in Similarity component in Apache OpenNLP ${pom.version}
----------------------------------------
-SIMILARITY COMPONENT of OpenNLP
-
-1. Introduction
-This component does text relevance assessment. It takes two portions of texts (phrases, sentences, paragraphs) and returns a similarity score.
-Similarity component can be used on top of search to improve relevance, computing similarity score between a question and all search results (snippets).
-Also, this component is useful for web mining of images, videos, forums, blogs, and other media with textual descriptions. Such applications as content generation
-and filtering meaningless speech recognition results are included in the sample applications of this component.
- Relevance assessment is based on machine learning of syntactic parse trees (constituency trees, http://en.wikipedia.org/wiki/Parse_tree).
-The similarity score is calculated as the size of all maximal common sub-trees for sentences from a pair of texts (
-www.aaai.org/ocs/index.php/WS/AAAIW11/paper/download/3971/4187, www.aaai.org/ocs/index.php/FLAIRS/FLAIRS11/paper/download/2573/3018,
-www.aaai.org/ocs/index.php/SSS/SSS10/paper/download/1146/1448).
- The objective of Similarity component is to give an application engineer as tool for text relevance which can be used as a black box, no need to understand
- computational linguistics or machine learning.
-
- 2. Installation
- Please refer to OpenNLP installation instructions
-
- 3. First use case of Similarity component: search
-
- To start with this component, please refer to SearchResultsProcessorTest.java in package opennlp.tools.similarity.apps
- public void testSearchOrder() runs web search using Bing API and improves search relevance.
- Look at the code of
- public List<HitBase> runSearch(String query)
- and then at
- private BingResponse calculateMatchScoreResortHits(BingResponse resp, String searchQuery)
- which gets search results from Bing and re-ranks them based on computed similarity score.
-
- The main entry to Similarity component is
- SentencePairMatchResult matchRes = sm.assessRelevance(snapshot, searchQuery);
- where we pass the search query and the snapshot and obtain the similarity assessment structure which includes the similarity score.
-
- To run this test you need to obtain search API key from Bing at www.bing.com/developers/s/APIBasics.html and specify it in public class BingQueryRunner in
- protected static final String APP_ID.
-
- 4. Solving a unique problem: content generation
- To demonstrate the usability of Similarity component to tackle a problem which is hard to solve without a linguistic-based technology,
- we introduce a content generation component:
- RelatedSentenceFinder.java
-
- The entry point here is the function call
- hits = f.generateContentAbout("Albert Einstein");
- which writes a biography of Albert Einstein by finding sentences on the web about various kinds of his activities (such as 'born', 'graduate', 'invented' etc.).
- The key here is to compute similarity between the seed expression like "Albert Einstein invented relativity theory" and search result like
- "Albert Einstein College of Medicine | Medical Education | Biomedical ...
- www.einstein.yu.edu/Albert Einstein College of Medicine is one of the nation's premier institutions for medical education, ..."
- and filter out irrelevant search results.
-
- This is done in function
- public HitBase augmentWithMinedSentencesAndVerifyRelevance(HitBase item, String originalSentence,
- List<String> sentsAll)
-
- SentencePairMatchResult matchRes = sm.assessRelevance(pageSentence + " " + title, originalSentence);
- You can consult the results in gen.txt, where an essay on Einstein bio is written.
-
- These are examples of generated articles, given the article title
- http://www.allvoices.com/contributed-news/9423860/content/81937916-ichie-sings-jazz-blues-contemporary-tunes
- http://www.allvoices.com/contributed-news/9415063-britney-spears-femme-fatale-in-north-sf-bay-area
-
- 5. Solving a high-importance problem: filtering out meaningless speech recognition results.
- Speech recognitions SDKs usually produce a number of phrases as results, such as
- "remember to buy milk tomorrow from trader joes",
- "remember to buy milk tomorrow from 3 to jones"
- One can see that the former is meaningful, and the latter is meaningless (although similar in terms of how it is pronounced).
- We use web mining and Similarity component to detect a meaningful option (a mistake caused by trying to interpret meaningless
- request by a query understanding system such as Siri for iPhone can be costly).
-
- SpeechRecognitionResultsProcessor.java does the job:
- public List<SentenceMeaningfullnessScore> runSearchAndScoreMeaningfulness(List<String> sents)
- re-ranks the phrases in the order of decrease of meaningfulness.
-
- 6. Similarity component internals
- in the package opennlp.tools.textsimilarity.chunker2matcher
- ParserChunker2MatcherProcessor.java does parsing of two portions of text and matching the resultant parse trees to assess similarity between
- these portions of text.
- To run ParserChunker2MatcherProcessor
- private static String MODEL_DIR = "resources/models";
- needs to be specified
-
- The key function
- public SentencePairMatchResult assessRelevance(String para1, String para2)
- takes two portions of text and does similarity assessment by finding the set of all maximum common subtrees
- of the set of parse trees for each portion of text
-
- It splits paragraphs into sentences, parses them, obtained chunking information and produces grouped phrases (noun, evrn, prepositional etc.):
- public synchronized List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(String para)
-
- and then attempts to find common subtrees:
- in ParseTreeMatcherDeterministic.java
- List<List<ParseTreeChunk>> res = md.matchTwoSentencesGroupedChunksDeterministic(sent1GrpLst, sent2GrpLst)
-
- Phrase matching functionality is in package opennlp.tools.textsimilarity;
- ParseTreeMatcherDeterministic.java:
- Here's the key matching function which takes two phrases, aligns them and finds a set of maximum common sub-phrase
- public List<ParseTreeChunk> generalizeTwoGroupedPhrasesDeterministic
-
- 7. Package structure
- opennlp.tools.similarity.apps : 3 main applications
- opennlp.tools.similarity.apps.utils: utilities for above applications
-
- opennlp.tools.textsimilarity.chunker2matcher: parser which converts text into a form for matching parse trees
- opennlp.tools.textsimilarity: parse tree matching functionality
-
-
-
-
-Requirements
-------------
-Java 1.5 is required to run OpenNLP
-Maven 3.0.0 is required for building it
-
-Known OSGi Issues
-------------
-In an OSGi environment the following things are not supported:
-- The coreference resolution component
-- The ability to load a user provided feature generator class
-
-Note
-----
-The current API contains still many deprecated methods, these
-will be removed in one of our next releases, please
-migrate to our new API.
diff --git a/opennlp-similarity/pom.xml b/opennlp-similarity/pom.xml
index 35b768b..a583e8e 100644
--- a/opennlp-similarity/pom.xml
+++ b/opennlp-similarity/pom.xml
@@ -1,25 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
+ license agreements. See the NOTICE file distributed with this work for additional
+ information regarding copyright ownership. The ASF licenses this file to
+ you under the Apache License, Version 2.0 (the "License"); you may not use
+ this file except in compliance with the License. You may obtain a copy of
+ the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
+ by applicable law or agreed to in writing, software distributed under the
+ License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
+ OF ANY KIND, either express or implied. See the License for the specific
+ language governing permissions and limitations under the License. -->
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
@@ -31,35 +24,52 @@
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-similarity</artifactId>
- <version>0.0.1</version>
+ <version>0.1.0</version>
<packaging>jar</packaging>
<name>OpenNLP Tool Similarity distribution</name>
<scm>
- <connection>scm:svn:http://svn.apache.org/repos/asf/opennlp/sandbox/opennlp-similarity/tags/opennlp-similarity-0.0.1</connection>
- <developerConnection>scm:svn:https://svn.apache.org/repos/asf/opennlp/sandbox/opennlp-similarity/tags/opennlp-similarity-0.0.1</developerConnection>
- <url>http://svn.apache.org/viewvc/opennlp/tags/opennlp-similarity-0.0.1</url>
+ <connection>scm:svn:http://svn.apache.org/repos/asf/opennlp/sandbox/opennlp-similarity/tags/opennlp-similarity-0.0.1</connection>
+ <developerConnection>scm:svn:https://svn.apache.org/repos/asf/opennlp/sandbox/opennlp-similarity/tags/opennlp-similarity-0.0.1</developerConnection>
+ <url>http://svn.apache.org/viewvc/opennlp/tags/opennlp-similarity-1.1.0</url>
</scm>
<prerequisites>
<maven>3.0</maven>
</prerequisites>
-
+ <distributionManagement>
+ <snapshotRepository>
+ <id>ossrh</id>
+ <url>https://oss.sonatype.org/content/repositories/snapshots</url>
+ </snapshotRepository>
+ </distributionManagement>
+
+
<repositories>
<repository>
- <id>net.billylieurance</id>
- <name>BillyLieuranceNet</name>
- <url>http://www.billylieurance.net/maven2</url>
- </repository>
+ <id>net.billylieurance</id>
+ <name>BillyLieuranceNet</name>
+ <url>http://www.billylieurance.net/maven2</url>
+ </repository>
</repositories>
+
+ <properties>
+ <nd4j.version>0.4-rc3.4</nd4j.version>
+ <dl4j.version>0.4-rc3.3</dl4j.version>
+ </properties>
<dependencies>
<dependency>
- <groupId>org.apache.opennlp</groupId>
- <artifactId>opennlp-tools</artifactId>
- <version>1.5.2-incubating</version>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ <version>1.6.4</version>
</dependency>
-
+ <dependency>
+ <groupId>org.apache.opennlp</groupId>
+ <artifactId>opennlp-tools</artifactId>
+ <version>1.5.2-incubating</version>
+ </dependency>
+
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
@@ -77,11 +87,10 @@
<artifactId>json</artifactId>
<version>20090211</version>
</dependency>
-
<dependency>
<groupId>org.apache.tika</groupId>
- <artifactId>tika-core</artifactId>
- <version>0.7</version>
+ <artifactId>tika-app</artifactId>
+ <version>1.6</version>
</dependency>
<dependency>
<groupId>net.sf.opencsv</groupId>
@@ -91,57 +100,179 @@
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
- <version>4.0.0-BETA</version>
+ <version>4.10.0</version>
</dependency>
-
+
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-core</artifactId>
- <version>4.0.0-BETA</version>
+ <version>4.10.0</version>
</dependency>
<dependency>
- <groupId>commons-codec</groupId>
- <artifactId>commons-codec</artifactId>
- <version>1.7</version>
+ <groupId>commons-codec</groupId>
+ <artifactId>commons-codec</artifactId>
+ <version>1.7</version>
</dependency>
<dependency>
- <groupId>commons-logging</groupId>
- <artifactId>commons-logging</artifactId>
- <version>1.1.1</version>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <version>1.1.1</version>
</dependency>
<dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpclient</artifactId>
- <version>4.2.1</version>
- </dependency>
- <dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpclient-cache</artifactId>
- <version>4.2.1</version>
+ <groupId>commons-collections</groupId>
+ <artifactId>commons-collections</artifactId>
+ <version>3.1</version>
</dependency>
<dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpcore</artifactId>
- <version>4.2.1</version>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.5</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.2.1</version>
</dependency>
<dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpmime</artifactId>
- <version>4.2.1</version>
- </dependency>
- <dependency>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>fluent-hc</artifactId>
- <version>4.2.1</version>
- </dependency>
- <dependency>
- <groupId>net.billylieurance.azuresearch</groupId>
- <artifactId>azure-bing-search-java</artifactId>
- <version>0.11.0</version>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient-cache</artifactId>
+ <version>4.2.1</version>
</dependency>
-
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpcore</artifactId>
+ <version>4.2.1</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpmime</artifactId>
+ <version>4.2.1</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>fluent-hc</artifactId>
+ <version>4.2.1</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.jgrapht</groupId>
+ <artifactId>jgrapht-jdk1.5</artifactId>
+ <version>0.7.3</version>
+ </dependency>
+ <dependency>
+ <groupId>de.jollyday</groupId>
+ <artifactId>jollyday</artifactId>
+ <version>0.4.7</version>
+ </dependency>
+ <dependency>
+ <groupId>jgraph</groupId>
+ <artifactId>jgraph</artifactId>
+ <version>5.13.0.0</version>
+ </dependency>
+ <dependency>
+ <groupId>javax.mail</groupId>
+ <artifactId>mail</artifactId>
+ <version>1.4</version>
+ </dependency>
+ <dependency>
+ <groupId>com.restfb</groupId>
+ <artifactId>restfb</artifactId>
+ <version>1.6.12</version>
+ </dependency>
+ <dependency>
+ <groupId>com.memetix</groupId>
+ <artifactId>microsoft-translator-java-api</artifactId>
+ <version>0.3</version>
+ </dependency>
+
+ <dependency>
+ <groupId>net.billylieurance.azuresearch</groupId>
+ <artifactId>azure-bing-search-java</artifactId>
+ <version>0.11.0</version>
+ </dependency>
+ <dependency>
+ <groupId>edu.mit</groupId>
+ <artifactId>jverbnet</artifactId>
+ <version>1.2.0</version>
+ <systemPath>${project.basedir}/lib/edu.mit.jverbnet-1.2.0.jar</systemPath>
+ <scope>system</scope>
+ </dependency>
+ <dependency>
+ <groupId>edu.stanford.nlp</groupId>
+ <artifactId>stanford-corenlp</artifactId>
+ <version>3.5.2</version>
+ </dependency>
+ <dependency>
+ <groupId>edu.stanford.nlp</groupId>
+ <artifactId>stanford-corenlp-model</artifactId>
+ <version>3.5.2</version>
+ <systemPath>${project.basedir}/lib/stanford-corenlp-3.5.2-models.jar</systemPath>
+ <scope>system</scope>
+ </dependency>
+ <dependency>
+ <groupId>edu.stanford.nlp</groupId>
+ <artifactId>ejml</artifactId>
+ <version>0.23</version>
+ <systemPath>${project.basedir}/lib/ejml-0.23.jar</systemPath>
+ <scope>system</scope>
+ </dependency>
+ <dependency>
+ <groupId>edu.stanford.nlp</groupId>
+ <artifactId>joda-time</artifactId>
+ <version>0.23</version>
+ <systemPath>${project.basedir}/lib/joda-time.jar</systemPath>
+ <scope>system</scope>
+ </dependency>
+ <dependency>
+ <groupId>edu.stanford.nlp</groupId>
+ <artifactId>jollyday</artifactId>
+ <version>0.23</version>
+ <systemPath>${project.basedir}/lib/jollyday.jar</systemPath>
+ <scope>system</scope>
+ </dependency>
+ <dependency>
+ <groupId>edu.stanford.nlp</groupId>
+ <artifactId>xom</artifactId>
+ <version>0.23</version>
+ <systemPath>${project.basedir}/lib/xom.jar</systemPath>
+ <scope>system</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.docx4j</groupId>
+ <artifactId>docx4j</artifactId>
+ <version>2.7.1</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.clulab</groupId>
+ <artifactId>processors_2.11</artifactId>
+ <version>5.7.1</version>
+ </dependency>
+ <dependency>
+ <groupId>org.clulab</groupId>
+ <artifactId>processors_2.11</artifactId>
+ <version>5.7.1</version>
+ <classifier>models</classifier>
+ </dependency>
+ <dependency>
+ <groupId>org.deeplearning4j</groupId>
+ <artifactId>deeplearning4j-ui</artifactId>
+ <version>${dl4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.deeplearning4j</groupId>
+ <artifactId>deeplearning4j-nlp</artifactId>
+ <version>${dl4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.nd4j</groupId>
+ <artifactId>nd4j-jblas</artifactId>
+ <version>${nd4j.version}</version>
+ </dependency>
+
</dependencies>
-
+
<build>
<plugins>
<plugin>
@@ -150,10 +281,10 @@
<configuration>
<source>1.5</source>
<target>1.5</target>
- <compilerArgument>-Xlint</compilerArgument>
+ <compilerArgument>-Xlint</compilerArgument>
</configuration>
</plugin>
-
+
<plugin>
<artifactId>maven-source-plugin</artifactId>
<executions>
@@ -183,70 +314,93 @@
</execution>
</executions>
</plugin>
- <plugin>
- <artifactId>maven-antrun-plugin</artifactId>
- <version>1.6</version>
- <executions>
- <execution>
- <id>generate checksums for binary artifacts</id>
- <goals><goal>run</goal></goals>
- <phase>verify</phase>
- <configuration>
- <target>
- <checksum algorithm="sha1" format="MD5SUM">
- <fileset dir="${project.build.directory}">
- <include name="*.zip" />
- <include name="*.gz" />
- </fileset>
- </checksum>
- <checksum algorithm="md5" format="MD5SUM">
- <fileset dir="${project.build.directory}">
- <include name="*.zip" />
- <include name="*.gz" />
- </fileset>
- </checksum>
- </target>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
- <artifactId>maven-assembly-plugin</artifactId>
- <executions>
- <execution>
- <id>src</id>
- <goals>
- <goal>single</goal>
- </goals>
- <phase>package</phase>
- <configuration>
- <descriptors>
- <descriptor>src/main/assembly/assembly.xml</descriptor>
- </descriptors>
- </configuration>
- </execution>
- <execution>
- <id>source-release-assembly</id>
- <configuration>
- <skipAssembly>true</skipAssembly>
- <mavenExecutorId>forked-path</mavenExecutorId>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-gpg-plugin</artifactId>
- <executions>
- <execution>
- <id>sign-artifacts</id>
- <phase>verify</phase>
- <goals>
- <goal>sign</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
+ <plugin>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.6</version>
+ <executions>
+ <execution>
+ <id>generate checksums for binary artifacts</id>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ <phase>verify</phase>
+ <configuration>
+ <target>
+ <checksum algorithm="sha1" format="MD5SUM">
+ <fileset dir="${project.build.directory}">
+ <include name="*.zip" />
+ <include name="*.gz" />
+ </fileset>
+ </checksum>
+ <checksum algorithm="md5" format="MD5SUM">
+ <fileset dir="${project.build.directory}">
+ <include name="*.zip" />
+ <include name="*.gz" />
+ </fileset>
+ </checksum>
+ </target>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>src</id>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <phase>package</phase>
+ <configuration>
+ <descriptors>
+ <descriptor>src/main/assembly/assembly.xml</descriptor>
+ </descriptors>
+ </configuration>
+ </execution>
+ <execution>
+ <id>source-release-assembly</id>
+ <configuration>
+ <skipAssembly>true</skipAssembly>
+ <mavenExecutorId>forked-path</mavenExecutorId>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <!-- <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-gpg-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>sign-artifacts</id>
+ <phase>verify</phase>
+ <goals>
+ <goal>sign</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.1</version>
+ <configuration>
+ <source>1.8</source>
+ <target>1.8</target>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.sonatype.plugins</groupId>
+ <artifactId>nexus-staging-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <extensions>true</extensions>
+ <configuration>
+ <serverId>ossrh</serverId>
+ <nexusUrl>https://oss.sonatype.org/</nexusUrl>
+ <autoReleaseAfterClose>true</autoReleaseAfterClose>
+ </configuration>
+ </plugin>
</plugins>
</build>
</project>
\ No newline at end of file
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/contentgen/multithreaded/BingWebQueryRunnerThread.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/contentgen/multithreaded/BingWebQueryRunnerThread.java
index b75a13b..b712847 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/contentgen/multithreaded/BingWebQueryRunnerThread.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/contentgen/multithreaded/BingWebQueryRunnerThread.java
@@ -6,7 +6,7 @@
import opennlp.tools.similarity.apps.BingQueryRunner;
import opennlp.tools.similarity.apps.HitBase;
- public class BingWebQueryRunnerThread extends BingQueryRunner implements Runnable{
+public class BingWebQueryRunnerThread extends BingQueryRunner implements Runnable{
private String query;
private List<HitBase> results= new ArrayList<HitBase>();
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/POStags.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/POStags.java
index 45dadf9..fafdef0 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/POStags.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/POStags.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.apps.relevanceVocabs;
public interface POStags {
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/PhraseProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/PhraseProcessor.java
index ae2772b..0d2ba00 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/PhraseProcessor.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/PhraseProcessor.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.apps.relevanceVocabs;
import java.util.ArrayList;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SentimentVocab.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SentimentVocab.java
index 150b3df..aced079 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SentimentVocab.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SentimentVocab.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.apps.relevanceVocabs;
import java.util.HashMap;
@@ -59,7 +76,7 @@
private static final String[] POSITIVE_NOUN_LIST = { "ability", "benefit",
"character", "charm", "comfort", "discount", "dream", "elegance",
"favourite", "feature", "improvement", "luck", "luxury", "offer",
- "pro", "quality", "requirement", "usability" };
+ "quality", "requirement", "usability" };
private static final String[] NEGATIVE_NOUN_LIST = { "blocker",
"challenge", "complain", "complaint", "compromise", "con",
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymListFilter.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymListFilter.java
index 7c12c9a..37f57e4 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymListFilter.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymListFilter.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.apps.relevanceVocabs;
import java.io.BufferedReader;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymMap.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymMap.java
index 804fc2b..7e680de 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymMap.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/SynonymMap.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.apps.relevanceVocabs;
import java.io.IOException;
@@ -12,50 +29,7 @@
import java.util.TreeMap;
import java.util.TreeSet;
- /**
- * Loads the <a target="_blank"
- * href="http://www.cogsci.princeton.edu/~wn/">WordNet </a> prolog file <a
- * href="http://www.cogsci.princeton.edu/2.0/WNprolog-2.0.tar.gz">wn_s.pl </a>
- * into a thread-safe main-memory hash map that can be used for fast
- * high-frequency lookups of synonyms for any given (lowercase) word string.
- * <p>
- * There holds: If B is a synonym for A (A -> B) then A is also a synonym for B (B -> A).
- * There does not necessarily hold: A -> B, B -> C then A -> C.
- * <p>
- * Loading typically takes some 1.5 secs, so should be done only once per
- * (server) program execution, using a singleton pattern. Once loaded, a
- * synonym lookup via {@link #getSynonyms(String)}takes constant time O(1).
- * A loaded default synonym map consumes about 10 MB main memory.
- * An instance is immutable, hence thread-safe.
- * <p>
- * This implementation borrows some ideas from the Lucene Syns2Index demo that
- * Dave Spencer originally contributed to Lucene. Dave's approach
- * involved a persistent Lucene index which is suitable for occasional
- * lookups or very large synonym tables, but considered unsuitable for
- * high-frequency lookups of medium size synonym tables.
- * <p>
- * Example Usage:
- * <pre>
- * String[] words = new String[] { "hard", "woods", "forest", "wolfish", "xxxx"};
- * SynonymMap map = new SynonymMap(new FileInputStream("samples/fulltext/wn_s.pl"));
- * for (int i = 0; i < words.length; i++) {
- * String[] synonyms = map.getSynonyms(words[i]);
- * System.out.println(words[i] + ":" + java.util.Arrays.asList(synonyms).toString());
- * }
- *
- * Example output:
- * hard:[arduous, backbreaking, difficult, fermented, firmly, grueling, gruelling, heavily, heavy, intemperately, knockout, laborious, punishing, severe, severely, strong, toilsome, tough]
- * woods:[forest, wood]
- * forest:[afforest, timber, timberland, wood, woodland, woods]
- * wolfish:[edacious, esurient, rapacious, ravening, ravenous, voracious, wolflike]
- * xxxx:[]
- * </pre>
- *
- * @see <a target="_blank"
- * href="http://www.cogsci.princeton.edu/~wn/man/prologdb.5WN.html">prologdb
- * man page </a>
- * @see <a target="_blank" href="http://www.hostmon.com/rfc/advanced.jsp">Dave's synonym demo site</a>
- */
+
public class SynonymMap {
/** the index data; Map<String word, String[] synonyms> */
@@ -73,7 +47,7 @@
* @param input
* the stream to read from (null indicates an empty synonym map)
* @throws IOException
- * if an error occured while reading the stream.
+ * if an error occurred while reading the stream.
*/
public SynonymMap(InputStream input) throws IOException {
this.table = input == null ? new HashMap<String,String[]>(0) : read(toByteArray(input));
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/WordDictionary.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/WordDictionary.java
index dbbec1d..cfae086 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/WordDictionary.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/relevanceVocabs/WordDictionary.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.apps.relevanceVocabs;
import java.util.HashMap;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/utils/email/EmailSender.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/utils/email/EmailSender.java
index 0b99fc2..ac7cb95 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/apps/utils/email/EmailSender.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/utils/email/EmailSender.java
@@ -14,7 +14,7 @@
*/
public class EmailSender {
private static final long serialVersionUID = 1L;
- private static final String mailboxAddress="bgalitsky@hotmail.com";
+ private static final String mailboxAddress="boris_galitsky@rambler.ru";
public boolean sendMail(String smtp, String user, String pass, InternetAddress from, InternetAddress[] to, InternetAddress[] cc, InternetAddress[] bcc, String subject, String body, String file) throws Exception
{
@@ -34,7 +34,7 @@
Properties props = new Properties();
props.put("mail.smtp.host", smtp);
props.put("mail.smtp.auth", "true");
- props.put("mail.smtp.port", "587");
+ props.put("mail.smtp.port", "465");
props.put("mail.smtp.starttls.enable", "true");
Authenticator auth = new SMTP_Authenticator (user, pass);
Session session = Session.getInstance(props, auth);
@@ -158,7 +158,8 @@
public static void main(String[] args){
EmailSender s = new EmailSender();
try {
- s.sendMail("smtp.live.com", "bgalitsky@hotmail.com", "******", new InternetAddress("bgalitsky@hotmail.com"), new InternetAddress[]{new InternetAddress("bgalitsky@hotmail.com")}, new InternetAddress[]{}, new InternetAddress[]{},
+ s.sendMail("smtp.rambler.ru", "boris_galitsky@rambler.ru", "b06g93",
+ new InternetAddress("bgalitsky@hotmail.com"), new InternetAddress[]{new InternetAddress("bgalitsky@hotmail.com")}, new InternetAddress[]{}, new InternetAddress[]{},
"Generated content for you", "body", null);
} catch (AddressException e) {
// TODO Auto-generated catch block
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/ProfileReaderWriter.java b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/ProfileReaderWriter.java
index 9081e1a..694da0a 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/ProfileReaderWriter.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/ProfileReaderWriter.java
@@ -123,6 +123,31 @@
e.printStackTrace();
}
}
+ public static void appendReport( List<String[]> allLines, String reportName){
+ List<String[]> previous;
+ try {
+ previous = readProfiles(reportName);
+ allLines.addAll(previous);
+ } catch (Exception e1) {
+ System.out.println("Creating file "+reportName);
+ }
+
+ CSVWriter writer = null;
+ try {
+ writer = new CSVWriter(new PrintWriter(reportName));
+ } catch (FileNotFoundException e) {
+ e.printStackTrace();
+ }
+
+ writer.writeAll(allLines);
+
+ try {
+ writer.flush();
+ writer.close();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
public static void writeReportListStr(List<String> res, String string) {
// TODO Auto-generated method stub
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefsBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefsBuilder.java
index 10e9683..8f215f7 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefsBuilder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefsBuilder.java
@@ -19,8 +19,8 @@
public class ParseCorefsBuilder {
protected static ParseCorefsBuilder instance;
- private Annotation annotation;
- StanfordCoreNLP pipeline;
+ protected Annotation annotation;
+ protected StanfordCoreNLP pipeline;
CommunicativeActionsArcBuilder caFinder = new CommunicativeActionsArcBuilder();
/**
@@ -35,9 +35,9 @@
return instance;
}
- ParseCorefsBuilder(){
+ protected ParseCorefsBuilder(){
Properties props = new Properties();
- props.put("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref");
+ props.put("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref, sentiment");
pipeline = new StanfordCoreNLP(props);
}
@@ -104,30 +104,18 @@
new Pair<Integer, Integer>(njSentence,njWord), mi.mentionSpan, mj.mentionSpan,
arcType);
arcs.add(arc);
-
- /*
- System.out.println("animacy = "+m.animacy);
- System.out.println("mention span = "+m.mentionSpan);
- System.out.println(" id = "+m.mentionID);
- System.out.println(" position = "+m.position);
- System.out.println(" start index = "+m.startIndex);
- System.out.println(" end index = "+m.endIndex);
- System.out.println(" mentionType = "+m.mentionType);
- System.out.println(" number = = "+m.number);
- */
}
}
-
-
}
List<WordWordInterSentenceRelationArc> arcsCA = buildCAarcs(nodesThicket);
+ arcs.addAll(arcsCA);
ParseThicket result = new ParseThicket(ptTrees, arcs);
result.setNodesThicket(nodesThicket);
return result;
}
- private List<WordWordInterSentenceRelationArc> buildCAarcs(
+ public List<WordWordInterSentenceRelationArc> buildCAarcs(
List<List<ParseTreeNode>> nodesThicket) {
List<WordWordInterSentenceRelationArc> arcs = new ArrayList<WordWordInterSentenceRelationArc>();
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseThicket.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseThicket.java
index e584d1e..8723e53 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseThicket.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseThicket.java
@@ -13,6 +13,36 @@
// then list for all sentences
private List<List<ParseTreeNode>> sentenceNodes;
+ private List<Float> sentimentProfile;
+
+ private String origText;
+ private List<List<ParseTreeNode>> phrases;
+
+
+ public List<Tree> getSentenceTrees() {
+ return sentenceTrees;
+ }
+
+ public void setSentenceTrees(List<Tree> sentenceTrees) {
+ this.sentenceTrees = sentenceTrees;
+ }
+
+ public List<List<ParseTreeNode>> getSentenceNodes() {
+ return sentenceNodes;
+ }
+
+ public void setSentenceNodes(List<List<ParseTreeNode>> sentenceNodes) {
+ this.sentenceNodes = sentenceNodes;
+ }
+
+ public String getOrigText() {
+ return origText;
+ }
+
+ public void setOrigText(String origText) {
+ this.origText = origText;
+ }
+
public List<Tree> getSentences() {
return sentenceTrees;
}
@@ -53,6 +83,22 @@
public String toString(){
return this.sentenceTrees+"\n"+this.arcs;
}
+
+ public void setPhrases(List<List<ParseTreeNode>> phrs) {
+ this.phrases = phrs;
+ }
+
+ public List<List<ParseTreeNode>> getPhrases() {
+ return phrases;
+ }
+
+ public List<Float> getSentimentProfile() {
+ return sentimentProfile;
+ }
+
+ public void setSentimentProfile(List<Float> sentimentProfile) {
+ this.sentimentProfile = sentimentProfile;
+ }
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseTreeNode.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseTreeNode.java
index 528eb4d..689a4b8 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseTreeNode.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseTreeNode.java
@@ -2,25 +2,92 @@
import java.util.ArrayList;
import java.util.List;
+import java.util.Map;
public class ParseTreeNode implements IGeneralizer<ParseTreeNode>{
- String word;
- // this is the POS tag of the token
- String pos;
- // this is the NER label of the token
- String ne;
- Integer id;
- //PhraseType
- String phraseType;
-
- public enum PhraseType {NP("NP"), VP("VP"), PRP("PRP");
- private PhraseType(final String text) {
- this.text = text;
- }
- private final String text;
-
- }
-
+ String word; // word in normal form, lemma
+ // this is the POS tag of the token
+ String pos;
+ // this is the NER label of the token
+ String ne;
+ Integer id;
+ //PhraseType
+ String phraseType;
+ Map<String, Object> attributes;
+ String normalizedWord;
+ String syntacticDependence;
+ String originalWord; //what actually occurs in a sentence
+
+ String head;
+ String label;
+ String modifier;
+
+
+
+ public String getOriginalWord() {
+ return originalWord;
+ }
+
+ public void setOriginalWord(String originalWord) {
+ this.originalWord = originalWord;
+ }
+
+ public String getHead() {
+ return head;
+ }
+
+ public void setHead(String head) {
+ this.head = head;
+ }
+
+ public String getLabel() {
+ return label;
+ }
+
+ public void setLabel(String label) {
+ this.label = label;
+ }
+
+ public String getModifier() {
+ return modifier;
+ }
+
+ public void setModifier(String modifier) {
+ this.modifier = modifier;
+ }
+
+ public String getNormalizedWord() {
+ return normalizedWord;
+ }
+
+ public void setNormalizedWord(String normalizedWord) {
+ this.normalizedWord = normalizedWord;
+ }
+
+ public String getSyntacticDependence() {
+ return syntacticDependence;
+ }
+
+ public void setSyntacticDependence(String syntacticDependence) {
+ this.syntacticDependence = syntacticDependence;
+ }
+
+ public Map<String, Object> getAttributes() {
+ return attributes;
+ }
+
+ public void setAttributes(Map<String, Object> attributes) {
+ this.attributes = attributes;
+ }
+
+ public enum PhraseType {NP("NP"), VP("VP"), PRP("PRP");
+ private PhraseType(final String text) {
+ this.text = text;
+ }
+ private final String text;
+
+ }
+
public ParseTreeNode(String word, String pos, String ne, Integer id) {
super();
this.word = word;
@@ -28,15 +95,14 @@
this.ne = ne;
this.id = id;
}
-
+
public ParseTreeNode(String word, String pos) {
super();
this.word = word;
this.pos = pos;
- this.ne = ne;
- this.id = id;
+
}
-
+
public String getPhraseType() {
return phraseType;
}
@@ -67,7 +133,7 @@
public void setId(Integer id) {
this.id = id;
}
-
+
public String toString(){
StringBuffer buf = new StringBuffer();
if (id!=null)
@@ -81,10 +147,27 @@
return buf.toString();
}
+ public static String toTreeRepresentationString(List<ParseTreeNode> chList){
+ StringBuffer buf = new StringBuffer();
+ for(ParseTreeNode ch: chList){
+ if (ch.getPos().startsWith(".") || ch.getPos().startsWith(",") || ch.getPos().startsWith(";") || ch.getPos().startsWith("!"))
+ continue;
+ buf.append( "("+ch.getWord()+ " " + ch.getPos() + ")" );
+ }
+ return buf.toString().trim();
+ }
+ public static String toWordString(List<ParseTreeNode> chList){
+ String buf = "";
+ for(ParseTreeNode ch: chList){
+ buf+=ch.getWord()+ " ";
+ }
+ return buf.trim();
+ }
+
@Override
public List<ParseTreeNode> generalize(Object o1, Object o2) {
List<ParseTreeNode> result = new ArrayList<ParseTreeNode>();
-
+
ParseTreeNode w1 = (ParseTreeNode) o1;
ParseTreeNode w2 = (ParseTreeNode) o2;
String posGen = generalizePOS(w1.pos, w2.pos);
@@ -95,7 +178,7 @@
result.add(newNode);
return result;
}
-
+
public String generalizeWord(String lemma1, String lemma2){
if (lemma1.equals(lemma2))
return lemma1;
@@ -105,49 +188,49 @@
return "*";
//TODO
return "*";
-
+
}
-
+
public String generalizePOS(String pos1, String pos2) {
- if ((pos1.startsWith("NN") && pos2.equals("NP") || pos2.startsWith("NN")
- && pos1.equals("NP"))) {
- return "NN";
- }
- if ((pos1.startsWith("NN") && pos2.equals("VBG") || pos2.startsWith("VBG")
- && pos1.equals("NN"))) {
- return "NN";
- }
+ if ((pos1.startsWith("NN") && pos2.equals("NP") || pos2.startsWith("NN")
+ && pos1.equals("NP"))) {
+ return "NN";
+ }
+ if ((pos1.startsWith("NN") && pos2.equals("VBG") || pos2.startsWith("VBG")
+ && pos1.equals("NN"))) {
+ return "NN";
+ }
- if ((pos1.startsWith("NN") && pos2.equals("ADJP") || pos2.startsWith("NN")
- && pos1.equals("ADJP"))) {
- return "NN";
- }
- if ((pos1.equals("IN") && pos2.equals("TO") || pos1.equals("TO")
- && pos2.equals("IN"))) {
- return "IN";
- }
- // VBx vs VBx = VB (does not matter which form for verb)
- if (pos1.startsWith("VB") && pos2.startsWith("VB")) {
- return "VB";
- }
+ if ((pos1.startsWith("NN") && pos2.equals("ADJP") || pos2.startsWith("NN")
+ && pos1.equals("ADJP"))) {
+ return "NN";
+ }
+ if ((pos1.equals("IN") && pos2.equals("TO") || pos1.equals("TO")
+ && pos2.equals("IN"))) {
+ return "IN";
+ }
+ // VBx vs VBx = VB (does not matter which form for verb)
+ if (pos1.startsWith("VB") && pos2.startsWith("VB")) {
+ return "VB";
+ }
- // ABx vs ABy always gives AB
- if (pos1.equalsIgnoreCase(pos2)) {
- return pos1;
- }
- if (pos1.length() > 2) {
- pos1 = pos1.substring(0, 2);
- }
+ // ABx vs ABy always gives AB
+ if (pos1.equalsIgnoreCase(pos2)) {
+ return pos1;
+ }
+ if (pos1.length() > 2) {
+ pos1 = pos1.substring(0, 2);
+ }
- if (pos2.length() > 2) {
- pos2 = pos2.substring(0, 2);
- }
- if (pos1.equalsIgnoreCase(pos2)) {
- return pos1 + "*";
- }
- return null;
- }
+ if (pos2.length() > 2) {
+ pos2 = pos2.substring(0, 2);
+ }
+ if (pos1.equalsIgnoreCase(pos2)) {
+ return pos1 + "*";
+ }
+ return null;
+ }
-
+
};
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/WordWordInterSentenceRelationArc.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/WordWordInterSentenceRelationArc.java
index db7905d..265a3fa 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/WordWordInterSentenceRelationArc.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/WordWordInterSentenceRelationArc.java
@@ -61,7 +61,7 @@
}
public String toString(){
- return "<sent="+codeFrom.getFirst()+"-word="+codeFrom.getSecond()+".."+lemmaFrom+"> ===> "+
+ return arcType.toString()+"&<sent="+codeFrom.getFirst()+"-word="+codeFrom.getSecond()+".."+lemmaFrom+"> ===> "+
"<sent="+codeTo.getFirst()+"-word="+codeTo.getSecond()+".."+lemmaTo+">";
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/apps/MultiSentenceSearchResultsProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/apps/MultiSentenceSearchResultsProcessor.java
index ce4b600..edd164f 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/apps/MultiSentenceSearchResultsProcessor.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/apps/MultiSentenceSearchResultsProcessor.java
@@ -73,7 +73,7 @@
hit.setSource(match.toString());
}
if (score < 2){ // attempt to match with snippet, if not much luck with original text
- match = matcher.assessRelevanceCache(pageSentsAndSnippet[0] ,
+ match = matcher.assessRelevanceCache(pageSentsAndSnippet[1] ,
searchQuery);
score = parseTreeChunkListScorer.getParseTreeChunkListScore(match);
}
@@ -161,7 +161,7 @@
LOG.info("No search results for query '" + query);
return null;
}
- ProfileReaderWriter.writeReport(reportData, "resultsForQuery_"+query.replace(' ', '_')+".csv");
+ //ProfileReaderWriter.writeReport(reportData, "resultsForQuery_"+query.replace(' ', '_')+".csv");
return hits;
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceExtendedForestSearchResultsProcessorSetFormer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceExtendedForestSearchResultsProcessorSetFormer.java
index eb67724..c568035 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceExtendedForestSearchResultsProcessorSetFormer.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceExtendedForestSearchResultsProcessorSetFormer.java
@@ -66,7 +66,7 @@
private List<HitBase> formTreeForestDataSet(
List<HitBase> hits, String query, boolean isPositive) {
List<HitBase> newHitList = new ArrayList<HitBase>(), newHitListReRanked = new ArrayList<HitBase>();
- // form the training set from original documets. Since search results are ranked, we set the first half as positive set,
+ // form the training set from original documents. Since search results are ranked, we set the first half as positive set,
//and the second half as negative set.
// after re-classification, being re-ranked, the search results might end up in a different set
List<String[]> treeBankBuffer = new ArrayList<String[]>();
@@ -117,7 +117,6 @@
treeBankBuffer.add(new String[] {posOrNeg+" |BT| "+t.toString()+ " |ET|"});
}
} catch (Exception e) {
- // TODO Auto-generated catch block
e.printStackTrace();
}
return treeBankBuffer;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceKernelBasedSearchResultsProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceKernelBasedSearchResultsProcessor.java
index df6189d..39d348e 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceKernelBasedSearchResultsProcessor.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/MultiSentenceKernelBasedSearchResultsProcessor.java
@@ -90,7 +90,7 @@
private List<HitBase> filterOutIrrelevantHitsByTreeKernelLearning(
List<HitBase> hits, String query) {
List<HitBase> newHitList = new ArrayList<HitBase>(), newHitListReRanked = new ArrayList<HitBase>();
- // form the training set from original documets. Since search results are ranked, we set the first half as positive set,
+ // form the training set from original documents. Since search results are ranked, we set the first half as positive set,
//and the second half as negative set.
// after re-classification, being re-ranked, the search results might end up in a different set
List<String[]> treeBankBuffer = new ArrayList<String[]>();
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/PT2ExtendedTreeForestBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/PT2ExtendedTreeForestBuilder.java
index 9c1c44a..fb5eed8 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/PT2ExtendedTreeForestBuilder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/PT2ExtendedTreeForestBuilder.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.kernel_interface;
import java.util.ArrayList;
@@ -32,6 +49,22 @@
return treeBankBuffer;
}
+ private String formTrainingSetFromTextOneLine(String para, boolean positive){
+ String prefix = null;
+ if (positive)
+ prefix=" 1 ";
+ else
+ prefix=" -1 ";
+
+ ParseThicket pt = matcher.buildParseThicketFromTextWithRST(para);
+ List<Tree> forest = pt.getSentences();
+ String line = prefix;
+ for(Tree t: forest){
+ line+= "|BT| "+t.toString()+ " |ET| ";
+ }
+ return line;
+ }
+
public void formPosNegTrainingSet(String pos, String neg, String path){
List<String[]> list = formTrainingSetFromText(pos, true),
negList= formTrainingSetFromText(neg, false);
@@ -50,8 +83,6 @@
ProfileReaderWriter.writeReport(treeBankBuffer, path+"unknown.txt", ' ');
tkRunner.runClassifier(path, "unknown.txt", modelFileName, "classifier_output.txt");
-
-
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/SnippetToParagraphFull.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/SnippetToParagraphFull.java
index 4cf3b34..d6a295f 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/SnippetToParagraphFull.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/SnippetToParagraphFull.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.kernel_interface;
import java.util.ArrayList;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeExtenderByAnotherLinkedTree.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeExtenderByAnotherLinkedTree.java
index 47e474f..c980f9f 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeExtenderByAnotherLinkedTree.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeExtenderByAnotherLinkedTree.java
@@ -1,29 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.kernel_interface;
import java.util.ArrayList;
import java.util.List;
+import java.util.logging.Logger;
import opennlp.tools.jsmlearning.ProfileReaderWriter;
import opennlp.tools.parse_thicket.ParseThicket;
import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
import opennlp.tools.parse_thicket.matching.Matcher;
import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
import edu.stanford.nlp.trees.Tree;
public class TreeExtenderByAnotherLinkedTree extends PT2ThicketPhraseBuilder {
+ private static Logger log = Logger
+ .getLogger("opennlp.tools.parse_thicket.kernel_interface.TreeExtenderByAnotherLinkedTree");
public List<String> buildForestForCorefArcs(ParseThicket pt){
List<String> results = new ArrayList<String>();
for(WordWordInterSentenceRelationArc arc: pt.getArcs()){
- if (!arc.getArcType().getType().startsWith("coref"))
- continue;
+ //if (!arc.getArcType().getType().startsWith("coref"))
+ // continue;
int fromSent = arc.getCodeFrom().getFirst();
int toSent = arc.getCodeTo().getFirst();
+ if (fromSent <1 || toSent <1 ) // TODO problem in sentence enumeration => skip building extended trees
+ return results;
+
String wordFrom = arc.getLemmaFrom();
String wordTo = arc.getLemmaTo();
- List<Tree> trees = getASubtreeWithRootAsNodeForWord1(pt.getSentences().get(fromSent-1), pt.getSentences().get(fromSent-1), new String[]{ wordFrom});
+ List<Tree> trees = getASubtreeWithRootAsNodeForWord1(pt.getSentences().get(fromSent-1),
+ pt.getSentences().get(fromSent-1), new String[]{ wordFrom});
if (trees==null || trees.size()<1)
continue;
System.out.println(trees);
@@ -32,13 +57,52 @@
System.out.println(sb.toString());
results.add(sb.toString());
}
- /*
- List<String[]> treeBankBuffer = new ArrayList<String[]>();
- for(String t: results){
- treeBankBuffer.add(new String[] {" 0 |BT|"+t.toString()+ "|ET|"});
+ // if no arcs then orig sentences
+ if (results.isEmpty()){
+ for(Tree t: pt.getSentences()){
+ results.add(t.toString());
+ }
}
- ProfileReaderWriter.writeReport(treeBankBuffer, "C:\\stanford-corenlp\\tree_kernel\\unknownForest.txt", ' ');
- */
+ return results;
+ }
+ // sentences in pt are enumerarted starting from 0;
+ //this func works with Sista version of Stanford NLP and sentences are coded from 0
+ public List<String> buildForestForRSTArcs(ParseThicket pt){
+ List<String> results = new ArrayList<String>();
+ for(WordWordInterSentenceRelationArc arc: pt.getArcs()){
+ // TODO - uncomment
+ //if (!arc.getArcType().getType().startsWith("rst"))
+ // continue;
+ int fromSent = arc.getCodeFrom().getFirst();
+ int toSent = arc.getCodeTo().getFirst();
+
+ String wordFrom = arc.getLemmaFrom();
+ String wordTo = arc.getLemmaTo();
+
+ if (wordFrom == null || wordFrom.length()<1 || wordTo == null || wordTo.length()<1)
+ log.severe("Empty lemmas for RST arc "+ arc);
+
+ List<Tree> trees = getASubtreeWithRootAsNodeForWord1(pt.getSentences().get(fromSent),
+ pt.getSentences().get(fromSent), new String[]{ wordFrom});
+ if (trees==null || trees.size()<1)
+ continue;
+ System.out.println(trees);
+ StringBuilder sb = new StringBuilder(10000);
+ Tree tree = trees.get(0);
+ // instead of phrase type for the root of the tree, we want to put the RST relation name
+ if (arc.getArcType().getType().startsWith("rst"))
+ tree.setValue(arc.getArcType().getSubtype());
+
+ toStringBuilderExtenderByAnotherLinkedTree1(sb, pt.getSentences().get(toSent), tree, new String[]{wordTo});
+ System.out.println(sb.toString());
+ results.add(sb.toString());
+ }
+ // if no arcs then orig sentences
+ if (results.isEmpty()){
+ for(Tree t: pt.getSentences()){
+ results.add(t.toString());
+ }
+ }
return results;
}
@@ -75,8 +139,6 @@
}
sb.append(' ');
toStringBuilderExtenderByAnotherLinkedTree1(sb, treeToInsert, null, null);
- int z=0; z++;
-
} else {
for (Tree kid : kids) {
sb.append(' ');
@@ -90,6 +152,7 @@
}
}
+ // given a parse tree and a
public List<Tree> getASubtreeWithRootAsNodeForWord1(Tree tree, Tree currentSubTree, String[] corefWords){
if (currentSubTree.isLeaf()){
return null;
@@ -97,26 +160,23 @@
List<Tree> result = null;
Tree[] kids = currentSubTree.children();
if (kids != null) {
- boolean bInsert=false;
+ boolean bFound=false;
String word = corefWords[corefWords.length-1];
-
for (Tree kid : kids) {
- if (bInsert){
+ if (bFound){
result.add(kid);
} else {
-
String phraseStr = kid.toString();
phraseStr=phraseStr.replace(")", "");
- if (phraseStr.endsWith(word)){
- bInsert=true;
+ if (phraseStr.endsWith(word)){ // found
+ bFound=true;
result = new ArrayList<Tree>();
}
}
}
- if (bInsert){
+ if (bFound){
return result;
}
-
// if not a selected node, proceed with iteration
for (Tree kid : kids) {
List<Tree> ts = getASubtreeWithRootAsNodeForWord1(tree, kid, corefWords);
@@ -128,7 +188,7 @@
return null;
}
-
+ // now obsolete
public Tree[] getASubtreeWithRootAsNodeForWord(Tree tree, Tree currentSubTree, String[] corefWords){
if (currentSubTree.isLeaf()){
return null;
@@ -238,7 +298,7 @@
}
}
- private StringBuilder toStringBuilder(StringBuilder sb, Tree t) {
+ public StringBuilder toStringBuilder(StringBuilder sb, Tree t) {
if (t.isLeaf()) {
if (t.label() != null) {
sb.append(t.label().value());
@@ -263,22 +323,25 @@
}
public static void main(String[] args){
+ VerbNetProcessor p = VerbNetProcessor.
+ getInstance("/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources");
+
Matcher matcher = new Matcher();
TreeExtenderByAnotherLinkedTree extender = new TreeExtenderByAnotherLinkedTree();
ParseThicket pt = matcher.buildParseThicketFromTextWithRST(//"I went to the forest to look for a tree. I found out that it was thick and green");
- "Iran refuses to accept the UN proposal to end its dispute over its work on nuclear weapons."+
+ "Iran refuses to accept the UN proposal to end its dispute over its work on nuclear weapons. "+
"UN nuclear watchdog passes a resolution condemning Iran for developing its second uranium enrichment site in secret. " +
"A recent IAEA report presented diagrams that suggested Iran was secretly working on nuclear weapons. " +
"Iran envoy says its nuclear development is for peaceful purpose, and the material evidence against it has been fabricated by the US. ");
List<String> results = extender.buildForestForCorefArcs(pt);
System.out.println(results);
- System.exit(0);
+ //System.exit(0);
List<Tree> forest = pt.getSentences();
- List<Tree> trees = extender.getASubtreeWithRootAsNodeForWord1(forest.get(1), forest.get(1), new String[]{"it"});
+ List<Tree> trees = extender.getASubtreeWithRootAsNodeForWord1(forest.get(1), forest.get(1), new String[]{"its"});
System.out.println(trees);
StringBuilder sb = new StringBuilder(10000);
extender.toStringBuilderExtenderByAnotherLinkedTree1(sb, forest.get(0), trees.get(0), new String[]{"the", "forest"});
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelRunner.java
index f00904f..294fb38 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelRunner.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelRunner.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.kernel_interface;
import java.io.BufferedReader;
@@ -30,11 +47,18 @@
public void runLearner(String dir, String learning_file, String model_file)
{
+ if (!dir.endsWith("/"))
+ dir+="/";
+ String[] runString = new String[]{dir+"svm_learn","-t", "5","-j","2","-W","A", dir+learning_file, dir+model_file};
+ runEXE(runString, dir);
+ }
+ public void runLearnerWin(String dir, String learning_file, String model_file)
+ {
dir = dir.replace('/', '\\');
if (!dir.endsWith("\\"))
dir+="\\";
- String[] runString = new String[]{dir+"svm_learn.exe","-t", "5", dir+learning_file, dir+model_file};
+ String[] runString = new String[]{dir+"svm_learn.exe","-t", "5","-j","2","-W","A", dir+learning_file, dir+model_file};
runEXE(runString, dir);
}
@@ -42,6 +66,13 @@
//svm_classify example_file model_file predictions_file
public void runClassifier(String dir, String example_file, String model_file, String predictions_file)
{
+ if (!dir.endsWith("/"))
+ dir+="/";
+ String[] runString = new String[]{dir+"svm_classify", dir+example_file, dir+model_file, dir+predictions_file};
+ runEXE(runString, dir);
+ }
+ public void runClassifierWin(String dir, String example_file, String model_file, String predictions_file)
+ {
dir = dir.replace('/', '\\');
if (!dir.endsWith("\\"))
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationListReducer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationListReducer.java
deleted file mode 100644
index ef0569a..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationListReducer.java
+++ /dev/null
@@ -1,148 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.parse_thicket.matching;
-
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.List;
-
-public class GeneralizationListReducer {
- public List<ParseTreePath> applyFilteringBySubsumption_OLD(
- List<ParseTreePath> result) {
- List<ParseTreePath> resultDupl = new ArrayList<ParseTreePath>();
- resultDupl.addAll(new HashSet<ParseTreePath>(result));
- result = resultDupl;
- if (result.size() < 2)
- return result; // nothing to reduce
- List<ParseTreePath> resultReduced = new ArrayList<ParseTreePath>();
- int size = result.size();
- for (int i = 0; i < size; i++) {
- Boolean bSubChunk = false;
- for (int j = 0; j < size; j++) {
- if (i == j) {
- continue;
- }
- if (result.get(j).isASubChunk(result.get(i))) {
- bSubChunk = true;
- }
- }
- if (!bSubChunk)
- resultReduced.add(result.get(i));
- }
-
- if (resultReduced.size() < 1) {
- System.err.println("Wrong subsumption reduction");
- }
-
- if (resultReduced.size() > 1) {
- int z = 0;
- z++;
- }
- return resultReduced;
-
- }
-
- public List<ParseTreePath> applyFilteringBySubsumptionOLD(
- List<ParseTreePath> result) {
- List<ParseTreePath> resultDupl = null;
- if (result.size() < 2)
- return result; // nothing to reduce
- List<ParseTreePath> resultReduced = new ArrayList<ParseTreePath>();
- int size = result.size();
- resultDupl = new ArrayList<ParseTreePath>(result);
- for (int s = 0; s < size; s++) {
- for (int i = 0; i < resultDupl.size(); i++) {
- Boolean bStop = false;
- for (int j = 0; j < resultDupl.size(); j++) {
- if (i == j) {
- continue;
- }
- if (result.get(j).isASubChunk(result.get(i))
- && !result.get(i).isASubChunk(result.get(j))) {
- resultDupl.remove(i);
- bStop = true;
- break;
- }
- }
- if (bStop) {
- break;
- }
- }
- }
- resultReduced = resultDupl;
- if (resultReduced.size() < 1) {
- System.err.println("Wrong subsumption reduction");
- }
-
- if (resultReduced.size() > 1) {
- int z = 0;
- z++;
- }
- return resultReduced;
-
- }
-
- public List<ParseTreePath> applyFilteringBySubsumption(
- List<ParseTreePath> result) {
- List<Integer> resultDuplIndex = new ArrayList<Integer>();
- List<ParseTreePath> resultReduced = new ArrayList<ParseTreePath>();
-
- if (result.size() < 2) {
- return result; // nothing to reduce
- }
- // remove empty
- for (ParseTreePath ch : result) {
- if (ch.getLemmas().size() > 0) {
- resultReduced.add(ch);
- }
- }
- result = resultReduced;
-
- for (int i = 0; i < result.size(); i++) {
- for (int j = i + 1; j < result.size(); j++) {
- if (i == j) {
- continue;
- }
- if (result.get(j).isASubChunk(result.get(i))) {
- resultDuplIndex.add(i);
- } else if (result.get(i).isASubChunk(result.get(j))) {
- resultDuplIndex.add(j);
- }
- }
-
- }
- resultReduced = new ArrayList<ParseTreePath>();
- for (int i = 0; i < result.size(); i++) {
- if (!resultDuplIndex.contains(i)) {
- resultReduced.add(result.get(i));
- }
- }
-
- if (resultReduced.size() < 1) {
- System.err.println("Wrong subsumption reduction");
- resultReduced = result;
- }
-
- return resultReduced;
-
- }
-
- // testing sub-chunk functionality and
- // elimination more general according to subsumption relation
-
-}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaFormManager.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaFormManager.java
index cb6f3e9..694abce 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaFormManager.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaFormManager.java
@@ -19,11 +19,11 @@
import java.util.List;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
public class LemmaFormManager {
- public String matchLemmas(PorterStemmer ps, String lemma1, String lemma2,
+ public String matchLemmas(PStemmer ps, String lemma1, String lemma2,
String POS) {
if (POS == null) {
return null;
@@ -95,7 +95,7 @@
// if (sim!=null && (lemmaMatch!=null && !lemmaMatch.equals("fail"))){
}
-
+/*
// all lemmas ending with # in ch1 and/or ch2 SHOULD occur in chunkToAdd
public boolean mustOccurVerifier(ParseTreePath ch1, ParseTreePath ch2,
ParseTreePath chunkToAdd) {
@@ -112,5 +112,5 @@
}
return true;
}
-
+*/
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/Matcher.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/Matcher.java
index 0830276..8540ff2 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/Matcher.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/Matcher.java
@@ -1,26 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.matching;
+import java.io.File;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-
-
import opennlp.tools.parse_thicket.IGeneralizer;
-import opennlp.tools.parse_thicket.ParseCorefsBuilder;
+import opennlp.tools.parse_thicket.ParseCorefBuilderWithNER;
import opennlp.tools.parse_thicket.ParseThicket;
import opennlp.tools.parse_thicket.ParseTreeNode;
-import opennlp.tools.textsimilarity.LemmaPair;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
import opennlp.tools.textsimilarity.ParseTreeChunk;
-import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
-import opennlp.tools.textsimilarity.SentencePairMatchResult;
-import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
public class Matcher implements IGeneralizer<List<List<ParseTreeNode>>>{
- ParseTreeMatcherDeterministic md = new ParseTreeMatcherDeterministic();
- ParseCorefsBuilder ptBuilder = ParseCorefsBuilder.getInstance();
+ public static String resourceDir = new File(".").getAbsolutePath().replace("/.", "") + "/src/test/resources";
+ VerbNetProcessor proc = VerbNetProcessor.getInstance(resourceDir);
+
+ protected PhraseGroupGeneralizer pgGen = new PhraseGroupGeneralizer();
+
+ protected static ParseCorefBuilderWithNER ptBuilder = null;
+
+ static {
+ synchronized (Matcher.class) {
+ ptBuilder = ParseCorefBuilderWithNER.getInstance();
+ }
+ }
+
+
PT2ThicketPhraseBuilder phraseBuilder = new PT2ThicketPhraseBuilder();
- Map<String, ParseThicket> parseThicketHash = new HashMap<String, ParseThicket>();
+ protected Map<String, ParseThicket> parseThicketHash = new HashMap<String, ParseThicket>();
+
+
/** * The key function of similarity component which takes two portions of text
* and does similarity assessment by finding the set of all maximum common
* subtrees of the set of parse trees for each portion of text
@@ -31,11 +58,16 @@
* text 2
* @return the matching results structure, which includes the similarity score
*/
-
- public Matcher(){
-
+ private static Matcher instance;
+
+ public synchronized static Matcher getInstance() {
+ if (instance == null)
+ instance = new Matcher();
+
+ return instance;
}
-
+
+
public List<List<ParseTreeChunk>> assessRelevance(String para1, String para2) {
// first build PTs for each text
ParseThicket pt1 = ptBuilder.buildParseThicket(para1);
@@ -47,28 +79,60 @@
List<List<ParseTreeChunk>> sent1GrpLst = formGroupedPhrasesFromChunksForPara(phrs1),
sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
-
- List<List<ParseTreeChunk>> res = md
- .matchTwoSentencesGroupedChunksDeterministic(sent1GrpLst, sent2GrpLst);
+
+ List<List<ParseTreeChunk>> res = pgGen.generalize(sent1GrpLst, sent2GrpLst);
+
return res;
}
-
+
+
+ public List<List<ParseTreeChunk>> assessRelevance(List<List<ParseTreeChunk>> para0, String para2) {
+ // first build PTs for each text
+
+ ParseThicket pt2 = ptBuilder.buildParseThicket(para2);
+ // then build phrases and rst arcs
+ List<List<ParseTreeNode>> phrs2 = phraseBuilder.buildPT2ptPhrases(pt2);
+ // group phrases by type
+ List<List<ParseTreeChunk>> sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
+
+
+ List<List<ParseTreeChunk>> res = pgGen.generalize(para0, sent2GrpLst);
+
+ return res;
+
+ }
+
+ public GeneralizationResult assessRelevanceG(List<List<ParseTreeChunk>> para0, String para2) {
+ List<List<ParseTreeChunk>> res = assessRelevance( para0, para2);
+ return new GeneralizationResult(res);
+ }
+
+ public GeneralizationResult assessRelevanceG(String para0, String para2) {
+ List<List<ParseTreeChunk>> res = assessRelevance( para0, para2);
+ return new GeneralizationResult(res);
+ }
+
+ public GeneralizationResult assessRelevanceG(GeneralizationResult para0, String para2) {
+ List<List<ParseTreeChunk>> res = assessRelevance( para0.getGen(), para2);
+ return new GeneralizationResult(res);
+ }
+
public List<List<ParseTreeChunk>> assessRelevanceCache(String para1, String para2) {
// first build PTs for each text
-
+
ParseThicket pt1 = parseThicketHash.get(para1);
if (pt1==null){
- pt1= ptBuilder.buildParseThicket(para1);
- parseThicketHash.put(para1, pt1);
+ pt1= ptBuilder.buildParseThicket(para1);
+ parseThicketHash.put(para1, pt1);
}
-
+
ParseThicket pt2 = parseThicketHash.get(para2);
if (pt2==null){
- pt2= ptBuilder.buildParseThicket(para2);
- parseThicketHash.put(para2, pt2);
+ pt2= ptBuilder.buildParseThicket(para2);
+ parseThicketHash.put(para2, pt2);
}
-
+
// then build phrases and rst arcs
List<List<ParseTreeNode>> phrs1 = phraseBuilder.buildPT2ptPhrases(pt1);
List<List<ParseTreeNode>> phrs2 = phraseBuilder.buildPT2ptPhrases(pt2);
@@ -76,31 +140,29 @@
List<List<ParseTreeChunk>> sent1GrpLst = formGroupedPhrasesFromChunksForPara(phrs1),
sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
-
- List<List<ParseTreeChunk>> res = md
- .matchTwoSentencesGroupedChunksDeterministic(sent1GrpLst, sent2GrpLst);
+
+ List<List<ParseTreeChunk>> res = pgGen.generalize(sent1GrpLst, sent2GrpLst);
return res;
}
-
+
public List<List<ParseTreeChunk>> generalize(List<List<ParseTreeNode>> phrs1,
List<List<ParseTreeNode>> phrs2) {
// group phrases by type
- List<List<ParseTreeChunk>> sent1GrpLst = formGroupedPhrasesFromChunksForPara(phrs1),
- sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
+ List<List<ParseTreeChunk>> sent1GrpLst = formGroupedPhrasesFromChunksForPara(phrs1),
+ sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
-
- List<List<ParseTreeChunk>> res = md
- .matchTwoSentencesGroupedChunksDeterministic(sent1GrpLst, sent2GrpLst);
- return res;
+
+ List<List<ParseTreeChunk>> res = pgGen.generalize(sent1GrpLst, sent2GrpLst);
+ return res;
}
- private List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(
+ protected List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(
List<List<ParseTreeNode>> phrs) {
List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
List<ParseTreeChunk> nps = new ArrayList<ParseTreeChunk>(), vps = new ArrayList<ParseTreeChunk>(),
pps = new ArrayList<ParseTreeChunk>();
for(List<ParseTreeNode> ps:phrs){
- ParseTreeChunk ch = convertNodeListIntoChunk(ps);
+ ParseTreeChunk ch = new ParseTreeChunk(ps);
String ptype = ps.get(0).getPhraseType();
if (ptype.equals("NP")){
nps.add(ch);
@@ -122,16 +184,31 @@
}
ParseTreeChunk ch = new ParseTreeChunk(lemmas, poss, 0, 0);
ch.setMainPOS(ps.get(0).getPhraseType());
+ ch.setParseTreeNodes(ps);
return ch;
}
-
+
// this function is the main entry point into the PT builder if rst arcs are required
public ParseThicket buildParseThicketFromTextWithRST(String para){
ParseThicket pt = ptBuilder.buildParseThicket(para);
- phraseBuilder.buildPT2ptPhrases(pt);
+ List<List<ParseTreeNode>> phrs = phraseBuilder.buildPT2ptPhrases(pt);
+ pt.setPhrases(phrs);
return pt;
}
+ // verify that all sections (NP, PRP and VP are present
+ public boolean isCoveredByTemplate(List<List<ParseTreeChunk>> template, List<List<ParseTreeChunk>> sampleGen){
+ try {
+ if (template.size() == sampleGen.size() && sampleGen.get(0).size()>0 && sampleGen.get(1).size()>0 )
+ //template.get(0).get(0).getParseTreeNodes().size() == template.get(0).get(0).size())
+ return true;
+ } catch (Exception e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+ }
+
+ return false;
+ }
@Override
public List<List<List<ParseTreeNode>>> generalize(Object o1, Object o2) {
@@ -139,4 +216,48 @@
return null;
}
+
+ public static void main(String[] args){
+ Matcher m = new Matcher();
+
+ m.buildParseThicketFromTextWithRST("Mary Poppins got her identification 8765");
+
+ List<List<ParseTreeChunk>> template = m.assessRelevance("John Doe send his California driver license 1234567",
+ "John Travolta send her california license 4567456"
+ //"New York hid her US social number 666-66-6666");
+ );
+
+ System.out.println(template+"\n");
+ //in
+ List<List<ParseTreeChunk>> res = m.assessRelevance(template, "Mary Jones send her Canada prisoner id number 666666666");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+ res = m.assessRelevance(template, "Mary Stewart hid her Mexico cook id number 666666666");
+ System.out.println(res + " => "+
+ m.isCoveredByTemplate(template, res));
+ res = m.assessRelevance(template, "Robin mentioned her Peru fisher id 2345");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+ res = m.assessRelevance(template, "Yesterday Peter Doe hid his Bolivia set id number 666666666");
+ System.out.println(res + " => "+
+ m.isCoveredByTemplate(template, res));
+ res = m.assessRelevance(template, "Robin mentioned her best Peru fisher man id 2345");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+ //out
+ res = m.assessRelevance(template, "Spain hid her Canada driver id number 666666666");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+ res = m.assessRelevance(template, "John Poppins hid her prisoner id 666666666");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+
+ res = m.assessRelevance(template, "Microsoft announced its Windows Azure release number 666666666");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+ res = m.assessRelevance(template, "John Poppins hid her Google id 666666666");
+ System.out.println(res+ " => "+
+ m.isCoveredByTemplate(template, res));
+ }
}
+
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilder.java
index 7612f26..5f07593 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilder.java
@@ -1,26 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.matching;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.logging.Logger;
import opennlp.tools.parse_thicket.ParseThicket;
import opennlp.tools.parse_thicket.ParseTreeNode;
import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
import opennlp.tools.parse_thicket.rhetoric_structure.RhetoricStructureArcsBuilder;
-
-import org.jgrapht.Graph;
-import org.jgrapht.graph.DefaultEdge;
-import org.jgrapht.graph.SimpleGraph;
-
-
import edu.stanford.nlp.trees.Tree;
public class PT2ThicketPhraseBuilder {
-
+
RhetoricStructureArcsBuilder rstBuilder = new RhetoricStructureArcsBuilder();
-
+ private static Logger log = Logger
+ .getLogger("opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder");
+
/*
* Building phrases takes a Parse Thicket and forms phrases for each sentence individually
* Then based on built phrases and obtained arcs, it builds arcs for RST
@@ -29,108 +43,111 @@
public List<List<ParseTreeNode>> buildPT2ptPhrases(ParseThicket pt ) {
List<List<ParseTreeNode>> phrasesAllSent = new ArrayList<List<ParseTreeNode>> ();
+ if (pt ==null) // parsing failed, return empty
+ return phrasesAllSent;
Map<Integer, List<List<ParseTreeNode>>> sentNumPhrases = new HashMap<Integer, List<List<ParseTreeNode>>>();
// build regular phrases
for(int nSent=0; nSent<pt.getSentences().size(); nSent++){
-
-
List<ParseTreeNode> sentence = pt.getNodesThicket().get(nSent);
Tree ptree = pt.getSentences().get(nSent);
//ptree.pennPrint();
List<List<ParseTreeNode>> phrases = buildPT2ptPhrasesForASentence(ptree, sentence);
- System.out.println(phrases);
+ log.info(phrases.toString());
phrasesAllSent.addAll(phrases);
sentNumPhrases.put(nSent, phrases);
}
-
+
// discover and add RST arcs
List<WordWordInterSentenceRelationArc> arcsRST =
rstBuilder.buildRSTArcsFromMarkersAndCorefs(pt.getArcs(), sentNumPhrases, pt);
-
+
List<WordWordInterSentenceRelationArc> arcs = pt.getArcs();
arcs.addAll(arcsRST);
pt.setArcs(arcs);
-
+ if (pt.getArcs().size()>20){
+ log.info(pt.toString());
+ }
+
List<List<ParseTreeNode>> expandedPhrases = expandTowardsThicketPhrases(phrasesAllSent, pt.getArcs(), sentNumPhrases, pt);
return expandedPhrases;
}
-/* Take all phrases, all arcs and merge phrases into Thicket phrases.
- * Then add the set of generalized (Thicket) phrases to the input set of phrases
- * phrasesAllSent - list of lists of phrases for each sentence
- * sentNumPhrase - map , gives for each sentence id, the above list
- * arcs - arcs formed so far
- * pt - the built Parse Thicket
- */
- private List<List<ParseTreeNode>> expandTowardsThicketPhrases(
+ /* Take all phrases, all arcs and merge phrases into Thicket phrases.
+ * Then add the set of generalized (Thicket) phrases to the input set of phrases
+ * phrasesAllSent - list of lists of phrases for each sentence
+ * sentNumPhrase - map , gives for each sentence id, the above list
+ * arcs - arcs formed so far
+ * pt - the built Parse Thicket
+ */
+ protected List<List<ParseTreeNode>> expandTowardsThicketPhrases(
List<List<ParseTreeNode>> phrasesAllSent,
List<WordWordInterSentenceRelationArc> arcs,
Map<Integer, List<List<ParseTreeNode>>> sentNumPhrases,
ParseThicket pt ) {
List<List<ParseTreeNode>> thicketPhrasesAllSent = new ArrayList<List<ParseTreeNode>>();
-
-
- for(int nSent=0; nSent<pt.getSentences().size(); nSent++){
- for(int mSent=nSent+1; mSent<pt.getSentences().size(); mSent++){
- // for given arc, find phrases connected by this arc and add to the list of phrases
- for(WordWordInterSentenceRelationArc arc: arcs){
- List<List<ParseTreeNode>> phrasesFrom = sentNumPhrases.get(nSent);
- List<List<ParseTreeNode>> phrasesTo = sentNumPhrases.get(mSent);
- int fromIndex = arc.getCodeFrom().getFirst();
- int toIndex = arc.getCodeTo().getFirst();
- if (nSent==fromIndex && mSent==toIndex){
- int sentPosFrom = arc.getCodeFrom().getSecond();
- int sentPosTo = arc.getCodeTo().getSecond();
- // for the given arc arc, find phrases which are connected by it
- List<ParseTreeNode> lFromFound = null, lToFound = null;
- for(List<ParseTreeNode> lFrom: phrasesFrom){
- if (lToFound!=null)
+
+
+ for(int nSent=0; nSent<pt.getSentences().size(); nSent++){
+ for(int mSent=nSent+1; mSent<pt.getSentences().size(); mSent++){
+ // for given arc, find phrases connected by this arc and add to the list of phrases
+ for(WordWordInterSentenceRelationArc arc: arcs){
+ List<List<ParseTreeNode>> phrasesFrom = sentNumPhrases.get(nSent);
+ List<List<ParseTreeNode>> phrasesTo = sentNumPhrases.get(mSent);
+ int fromIndex = arc.getCodeFrom().getFirst();
+ int toIndex = arc.getCodeTo().getFirst();
+ if (nSent==fromIndex && mSent==toIndex){
+ int sentPosFrom = arc.getCodeFrom().getSecond();
+ int sentPosTo = arc.getCodeTo().getSecond();
+ // for the given arc arc, find phrases which are connected by it
+ List<ParseTreeNode> lFromFound = null, lToFound = null;
+ for(List<ParseTreeNode> lFrom: phrasesFrom){
+ if (lToFound!=null)
+ break;
+ for(ParseTreeNode lFromP: lFrom){
+ if (lFromP.getId()!=null && lFromP.getId()==sentPosFrom){
+ lFromFound = lFrom;
break;
- for(ParseTreeNode lFromP: lFrom){
- if (lFromP.getId()!=null && lFromP.getId()==sentPosFrom){
- lFromFound = lFrom;
- break;
- }
}
}
- for(List<ParseTreeNode> lTo: phrasesTo){
- if (lToFound!=null)
- break;
- for(ParseTreeNode lToP: lTo)
- if (lToP.getId()!=null && lToP.getId()==sentPosTo){
- lToFound = lTo;
- break;
- }
- }
- // obtain a thicket phrase and add it to the list
- if (lFromFound!=null && lToFound!=null){
-
- if (identicalSubPhrase(lFromFound, lToFound))
- continue;
- List<ParseTreeNode> appended = append(lFromFound, lToFound);
- if (thicketPhrasesAllSent.contains(appended))
- continue;
- System.out.println("rel: "+arc);
- System.out.println("From "+lFromFound);
- System.out.println("TO "+lToFound);
- thicketPhrasesAllSent.add(append(lFromFound, lToFound));
- //break;
- }
}
-
+ for(List<ParseTreeNode> lTo: phrasesTo){
+ if (lToFound!=null)
+ break;
+ for(ParseTreeNode lToP: lTo)
+ if (lToP.getId()!=null && lToP.getId()==sentPosTo){
+ lToFound = lTo;
+ break;
+ }
+ }
+ // obtain a thicket phrase and add it to the list
+ if (lFromFound!=null && lToFound!=null){
+
+ if (identicalSubPhrase(lFromFound, lToFound))
+ continue;
+ List<ParseTreeNode> appended = append(lFromFound, lToFound);
+ if (thicketPhrasesAllSent.contains(appended))
+ continue;
+ log.info("rel: "+arc);
+ log.info("From "+lFromFound);
+ System.out.println("TO "+lToFound);
+ thicketPhrasesAllSent.add(append(lFromFound, lToFound));
+ //break;
+ }
}
+
}
}
- phrasesAllSent.addAll(thicketPhrasesAllSent);
- return phrasesAllSent;
+ }
+ phrasesAllSent.addAll(thicketPhrasesAllSent);
+ return phrasesAllSent;
}
-/* check that one phrase is subphrase of another by lemma (ignoring other node properties)
- * returns true if not found different word
- */
-
+ /* check that one phrase is subphrase of another by lemma (ignoring other node properties)
+ * returns true if not found different word
+ */
+
private boolean identicalSubPhrase(List<ParseTreeNode> lFromFound,
List<ParseTreeNode> lToFound) {
for(int pos=0; pos<lFromFound.size()&& pos<lToFound.size(); pos++){
@@ -143,8 +160,17 @@
private List<ParseTreeNode> append(List<ParseTreeNode> lFromFound,
List<ParseTreeNode> lToFound) {
List<ParseTreeNode> appendList = new ArrayList<ParseTreeNode>();
- appendList.addAll(lFromFound);
- appendList.addAll(lToFound);
+ if (lFromFound.get(0).getPhraseType().equals(lToFound.get(0).getPhraseType())){
+ appendList.addAll(lFromFound);
+ appendList.addAll(lToFound);
+ } else {
+ String pType = lFromFound.get(0).getPhraseType();
+ appendList.addAll(lFromFound);
+ for(ParseTreeNode p: lToFound){
+ p.setPhraseType(pType);
+ appendList.add(p);
+ }
+ }
return appendList;
}
@@ -159,10 +185,10 @@
}
-
-/*
- *
+
+ /*
+ *
[[<1>NP'Iran':NNP], [<2>VP'refuses':VBZ, <3>VP'to':TO, <4>VP'accept':VB, <5>VP'the':DT, <6>VP'UN':NNP,
<7>VP'proposal':NN, <8>VP'to':TO, <9>VP'end':VB, <10>VP'its':PRP$, <11>VP'dispute':NN, <12>VP'over':IN, <13>VP'its':PRP$,
<14>VP'work':NN, <15>VP'on':IN, <16>VP'nuclear':JJ, <17>VP'weapons':NNS], [<3>VP'to':TO, <4>VP'accept':VB, <5>VP'the':DT,
@@ -177,9 +203,9 @@
<14>PP'work':NN, <15>PP'on':IN, <16>PP'nuclear':JJ, <17>PP'weapons':NNS], [<13>NP'its':PRP$, <14>NP'work':NN,
<15>NP'on':IN, <16>NP'nuclear':JJ, <17>NP'weapons':NNS], [<13>NP'its':PRP$, <14>NP'work':NN],
[<15>PP'on':IN, <16>PP'nuclear':JJ, <17>PP'weapons':NNS], [<16>NP'nuclear':JJ, <17>NP'weapons':NNS]]
- *
- *
- */
+ *
+ *
+ */
private void navigateR(Tree t, List<ParseTreeNode> sentence,
List<List<ParseTreeNode>> phrases) {
if (!t.isPreTerminal()) {
@@ -191,7 +217,17 @@
if (!nodes.isEmpty())
phrases.add(nodes);
if (nodes.size()>0 && nodes.get(0).getId()==null){
- System.err.println("Failed alignment:"+nodes);
+ if (nodes.size()>1 && nodes.get(1)!=null && nodes.get(1).getId()!=null){
+ try {
+ ParseTreeNode n = nodes.get(0);
+ n.setId(nodes.get(1).getId()-1);
+ nodes.set(0, n);
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ } else {
+ log.severe("Failed alignment:"+nodes);
+ }
}
}
}
@@ -204,22 +240,22 @@
return ;
}
}
-
-
+
+
/* alignment of phrases extracted from tree against the sentence as a list of lemma-pos */
-
+
private List<ParseTreeNode> assignIndexToNodes(List<ParseTreeNode> node,
List<ParseTreeNode> sentence) {
if (sentence==null || sentence.size()<1)
return node;
-
+
List<ParseTreeNode> results = new ArrayList<ParseTreeNode>();
-
+
for(int i= 0; i<node.size(); i++){
String thisLemma = node.get(i).getWord();
String thisPOS = node.get(i).getPos();
String nextLemma = null, nextPOS = null;
-
+
if (i+1<node.size()){
nextLemma = node.get(i+1).getWord();
nextPOS = node.get(i+1).getPos();
@@ -231,20 +267,21 @@
continue;
if (i+1<node.size() && j+1 < sentence.size() && nextLemma!=null
&& ! (sentence.get(j+1).getWord().equals(nextLemma)
- && sentence.get(j+1).getPos().equals(nextPOS)))
+ && sentence.get(j+1).getPos().equals(nextPOS)))
continue;
matchOccurred = true;
break;
}
-
+
ParseTreeNode n = node.get(i);
if (matchOccurred){
n.setId(sentence.get(j).getId());
n.setNe(sentence.get(j).getNe());
+ n.setAttributes(sentence.get(j).getAttributes());
}
results.add(n);
}
-
+
try {
if (results!=null && results.size()>1 && results.get(0)!=null && results.get(0).getId()!=null &&
results.get(1) !=null && results.get(1).getId()!=null && results.get(1).getId()>0){
@@ -313,53 +350,55 @@
return nlist;
if (value.equals("ROOT")|| value.equals("S"))
return nlist;
-
+
String[] pos_value = value.split(" ");
ParseTreeNode node = null;
if (value.endsWith("P")){
node = new ParseTreeNode("", "");
- node.setPhraseType(value);
+ node.setPhraseType(value);
} else
- if (pos_value != null && pos_value.length==2){
- node = new ParseTreeNode(pos_value[0], pos_value[1]);
- } else {
- node = new ParseTreeNode(value, "");
- }
-
+ if (pos_value != null && pos_value.length==2){
+ node = new ParseTreeNode(pos_value[0], pos_value[1]);
+ } else {
+ node = new ParseTreeNode(value, "");
+ }
+
nlist.add(node);
return nlist;
}
-
+
private ParseTreeNode parsePhraseNode(String value) {
-
+
if (value.equals("ROOT")|| value.equals("S"))
return null;
-
+
String[] pos_value = value.split(" ");
ParseTreeNode node = null;
if (value.endsWith("P")){
node = new ParseTreeNode("", "");
- node.setPhraseType(value);
+ node.setPhraseType(value);
} else
- if (pos_value != null && pos_value.length==2){
- node = new ParseTreeNode(pos_value[0], pos_value[1]);
- } else {
- node = new ParseTreeNode(value, "");
- }
-
+ if (pos_value != null && pos_value.length==2){
+ node = new ParseTreeNode(pos_value[0], pos_value[1]);
+ } else {
+ node = new ParseTreeNode(value, "");
+ }
+
return node;
}
-
+
public List<ParseTreeNode> parsePhrase(String value, String fullDump) {
-
+
List<ParseTreeNode> nlist = new ArrayList<ParseTreeNode>();
if (value.equals("S")|| value.equals("ROOT"))
- return nlist;
+ return nlist;
+ // first phrase type normalization
+ fullDump = fullDump.replace("NP-TMP", "NP");
String flattened = fullDump.replace("(ROOT","").replace("(NP","").replace("(VP","").replace("(PP","")
.replace("(ADVP","").replace("(UCP","").replace("(ADJP","").replace("(SBAR","").
replace("(PRT", "").replace("(WHNP","").
- replace("))))",")").replace(")))",")").replace("))",")")
+ replace("))))",")").replace(")))",")").replace("))",")")
.replace(" ", " ").replace(" ", " ").replace("(S","")
.replace(") (","#").replace(") (", "#");
String[] flattenedArr = flattened.split("#");
@@ -373,9 +412,9 @@
}
return nlist;
}
-
-/* recursion example */
-
+
+ /* recursion example */
+
private StringBuilder toStringBuilder(StringBuilder sb, Tree t) {
if (t.isLeaf()) {
if (t.label() != null) {
@@ -399,23 +438,40 @@
return sb.append(')');
}
}
-
+
public static void main(String[] args){
+ Matcher matcher = new Matcher();
+ String para =
+ "Last Wednesday, world powers reached agreement with Iran on limiting Iranian nuclear activity in return for the lifting of sanctions. "
+ /*+
+ "The Israeli Prime Minister called the deal an historic mistake which would only make it easier for Iran to back its proxies in the Middle East. "+
+ "That position may have hardened after Iran's supreme leader Ayatollah Ali Khamenei said his country would continue its support for the people of Palestine after the deal. "+
+ "Saudi Arabia has officially said it supports the deal, although it is also thought to have similar concerns to Israel that the agreement legitimises Iran. "
+ */
+ ;
+ matcher.buildParseThicketFromTextWithRST(para);
+
+
PT2ThicketPhraseBuilder phraseBuilder = new PT2ThicketPhraseBuilder();
String line = "(NP (NNP Iran)) (VP (VBZ refuses) (S (VP (TO to) (VP (VB accept) (S (NP (DT the) " +
"(NNP UN) (NN proposal)) (VP (TO to) (VP (VB end) (NP (PRP$ its) (NN dispute))))))))";
-
+
List<ParseTreeNode> res = phraseBuilder. parsePhrase("NP", line);
System.out.println(res);
-
+
line = "(VP (VBP am) (NP (NP (DT a) (NNP US) (NN citizen)) (UCP (VP (VBG living) (ADVP (RB abroad))) (, ,) (CC and) (ADJP (JJ concerned) (PP (IN about) (NP (NP (DT the) (NN health) (NN reform) (NN regulation)) (PP (IN of) (NP (CD 2014)))))))))";
res = phraseBuilder. parsePhrase("VP", line);
System.out.println(res);
-
+
line = "(VP (TO to) (VP (VB wait) (SBAR (IN till) (S (NP (PRP I)) (VP (VBP am) (ADJP (JJ sick) (S (VP (TO to) (VP (VB buy) (NP (NN health) (NN insurance)))))))))))";
res = phraseBuilder. parsePhrase("VP", line);
System.out.println(res);
}
-
+
}
+/*
+ * The Ukrainian government, Western leaders and Nato all say there is clear evidence that Russia is helping the rebels in the eastern Donetsk and Luhansk regions with heavy weapons and soldiers. Independent experts echo that accusation.
+Moscow denies it, insisting that any Russians serving with the rebels are volunteers.
+
+*/
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeChunkListScorer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeChunkListScorer.java
deleted file mode 100644
index 21e7f52..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeChunkListScorer.java
+++ /dev/null
@@ -1,96 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.parse_thicket.matching;
-
-import java.util.List;
-
-public class ParseTreeChunkListScorer {
- // find the single expression with the highest score
- public double getParseTreeChunkListScore(
- List<List<ParseTreePath>> matchResult) {
- double currScore = 0.0;
- for (List<ParseTreePath> chunksGivenPhraseType : matchResult)
- for (ParseTreePath chunk : chunksGivenPhraseType) {
- Double score = getScore(chunk);
- // System.out.println(chunk+ " => score >>> "+score);
- if (score > currScore) {
- currScore = score;
- }
- }
- return currScore;
- }
-
- // get max score per phrase type and then sum up
- public double getParseTreeChunkListScoreAggregPhraseType(
- List<List<ParseTreePath>> matchResult) {
- double currScoreTotal = 0.0;
- for (List<ParseTreePath> chunksGivenPhraseType : matchResult) {
- double currScorePT = 0.0;
- for (ParseTreePath chunk : chunksGivenPhraseType) {
- Double score = getScore(chunk);
- // System.out.println(chunk+ " => score >>> "+score);
- if (score > currScorePT) {
- currScorePT = score;
- }
- }
- // if substantial for given phrase type
- if (currScorePT > 0.5) {
- currScoreTotal += currScorePT;
- }
- }
- return currScoreTotal;
- }
-
- // score is meaningful only for chunks which are results of generalization
-
- public double getScore(ParseTreePath chunk) {
- double score = 0.0;
- int i = 0;
- for (String l : chunk.getLemmas()) {
- String pos = chunk.getPOSs().get(i);
- if (l.equals("*")) {
- if (pos.startsWith("CD")) { // number vs number gives high score
- // although different numbers
- score += 0.7;
- } else if (pos.endsWith("_high")) { // if query modification adds 'high'
- score += 1.0;
- } else {
- score += 0.1;
- }
- } else {
-
- if (pos.startsWith("NN") || pos.startsWith("NP")
- || pos.startsWith("CD") || pos.startsWith("RB")) {
- score += 1.0;
- } else if (pos.startsWith("VB") || pos.startsWith("JJ")) {
- if (l.equals("get")) { // 'common' verbs are not that important
- score += 0.3;
- } else {
- score += 0.5;
- }
- } else {
- score += 0.3;
- }
- }
- i++;
-
- }
- return score;
- }
-
-}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePath.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePath.java
deleted file mode 100644
index d0bf61f..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePath.java
+++ /dev/null
@@ -1,422 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.parse_thicket.matching;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import opennlp.tools.textsimilarity.LemmaPair;
-
-public class ParseTreePath {
- private String mainPOS;
-
- private List<String> lemmas;
-
- private List<String> POSs;
- //order number of a word in a sentence
- private List<Integer> wordUniqueCodes;
-
- private int startPos;
-
- private int endPos;
-
- private int size;
-
- private ParseTreePathMatcher parseTreeMatcher;
-
- private LemmaFormManager lemmaFormManager;
-
- private GeneralizationListReducer generalizationListReducer;
-
- public ParseTreePath() {
- }
-
- public ParseTreePath(List<String> lemmas, List<String> POSs, int startPos,
- int endPos) {
- this.lemmas = lemmas;
- this.POSs = POSs;
- this.startPos = startPos;
- this.endPos = endPos;
-
- }
-
- // constructor which takes lemmas and POS as lists so that phrases can be
- // conveniently specified.
- // usage: stand-alone runs
- public ParseTreePath(String mPOS, String[] lemmas, String[] POSss) {
- this.mainPOS = mPOS;
- this.lemmas = new ArrayList<String>();
- for (String l : lemmas) {
- this.lemmas.add(l);
- }
- if (mPOS.equals("SENTENCE")){
- for(int i=0; i<lemmas.length; i++){
- wordUniqueCodes.add(this.lemmas.get(i).hashCode());
- }
- }
-
- this.POSs = new ArrayList<String>();
- for (String p : POSss) {
- this.POSs.add(p);
- }
- }
-
- // constructor which takes lemmas and POS as lists so that phrases can be
- // conveniently specified.
- // usage: stand-alone runs
- public ParseTreePath(String mPOS, List<String> lemmas, List<String> POSss) {
- this.mainPOS = mPOS;
- this.lemmas = lemmas;
- this.POSs = POSss;
-
- }
-
- // Before:
- // [0(S-At home we like to eat great pizza deals), 0(PP-At home), 0(IN-At),
- // 3(NP-home), 3(NN-home), 8(NP-we),
- // 8(PRP-we), 11(VP-like to eat great pizza deals), 11(VBP-like), 16(S-to eat
- // great pizza deals), 16(VP-to eat great
- // pizza deals),
- // 16(TO-to), 19(VP-eat great pizza deals), 19(VB-eat), 23(NP-great pizza
- // deals), 23(JJ-great), 29(NN-pizza),
- // 35(NNS-deals)]
-
- // After:
- // [S [IN-At NP-home NP-we VBP-like ], PP [IN-At NP-home ], IN [IN-At ], NP
- // [NP-home ], NN [NP-home ], NP [NP-we ],
- // PRP [NP-we ], VP [VBP-like TO-to VB-eat JJ-great ], VBP [VBP-like ], S
- // [TO-to VB-eat JJ-great NN-pizza ], VP
- // [TO-to VB-eat JJ-great NN-pizza ], TO [TO-to ], VP [VB-eat JJ-great
- // NN-pizza NNS-deals ],
- // VB [VB-eat ], NP [JJ-great NN-pizza NNS-deals ], JJ [JJ-great ], NN
- // [NN-pizza ], NNS [NNS-deals ]]
-
- public List<ParseTreePath> buildChunks(List<LemmaPair> parseResults) {
- List<ParseTreePath> chunksResults = new ArrayList<ParseTreePath>();
- for (LemmaPair chunk : parseResults) {
- String[] lemmasAr = chunk.getLemma().split(" ");
- List<String> poss = new ArrayList<String>(), lems = new ArrayList<String>();
- for (String lem : lemmasAr) {
- lems.add(lem);
- // now looking for POSs for individual word
- for (LemmaPair chunkCur : parseResults) {
- if (chunkCur.getLemma().equals(lem)
- &&
- // check that this is a proper word in proper position
- chunkCur.getEndPos() <= chunk.getEndPos()
- && chunkCur.getStartPos() >= chunk.getStartPos()) {
- poss.add(chunkCur.getPOS());
- break;
- }
- }
- }
- if (lems.size() != poss.size()) {
- System.err.println("lems.size()!= poss.size()");
- }
- if (lems.size() < 2) { // single word phrase, nothing to match
- continue;
- }
- ParseTreePath ch = new ParseTreePath(lems, poss, chunk.getStartPos(),
- chunk.getEndPos());
- ch.setMainPOS(chunk.getPOS());
- chunksResults.add(ch);
- }
- return chunksResults;
- }
-
- public List<List<ParseTreePath>> matchTwoSentencesGivenPairLists(
- List<LemmaPair> sent1Pairs, List<LemmaPair> sent2Pairs) {
-
- List<ParseTreePath> chunk1List = buildChunks(sent1Pairs);
- List<ParseTreePath> chunk2List = buildChunks(sent2Pairs);
-
- List<List<ParseTreePath>> sent1GrpLst = groupChunksAsParses(chunk1List);
- List<List<ParseTreePath>> sent2GrpLst = groupChunksAsParses(chunk2List);
-
- System.out.println("=== Grouped chunks 1 " + sent1GrpLst);
- System.out.println("=== Grouped chunks 2 " + sent2GrpLst);
-
- return matchTwoSentencesGroupedChunks(sent1GrpLst, sent2GrpLst);
- }
-
- // groups noun phrases, verb phrases, propos phrases etc. for separate match
-
- public List<List<ParseTreePath>> groupChunksAsParses(
- List<ParseTreePath> parseResults) {
- List<ParseTreePath> np = new ArrayList<ParseTreePath>(), vp = new ArrayList<ParseTreePath>(), prp = new ArrayList<ParseTreePath>(), sbarp = new ArrayList<ParseTreePath>(), pp = new ArrayList<ParseTreePath>(), adjp = new ArrayList<ParseTreePath>(), whadvp = new ArrayList<ParseTreePath>(), restOfPhrasesTypes = new ArrayList<ParseTreePath>();
- List<List<ParseTreePath>> results = new ArrayList<List<ParseTreePath>>();
- for (ParseTreePath ch : parseResults) {
- String mainPos = ch.getMainPOS().toLowerCase();
-
- if (mainPos.equals("s")) {
- continue;
- }
- if (mainPos.equals("np")) {
- np.add(ch);
- } else if (mainPos.equals("vp")) {
- vp.add(ch);
- } else if (mainPos.equals("prp")) {
- prp.add(ch);
- } else if (mainPos.equals("pp")) {
- pp.add(ch);
- } else if (mainPos.equals("adjp")) {
- adjp.add(ch);
- } else if (mainPos.equals("whadvp")) {
- whadvp.add(ch);
- } else if (mainPos.equals("sbar")) {
- sbarp.add(ch);
- } else {
- restOfPhrasesTypes.add(ch);
- }
-
- }
- results.add(np);
- results.add(vp);
- results.add(prp);
- results.add(pp);
- results.add(adjp);
- results.add(whadvp);
- results.add(restOfPhrasesTypes);
-
- return results;
-
- }
-
- // main function to generalize two expressions grouped by phrase types
- // returns a list of generalizations for each phrase type with filtered
- // sub-expressions
- public List<List<ParseTreePath>> matchTwoSentencesGroupedChunks(
- List<List<ParseTreePath>> sent1, List<List<ParseTreePath>> sent2) {
- List<List<ParseTreePath>> results = new ArrayList<List<ParseTreePath>>();
- // first irerate through component
- for (int comp = 0; comp < 2 && // just np & vp
- comp < sent1.size() && comp < sent2.size(); comp++) {
- List<ParseTreePath> resultComps = new ArrayList<ParseTreePath>();
- // then iterate through each phrase in each component
- for (ParseTreePath ch1 : sent1.get(comp)) {
- for (ParseTreePath ch2 : sent2.get(comp)) { // simpler version
- ParseTreePath chunkToAdd = parseTreeMatcher
- .generalizeTwoGroupedPhrasesRandomSelectHighestScoreWithTransforms(
- ch1, ch2);
-
- if (!lemmaFormManager.mustOccurVerifier(ch1, ch2, chunkToAdd)) {
- continue; // if the words which have to stay do not stay, proceed to
- // other elements
- }
- Boolean alreadyThere = false;
- for (ParseTreePath chunk : resultComps) {
- if (chunk.equalsTo(chunkToAdd)) {
- alreadyThere = true;
- break;
- }
-
- if (parseTreeMatcher
- .generalizeTwoGroupedPhrasesRandomSelectHighestScore(chunk,
- chunkToAdd).equalsTo(chunkToAdd)) {
- alreadyThere = true;
- break;
- }
- }
-
- if (!alreadyThere) {
- resultComps.add(chunkToAdd);
- }
-
- List<ParseTreePath> resultCompsReduced = generalizationListReducer
- .applyFilteringBySubsumption(resultComps);
- // if (resultCompsReduced.size() != resultComps.size())
- // System.out.println("reduction of gen list occurred");
- }
- }
- results.add(resultComps);
- }
-
- return results;
- }
-
- public Boolean equals(ParseTreePath ch) {
- List<String> lems = ch.getLemmas();
- List<String> poss = ch.POSs;
-
- if (this.lemmas.size() <= lems.size())
- return false; // sub-chunk should be shorter than chunk
-
- for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
- if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
- poss.get(i))))
- return false;
- }
- return true;
- }
-
- // 'this' is super - chunk of ch, ch is sub-chunk of 'this'
- public Boolean isASubChunk(ParseTreePath ch) {
- List<String> lems = ch.getLemmas();
- List<String> poss = ch.POSs;
-
- if (this.lemmas.size() < lems.size())
- return false; // sub-chunk should be shorter than chunk
-
- for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
- if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
- poss.get(i))))
- return false;
- }
- return true;
- }
-
- public Boolean equalsTo(ParseTreePath ch) {
- List<String> lems = ch.getLemmas();
- List<String> poss = ch.POSs;
- if (this.lemmas.size() != lems.size() || this.POSs.size() != poss.size())
- return false;
-
- for (int i = 0; i < lems.size(); i++) {
- if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
- poss.get(i))))
- return false;
- }
-
- return true;
- }
-
- public String toString() {
- String buf = " [";
- if (mainPOS != null)
- buf = mainPOS + " [";
- for (int i = 0; i < lemmas.size() && i < POSs.size() // && i<=3
- ; i++) {
- buf += POSs.get(i) + "-" + lemmas.get(i) + " ";
- }
- return buf + "]";
- }
-
- public int compareTo(ParseTreePath o) {
- if (this.size > o.size)
- return -1;
- else
- return 1;
-
- }
-
- public String listToString(List<List<ParseTreePath>> chunks) {
- StringBuffer buf = new StringBuffer();
- if (chunks.get(0).size() > 0) {
- buf.append(" np " + chunks.get(0).toString());
- }
- if (chunks.get(1).size() > 0) {
- buf.append(" vp " + chunks.get(1).toString());
- }
- if (chunks.size() < 3) {
- return buf.toString();
- }
- if (chunks.get(2).size() > 0) {
- buf.append(" prp " + chunks.get(2).toString());
- }
- if (chunks.get(3).size() > 0) {
- buf.append(" pp " + chunks.get(3).toString());
- }
- if (chunks.get(4).size() > 0) {
- buf.append(" adjp " + chunks.get(4).toString());
- }
- if (chunks.get(5).size() > 0) {
- buf.append(" whadvp " + chunks.get(5).toString());
- }
- /*
- * if (mainPos.equals("np")) np.add(ch); else if (mainPos.equals( "vp"))
- * vp.add(ch); else if (mainPos.equals( "prp")) prp.add(ch); else if
- * (mainPos.equals( "pp")) pp.add(ch); else if (mainPos.equals( "adjp"))
- * adjp.add(ch); else if (mainPos.equals( "whadvp")) whadvp.add(ch);
- */
- return buf.toString();
- }
-
- public List<List<ParseTreePath>> obtainParseTreeChunkListByParsingList(
- String toParse) {
- List<List<ParseTreePath>> results = new ArrayList<List<ParseTreePath>>();
- // if (toParse.endsWith("]]]")){
- // toParse = toParse.replace("[[","").replace("]]","");
- // }
- toParse = toParse.replace(" ]], [ [", "&");
- String[] phraseTypeFragments = toParse.trim().split("&");
- for (String toParseFragm : phraseTypeFragments) {
- toParseFragm = toParseFragm.replace("], [", "#");
-
- List<ParseTreePath> resultsPhraseType = new ArrayList<ParseTreePath>();
- String[] indivChunks = toParseFragm.trim().split("#");
- for (String expr : indivChunks) {
- List<String> lems = new ArrayList<String>(), poss = new ArrayList<String>();
- expr = expr.replace("[", "").replace(" ]", "");
- String[] pairs = expr.trim().split(" ");
- for (String word : pairs) {
- word = word.replace("]]", "").replace("]", "");
- String[] pos_lem = word.split("-");
- lems.add(pos_lem[1].trim());
- poss.add(pos_lem[0].trim());
- }
- ParseTreePath ch = new ParseTreePath();
- ch.setLemmas(lems);
- ch.setPOSs(poss);
- resultsPhraseType.add(ch);
- }
- results.add(resultsPhraseType);
- }
- System.out.println(results);
- return results;
-
- // 2.1 | Vietnam <b>embassy</b> <b>in</b> <b>Israel</b>: information on how
- // to get your <b>visa</b> at Vietnam
- // <b>embassy</b> <b>in</b> <b>Israel</b>. <b>...</b> <b>Spain</b>.
- // Scotland. Sweden. Slovakia. Switzerland. T
- // [Top of Page] <b>...</b>
- // [[ [NN-* IN-in NP-israel ], [NP-* IN-in NP-israel ], [NP-* IN-* TO-* NN-*
- // ], [NN-visa IN-* NN-* IN-in ]], [
- // [VB-get NN-visa IN-* NN-* IN-in .-* ], [VBD-* IN-* NN-* NN-* .-* ], [VB-*
- // NP-* ]]]
-
- }
-
- public void setMainPOS(String mainPOS) {
- this.mainPOS = mainPOS;
- }
-
- public String getMainPOS() {
- return mainPOS;
- }
-
- public List<String> getLemmas() {
- return lemmas;
- }
-
- public void setLemmas(List<String> lemmas) {
- this.lemmas = lemmas;
- }
-
- public List<String> getPOSs() {
- return POSs;
- }
-
- public void setPOSs(List<String> pOSs) {
- POSs = pOSs;
- }
-
- public ParseTreePathMatcher getParseTreeMatcher() {
- return parseTreeMatcher;
- }
-
-}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathComparable.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathComparable.java
deleted file mode 100644
index 539c61e..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathComparable.java
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.parse_thicket.matching;
-
-import java.util.Comparator;
-
-public class ParseTreePathComparable implements Comparator<ParseTreePath> {
- public int compare(ParseTreePath ch1, ParseTreePath ch2) {
- for (int i = 0; i < ch1.getLemmas().size() && i < ch2.getLemmas().size(); i++) {
- if (!(ch1.getLemmas().get(i).equals(ch2.getLemmas().get(i)) && ch1
- .getPOSs().get(i).equals(ch2.getPOSs().get(i))))
- return -1;
- }
- return 0;
-
- }
-}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathMatcher.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathMatcher.java
deleted file mode 100644
index 7323a8e..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathMatcher.java
+++ /dev/null
@@ -1,254 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.parse_thicket.matching;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-
-import opennlp.tools.textsimilarity.POSManager;
-
-public class ParseTreePathMatcher {
-
- private static final int NUMBER_OF_ITERATIONS = 2;
-
- private ParseTreeChunkListScorer parseTreeChunkListScorer = new ParseTreeChunkListScorer();
- private POSManager posManager = new POSManager();
- private LemmaFormManager lemmaFormManager = new LemmaFormManager();
-
- public ParseTreePathMatcher() {
-
- }
-
- public ParseTreePath generalizeTwoGroupedPhrasesOLD(ParseTreePath chunk1,
- ParseTreePath chunk2) {
- List<String> pos1 = chunk1.getPOSs();
- List<String> pos2 = chunk1.getPOSs();
-
- List<String> commonPOS = new ArrayList<String>(), commonLemmas = new ArrayList<String>();
- int k1 = 0, k2 = 0;
- Boolean incrFirst = true;
- while (k1 < pos1.size() && k2 < pos2.size()) {
- // first check if the same POS
- String sim = posManager.similarPOS(pos1.get(k1), pos2.get(k2));
- if (sim != null) {
- commonPOS.add(pos1.get(k1));
- if (chunk1.getLemmas().size() > k1 && chunk2.getLemmas().size() > k2
- && chunk1.getLemmas().get(k1).equals(chunk2.getLemmas().get(k2))) {
- commonLemmas.add(chunk1.getLemmas().get(k1));
- } else {
- commonLemmas.add("*");
- }
- k1++;
- k2++;
- } else if (incrFirst) {
- k1++;
- } else {
- k2++;
- }
- incrFirst = !incrFirst;
- }
-
- ParseTreePath res = new ParseTreePath(commonLemmas, commonPOS, 0, 0);
- // if (parseTreeChunkListScorer.getScore(res)> 0.6)
- // System.out.println(chunk1 + " + \n"+ chunk2 + " = \n" + res);
- return res;
- }
-
- // A for B => B have A
- // transforms expr { A B C prep X Y }
- // into {A B {X Y} C}
- // should only be applied to a noun phrase
- public ParseTreePath prepositionalNNSTransform(ParseTreePath ch) {
- List<String> transfPOS = new ArrayList<String>(), transfLemmas = new ArrayList<String>();
- if (!ch.getPOSs().contains("IN"))
- return ch;
- int indexIN = ch.getPOSs().lastIndexOf("IN");
-
- if (indexIN < 2)// preposition is a first word - should not be in a noun
- // phrase
- return ch;
- String Word_IN = ch.getLemmas().get(indexIN);
- if (!(Word_IN.equals("to") || Word_IN.equals("on") || Word_IN.equals("in")
- || Word_IN.equals("of") || Word_IN.equals("with")
- || Word_IN.equals("by") || Word_IN.equals("from")))
- return ch;
-
- List<String> toShiftAfterPartPOS = ch.getPOSs().subList(indexIN + 1,
- ch.getPOSs().size());
- List<String> toShiftAfterPartLemmas = ch.getLemmas().subList(indexIN + 1,
- ch.getLemmas().size());
-
- if (indexIN - 1 > 0)
- transfPOS.addAll(ch.getPOSs().subList(0, indexIN - 1));
- transfPOS.addAll(toShiftAfterPartPOS);
- transfPOS.add(ch.getPOSs().get(indexIN - 1));
-
- if (indexIN - 1 > 0)
- transfLemmas.addAll(ch.getLemmas().subList(0, indexIN - 1));
- transfLemmas.addAll(toShiftAfterPartLemmas);
- transfLemmas.add(ch.getLemmas().get(indexIN - 1));
-
- return new ParseTreePath(transfLemmas, transfPOS, 0, 0);
- }
-
- public ParseTreePath generalizeTwoGroupedPhrasesRandomSelectHighestScoreWithTransforms(
- ParseTreePath chunk1, ParseTreePath chunk2) {
- ParseTreePath chRes1 = generalizeTwoGroupedPhrasesRandomSelectHighestScore(
- chunk1, chunk2);
- ParseTreePath chRes2 = generalizeTwoGroupedPhrasesRandomSelectHighestScore(
- prepositionalNNSTransform(chunk1), chunk2);
- ParseTreePath chRes3 = generalizeTwoGroupedPhrasesRandomSelectHighestScore(
- prepositionalNNSTransform(chunk2), chunk1);
-
- ParseTreePath chRes = null;
- if (parseTreeChunkListScorer.getScore(chRes1) > parseTreeChunkListScorer
- .getScore(chRes2))
- if (parseTreeChunkListScorer.getScore(chRes1) > parseTreeChunkListScorer
- .getScore(chRes3))
- chRes = chRes1;
- else
- chRes = chRes3;
- else if (parseTreeChunkListScorer.getScore(chRes2) > parseTreeChunkListScorer
- .getScore(chRes3))
- chRes = chRes2;
- else
- chRes = chRes3;
-
- return chRes;
- }
-
- public ParseTreePath generalizeTwoGroupedPhrasesRandomSelectHighestScore(
- ParseTreePath chunk1, ParseTreePath chunk2) {
- List<String> pos1 = chunk1.getPOSs();
- List<String> pos2 = chunk2.getPOSs();
- // Map <ParseTreeChunk, Double> scoredResults = new HashMap <ParseTreeChunk,
- // Double> ();
- int timesRepetitiveRun = NUMBER_OF_ITERATIONS;
-
- Double globalScore = -1.0;
- ParseTreePath result = null;
-
- for (int timesRun = 0; timesRun < timesRepetitiveRun; timesRun++) {
- List<String> commonPOS = new ArrayList<String>(), commonLemmas = new ArrayList<String>();
- int k1 = 0, k2 = 0;
- Double score = 0.0;
- while (k1 < pos1.size() && k2 < pos2.size()) {
- // first check if the same POS
- String sim = posManager.similarPOS(pos1.get(k1), pos2.get(k2));
- String lemmaMatch = lemmaFormManager.matchLemmas(null, chunk1
- .getLemmas().get(k1), chunk2.getLemmas().get(k2), sim);
- // if (LemmaFormManager.acceptableLemmaAndPOS(sim, lemmaMatch)){
- if ((sim != null)
- && (lemmaMatch == null || (lemmaMatch != null && !lemmaMatch
- .equals("fail")))) {
- // if (sim!=null){ // && (lemmaMatch!=null &&
- // !lemmaMatch.equals("fail"))){
- commonPOS.add(pos1.get(k1));
- if (chunk1.getLemmas().size() > k1 && chunk2.getLemmas().size() > k2
- && lemmaMatch != null) {
- commonLemmas.add(lemmaMatch);
-
- } else {
- commonLemmas.add("*");
-
- }
- k1++;
- k2++;
- } else if (Math.random() > 0.5) {
- k1++;
- } else {
- k2++;
- }
-
- }
- ParseTreePath currResult = new ParseTreePath(commonLemmas, commonPOS,
- 0, 0);
- score = parseTreeChunkListScorer.getScore(currResult);
- if (score > globalScore) {
- // System.out.println(chunk1 + " + \n"+ chunk2 + " = \n" +
- // result+" score = "+ score +"\n\n");
- result = currResult;
- globalScore = score;
- }
- }
-
- for (int timesRun = 0; timesRun < timesRepetitiveRun; timesRun++) {
- List<String> commonPOS = new ArrayList<String>(), commonLemmas = new ArrayList<String>();
- int k1 = pos1.size() - 1, k2 = pos2.size() - 1;
- Double score = 0.0;
- while (k1 >= 0 && k2 >= 0) {
- // first check if the same POS
- String sim = posManager.similarPOS(pos1.get(k1), pos2.get(k2));
- String lemmaMatch = lemmaFormManager.matchLemmas(null, chunk1
- .getLemmas().get(k1), chunk2.getLemmas().get(k2), sim);
- // if (acceptableLemmaAndPOS(sim, lemmaMatch)){
- if ((sim != null)
- && (lemmaMatch == null || (lemmaMatch != null && !lemmaMatch
- .equals("fail")))) {
- commonPOS.add(pos1.get(k1));
- if (chunk1.getLemmas().size() > k1 && chunk2.getLemmas().size() > k2
- && lemmaMatch != null) {
- commonLemmas.add(lemmaMatch);
- } else {
- commonLemmas.add("*");
-
- }
- k1--;
- k2--;
- } else if (Math.random() > 0.5) {
- k1--;
- } else {
- k2--;
- }
-
- }
- Collections.reverse(commonLemmas);
- Collections.reverse(commonPOS);
-
- ParseTreePath currResult = new ParseTreePath(commonLemmas, commonPOS,
- 0, 0);
- score = parseTreeChunkListScorer.getScore(currResult);
- if (score > globalScore) {
- // System.out.println(chunk1 + " + \n"+ chunk2 + " = \n" +
- // currResult+" score = "+ score +"\n\n");
- result = currResult;
- globalScore = score;
- }
- }
-
- // // System.out.println(chunk1 + " + \n"+ chunk2 + " = \n" + result
- // +" score = " +
- // // parseTreeChunkListScorer.getScore(result)+"\n\n");
- return result;
- }
-
- public Boolean acceptableLemmaAndPOS(String sim, String lemmaMatch) {
- if (sim == null) {
- return false;
- }
-
- if (lemmaMatch != null && !lemmaMatch.equals("fail")) {
- return false;
- }
- // even if lemmaMatch==null
- return true;
- // if (sim!=null && (lemmaMatch!=null && !lemmaMatch.equals("fail"))){
-
- }
-}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathMatcherDeterministic.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathMatcherDeterministic.java
deleted file mode 100644
index fc32380..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreePathMatcherDeterministic.java
+++ /dev/null
@@ -1,280 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.parse_thicket.matching;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import opennlp.tools.stemmer.PorterStemmer;
-import opennlp.tools.textsimilarity.POSManager;
-
-
-public class ParseTreePathMatcherDeterministic {
-
- private GeneralizationListReducer generalizationListReducer = new GeneralizationListReducer();
-
- private LemmaFormManager lemmaFormManager = new LemmaFormManager();
-
- private POSManager posManager = new POSManager();
-
- /**
- * key matching function which takes two phrases, aligns them and finds a set
- * of maximum common sub-phrase
- *
- * @param chunk1
- * @param chunk2
- * @return
- */
-
- public List<ParseTreePath> generalizeTwoGroupedPhrasesDeterministic(
- ParseTreePath chunk1, ParseTreePath chunk2) {
- List<String> pos1 = chunk1.getPOSs();
- List<String> pos2 = chunk2.getPOSs();
- List<String> lem1 = chunk1.getLemmas();
- List<String> lem2 = chunk2.getLemmas();
-
- List<String> lem1stem = new ArrayList<String>();
- List<String> lem2stem = new ArrayList<String>();
-
- PorterStemmer ps = new PorterStemmer();
- for (String word : lem1) {
- try {
- lem1stem.add(ps.stem(word.toLowerCase()).toString());
- } catch (Exception e) {
- // e.printStackTrace();
-
- if (word.length() > 2)
- System.err.println("Unable to stem: " + word);
- }
- }
- try {
- for (String word : lem2) {
- lem2stem.add(ps.stem(word.toLowerCase()).toString());
- }
- } catch (Exception e) {
- System.err.println("problem processing word " + lem2.toString());
- }
-
- List<String> overlap = new ArrayList(lem1stem);
- overlap.retainAll(lem2stem);
-
- if (overlap == null || overlap.size() < 1)
- return null;
-
- List<Integer> occur1 = new ArrayList<Integer>(), occur2 = new ArrayList<Integer>();
- for (String word : overlap) {
- Integer i1 = lem1stem.indexOf(word);
- Integer i2 = lem2stem.indexOf(word);
- occur1.add(i1);
- occur2.add(i2);
- }
-
- // now we search for plausible sublists of overlaps
- // if at some position correspondence is inverse (one of two position
- // decreases instead of increases)
- // then we terminate current alignment accum and start a new one
- List<List<int[]>> overlapsPlaus = new ArrayList<List<int[]>>();
- // starts from 1, not 0
- List<int[]> accum = new ArrayList<int[]>();
- accum.add(new int[] { occur1.get(0), occur2.get(0) });
- for (int i = 1; i < occur1.size(); i++) {
-
- if (occur1.get(i) > occur1.get(i - 1)
- && occur2.get(i) > occur2.get(i - 1))
- accum.add(new int[] { occur1.get(i), occur2.get(i) });
- else {
- overlapsPlaus.add(accum);
- accum = new ArrayList<int[]>();
- accum.add(new int[] { occur1.get(i), occur2.get(i) });
- }
- }
- if (accum.size() > 0) {
- overlapsPlaus.add(accum);
- }
-
- List<ParseTreePath> results = new ArrayList<ParseTreePath>();
- for (List<int[]> occur : overlapsPlaus) {
- List<Integer> occr1 = new ArrayList<Integer>(), occr2 = new ArrayList<Integer>();
- for (int[] column : occur) {
- occr1.add(column[0]);
- occr2.add(column[1]);
- }
-
- int ov1 = 0, ov2 = 0; // iterators over common words;
- List<String> commonPOS = new ArrayList<String>(), commonLemmas = new ArrayList<String>();
- // we start two words before first word
- int k1 = occr1.get(ov1) - 2, k2 = occr2.get(ov2) - 2;
- // if (k1<0) k1=0; if (k2<0) k2=0;
- Boolean bReachedCommonWord = false;
- while (k1 < 0 || k2 < 0) {
- k1++;
- k2++;
- }
- int k1max = pos1.size() - 1, k2max = pos2.size() - 1;
- while (k1 <= k1max && k2 <= k2max) {
- // first check if the same POS
- String sim = posManager.similarPOS(pos1.get(k1), pos2.get(k2));
- String lemmaMatch = lemmaFormManager.matchLemmas(ps, lem1.get(k1),
- lem2.get(k2), sim);
- if ((sim != null)
- && (lemmaMatch == null || (lemmaMatch != null && !lemmaMatch
- .equals("fail")))) {
- commonPOS.add(pos1.get(k1));
- if (lemmaMatch != null) {
- commonLemmas.add(lemmaMatch);
- // System.out.println("Added "+lemmaMatch);
- if (k1 == occr1.get(ov1) && k2 == occr2.get(ov2))
- bReachedCommonWord = true; // now we can have different increment
- // opera
- else {
- if (occr1.size() > ov1 + 1 && occr2.size() > ov2 + 1
- && k1 == occr1.get(ov1 + 1) && k2 == occr2.get(ov2 + 1)) {
- ov1++;
- ov2++;
- bReachedCommonWord = true;
- }
- // else
- // System.err.println("Next match reached '"+lemmaMatch+
- // "' | k1 - k2: "+k1 + " "+k2 +
- // "| occur index ov1-ov2 "+
- // ov1+" "+ov2+
- // "| identified positions of match: occr1.get(ov1) - occr2.get(ov1) "
- // +
- // occr1.get(ov1) + " "+ occr2.get(ov1));
- }
- } else {
- commonLemmas.add("*");
- } // the same parts of speech, proceed to the next word in both
- // expressions
- k1++;
- k2++;
-
- } else if (!bReachedCommonWord) {
- k1++;
- k2++;
- } // still searching
- else {
- // different parts of speech, jump to the next identified common word
- ov1++;
- ov2++;
- if (ov1 > occr1.size() - 1 || ov2 > occr2.size() - 1)
- break;
- // now trying to find
- int kk1 = occr1.get(ov1) - 2, // new positions of iterators
- kk2 = occr2.get(ov2) - 2;
- int countMove = 0;
- while ((kk1 < k1 + 1 || kk2 < k2 + 1) && countMove < 2) { // if it is
- // behind
- // current
- // position,
- // synchroneously
- // move
- // towards
- // right
- kk1++;
- kk2++;
- countMove++;
- }
- k1 = kk1;
- k2 = kk2;
-
- if (k1 > k1max)
- k1 = k1max;
- if (k2 > k2max)
- k2 = k2max;
- bReachedCommonWord = false;
- }
- }
- ParseTreePath currResult = new ParseTreePath(commonLemmas, commonPOS,
- 0, 0);
- results.add(currResult);
- }
-
- return results;
- }
-
- /**
- * main function to generalize two expressions grouped by phrase types returns
- * a list of generalizations for each phrase type with filtered
- * sub-expressions
- *
- * @param sent1
- * @param sent2
- * @return List<List<ParseTreeChunk>> list of list of POS-words pairs for each
- * resultant matched / overlapped phrase
- */
- public List<List<ParseTreePath>> matchTwoSentencesGroupedChunksDeterministic(
- List<List<ParseTreePath>> sent1, List<List<ParseTreePath>> sent2) {
- List<List<ParseTreePath>> results = new ArrayList<List<ParseTreePath>>();
- // first iterate through component
- for (int comp = 0; comp < 2 && // just np & vp
- comp < sent1.size() && comp < sent2.size(); comp++) {
- List<ParseTreePath> resultComps = new ArrayList<ParseTreePath>();
- // then iterate through each phrase in each component
- for (ParseTreePath ch1 : sent1.get(comp)) {
- for (ParseTreePath ch2 : sent2.get(comp)) { // simpler version
- List<ParseTreePath> chunkToAdd = generalizeTwoGroupedPhrasesDeterministic(
- ch1, ch2);
-
- if (chunkToAdd == null)
- chunkToAdd = new ArrayList<ParseTreePath>();
- // System.out.println("ch1 = "+
- // ch1.toString()+" | ch2="+ch2.toString()
- // +"\n result = "+chunkToAdd.toString() + "\n");
- /*
- * List<ParseTreeChunk> chunkToAdd1 =
- * ParseTreeMatcherDeterministic.generalizeTwoGroupedPhrasesDeterministic
- * ( ParseTreeMatcher.prepositionalNNSTransform(ch1), ch2); if
- * (chunkToAdd1!=null) chunkToAdd.addAll(chunkToAdd1);
- * List<ParseTreeChunk> chunkToAdd2 =
- * ParseTreeMatcherDeterministic.generalizeTwoGroupedPhrasesDeterministic
- * ( ParseTreeMatcher.prepositionalNNSTransform(ch2), ch1); if
- * (chunkToAdd2!=null) chunkToAdd.addAll(chunkToAdd2);
- */
-
- // For generalized match not with orig sentences but with templates
- // if (!LemmaFormManager.mustOccurVerifier(ch1, ch2, chunkToAdd))
- // continue; // if the words which have to stay do not stay, proceed
- // to other elements
- Boolean alreadyThere = false;
- for (ParseTreePath chunk : resultComps) {
- if (chunkToAdd.contains(chunk)) {
- alreadyThere = true;
- break;
- }
-
- // }
- }
-
- if (!alreadyThere && chunkToAdd != null && chunkToAdd.size() > 0) {
- resultComps.addAll(chunkToAdd);
- }
-
- }
- }
- List<ParseTreePath> resultCompsRed = generalizationListReducer
- .applyFilteringBySubsumption(resultComps);
-
- resultComps = resultCompsRed;
- results.add(resultComps);
- }
-
- return results;
- }
-
-}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/EdgeProductBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/EdgeProductBuilder.java
index fb97716..6b72e47 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/EdgeProductBuilder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/EdgeProductBuilder.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.parse_thicket.parse_thicket2graph;
import java.util.ArrayList;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/GraphFromPTreeBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/GraphFromPTreeBuilder.java
index bad6403..d19d7db 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/GraphFromPTreeBuilder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/GraphFromPTreeBuilder.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.parse_thicket.parse_thicket2graph;
import java.io.PrintWriter;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseGraphNode.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseGraphNode.java
index 9620499..6f9c3ea 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseGraphNode.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseGraphNode.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.parse_thicket.parse_thicket2graph;
import java.util.List;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseTreeVisualizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseTreeVisualizer.java
index d34d974..71c1fa3 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseTreeVisualizer.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/parse_thicket2graph/ParseTreeVisualizer.java
@@ -1,27 +1,20 @@
-/* ==========================================
- * JGraphT : a free Java graph-theory library
- * ==========================================
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
*
- * Project Info: http://jgrapht.sourceforge.net/
- * Project Creator: Barak Naveh (http://sourceforge.net/users/barak_naveh)
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * (C) Copyright 2003-2008, by Barak Naveh and Contributors.
- *
- * This library is free software; you can redistribute it and/or modify it
- * under the terms of the GNU Lesser General Public License as published by
- * the Free Software Foundation; either version 2.1 of the License, or
- * (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
- * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public
- * License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public License
- * along with this library; if not, write to the Free Software Foundation,
- * Inc.,
- * 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
*/
+
/* ----------------------
* JGraphAdapterDemo.java
* ----------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhraseConcept.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhraseConcept.java
index ecba4b5..d7f3e75 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhraseConcept.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhraseConcept.java
@@ -1,45 +1,84 @@
-package opennlp.tools.parse_thicket.pattern_structure;
-
-import java.util.*;
-import java.io.*;
-
-import opennlp.tools.textsimilarity.ParseTreeChunk;
-
-public class PhraseConcept {
- int position;
- //Set<Integer> intent;
- List<List<ParseTreeChunk>> intent;
- Set<Integer> parents;
- public PhraseConcept() {
- position = -1;
- intent = new ArrayList<List<ParseTreeChunk>>();
- parents = new HashSet<Integer>();
- }
- public void setPosition( int newPosition ){
- position = newPosition;
- }
- public void setIntent( List<List<ParseTreeChunk>> newIntent ){
- intent.clear();
- intent.addAll(newIntent);
- }
- public void setParents( Set<Integer> newParents ){
- //parents = newParents;
- parents.clear();
- parents.addAll(newParents);
- }
- public void printConcept() {
- System.out.println("Concept position:" + position);
- System.out.println("Concept intent:" + intent);
- System.out.println("Concept parents:" + parents);
- }
- public static void main(String []args) {
- PhraseConcept c = new PhraseConcept();
- c.printConcept();
- c.setPosition(10);
- c.printConcept();
- //List<List<ParseTreeChunk>> test = new List<List<ParseTreeChunk>>();
- //c.setIntent(test);
- c.printConcept();
-
- }
-}
\ No newline at end of file
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.pattern_structure;
+
+import java.util.*;
+import java.io.*;
+
+import opennlp.tools.fca.FormalConcept;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+
+public class PhraseConcept {
+ int position;
+ public List<List<ParseTreeChunk>> intent;
+ Set<Integer> parents;
+ Set<Integer> childs;
+ Set<Integer> extent;
+
+ double intLogStabilityBottom = 0;
+ double intLogStabilityUp = 0;
+
+
+ public PhraseConcept() {
+ position = -1;
+ intent = new ArrayList<List<ParseTreeChunk>>();
+ parents = new HashSet<Integer>();
+ extent = new HashSet<Integer>();
+ childs = new HashSet<Integer>();
+ }
+ public void setPosition( int newPosition ){
+ position = newPosition;
+ }
+ public void setIntent( List<List<ParseTreeChunk>> newIntent ){
+ intent.clear();
+ intent.addAll(newIntent);
+ }
+ public void setParents( Set<Integer> newParents ){
+ //parents = newParents;
+ parents.clear();
+ parents.addAll(newParents);
+ }
+ public void printConcept() {
+ System.out.println("Concept position:" + position);
+ System.out.println("Concept intent:" + intent);
+ System.out.println("Concept parents:" + parents);
+ }
+
+ public void printConceptExtended() {
+ System.out.println("Concept position:" + position);
+ System.out.println("Concept intent:" + intent);
+ System.out.println("Concept extent:" + extent);
+ System.out.println("Concept parents:" + parents);
+ System.out.println("Concept parents:" + childs);
+ System.out.println("log stab: ["+ intLogStabilityBottom + "; "+intLogStabilityUp+"]");
+ }
+
+ public void addExtents(LinkedHashSet<Integer> ext){
+ extent.addAll(ext);
+}
+
+
+ public static void main(String []args) {
+ PhraseConcept c = new PhraseConcept();
+ c.printConcept();
+ c.setPosition(10);
+ c.printConcept();
+ c.printConcept();
+
+ }
+}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructure.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructure.java
index 23fd5a3..25d5ac5 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructure.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructure.java
@@ -1,166 +1,192 @@
-package opennlp.tools.parse_thicket.pattern_structure;
-
-import java.util.*;
-import java.io.*;
-
-import opennlp.tools.parse_thicket.ParseCorefsBuilder;
-import opennlp.tools.parse_thicket.ParseThicket;
-import opennlp.tools.parse_thicket.ParseTreeNode;
-import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
-import opennlp.tools.textsimilarity.ParseTreeChunk;
-import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
-
-
-public class PhrasePatternStructure {
- int objectCount;
- int attributeCount;
- ArrayList<PhraseConcept> conceptList;
- ParseTreeMatcherDeterministic md;
- public PhrasePatternStructure(int objectCounts, int attributeCounts) {
- objectCount = objectCounts;
- attributeCount = attributeCounts;
- conceptList = new ArrayList<PhraseConcept>();
- PhraseConcept bottom = new PhraseConcept();
- md = new ParseTreeMatcherDeterministic();
- /*Set<Integer> b_intent = new HashSet<Integer>();
- for (int index = 0; index < attributeCount; ++index) {
- b_intent.add(index);
- }
- bottom.setIntent(b_intent);*/
- bottom.setPosition(0);
- conceptList.add(bottom);
- }
- public int GetMaximalConcept(List<List<ParseTreeChunk>> intent, int Generator) {
- boolean parentIsMaximal = true;
- while(parentIsMaximal) {
- parentIsMaximal = false;
- for (int parent : conceptList.get(Generator).parents) {
- if (conceptList.get(parent).intent.containsAll(intent)) {
- Generator = parent;
- parentIsMaximal = true;
- break;
- }
- }
- }
- return Generator;
- }
- public int AddIntent(List<List<ParseTreeChunk>> intent, int generator) {
- System.out.println("debug");
- System.out.println("called for " + intent);
- //printLattice();
- int generator_tmp = GetMaximalConcept(intent, generator);
- generator = generator_tmp;
- if (conceptList.get(generator).intent.equals(intent)) {
- System.out.println("at generator:" + conceptList.get(generator).intent);
- System.out.println("to add:" + intent);
-
- System.out.println("already generated");
- return generator;
- }
- Set<Integer> generatorParents = conceptList.get(generator).parents;
- Set<Integer> newParents = new HashSet<Integer>();
- for (int candidate : generatorParents) {
- if (!intent.containsAll(conceptList.get(candidate).intent)) {
- //if (!conceptList.get(candidate).intent.containsAll(intent)) {
- //Set<Integer> intersection = new HashSet<Integer>(conceptList.get(candidate).intent);
- //List<List<ParseTreeChunk>> intersection = new ArrayList<List<ParseTreeChunk>>(conceptList.get(candidate).intent);
- //intersection.retainAll(intent);
- List<List<ParseTreeChunk>> intersection = md
- .matchTwoSentencesGroupedChunksDeterministic(intent, conceptList.get(candidate).intent);
- System.out.println("recursive call (inclusion)");
- candidate = AddIntent(intersection, candidate);
- }
- boolean addParents = true;
- System.out.println("now iterating over parents");
- Iterator<Integer> iterator = newParents.iterator();
- while (iterator.hasNext()) {
- Integer parent = iterator.next();
- if (conceptList.get(parent).intent.containsAll(conceptList.get(candidate).intent)) {
- addParents = false;
- break;
- }
- else {
- if (conceptList.get(candidate).intent.containsAll(conceptList.get(parent).intent)) {
- iterator.remove();
- }
- }
- }
- /*for (int parent : newParents) {
- System.out.println("parent = " + parent);
- System.out.println("candidate intent:"+conceptList.get(candidate).intent);
- System.out.println("parent intent:"+conceptList.get(parent).intent);
-
- if (conceptList.get(parent).intent.containsAll(conceptList.get(candidate).intent)) {
- addParents = false;
- break;
- }
- else {
- if (conceptList.get(candidate).intent.containsAll(conceptList.get(parent).intent)) {
- newParents.remove(parent);
- }
- }
- }*/
- if (addParents) {
- newParents.add(candidate);
- }
- }
- System.out.println("size of lattice: " + conceptList.size());
- PhraseConcept newConcept = new PhraseConcept();
- newConcept.setIntent(intent);
- newConcept.setPosition(conceptList.size());
- conceptList.add(newConcept);
- conceptList.get(generator).parents.add(newConcept.position);
- for (int newParent: newParents) {
- if (conceptList.get(generator).parents.contains(newParent)) {
- conceptList.get(generator).parents.remove(newParent);
- }
- conceptList.get(newConcept.position).parents.add(newParent);
- }
- return newConcept.position;
- }
- public void printLatticeStats() {
- System.out.println("Lattice stats");
- System.out.println("max_object_index = " + objectCount);
- System.out.println("max_attribute_index = " + attributeCount);
- System.out.println("Current concept count = " + conceptList.size());
- }
- public void printLattice() {
- for (int i = 0; i < conceptList.size(); ++i) {
- printConceptByPosition(i);
- }
- }
- public void printConceptByPosition(int index) {
- System.out.println("Concept at position " + index);
- conceptList.get(index).printConcept();
- }
- public List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(
- List<List<ParseTreeNode>> phrs) {
- List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
- List<ParseTreeChunk> nps = new ArrayList<ParseTreeChunk>(), vps = new ArrayList<ParseTreeChunk>(),
- pps = new ArrayList<ParseTreeChunk>();
- for(List<ParseTreeNode> ps:phrs){
- ParseTreeChunk ch = convertNodeListIntoChunk(ps);
- String ptype = ps.get(0).getPhraseType();
- if (ptype.equals("NP")){
- nps.add(ch);
- } else if (ptype.equals("VP")){
- vps.add(ch);
- } else if (ptype.equals("PP")){
- pps.add(ch);
- }
- }
- results.add(nps); results.add(vps); results.add(pps);
- return results;
- }
- private ParseTreeChunk convertNodeListIntoChunk(List<ParseTreeNode> ps) {
- List<String> lemmas = new ArrayList<String>(), poss = new ArrayList<String>();
- for(ParseTreeNode n: ps){
- lemmas.add(n.getWord());
- poss.add(n.getPos());
- }
- ParseTreeChunk ch = new ParseTreeChunk(lemmas, poss, 0, 0);
- ch.setMainPOS(ps.get(0).getPhraseType());
- return ch;
- }
-
-}
\ No newline at end of file
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.pattern_structure;
+
+
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
+
+
+public class PhrasePatternStructure {
+ int objectCount;
+ int attributeCount;
+ public List<PhraseConcept> conceptList;
+ ParseTreeMatcherDeterministic md;
+ public PhrasePatternStructure(int objectCounts, int attributeCounts) {
+ objectCount = objectCounts;
+ attributeCount = attributeCounts;
+ conceptList = new ArrayList<PhraseConcept>();
+ PhraseConcept bottom = new PhraseConcept();
+ md = new ParseTreeMatcherDeterministic();
+ /*Set<Integer> b_intent = new HashSet<Integer>();
+ for (int index = 0; index < attributeCount; ++index) {
+ b_intent.add(index);
+ }
+ bottom.setIntent(b_intent);*/
+ bottom.setPosition(0);
+ conceptList.add(bottom);
+ }
+ public int GetMaximalConcept(List<List<ParseTreeChunk>> intent, int Generator) {
+ boolean parentIsMaximal = true;
+ while(parentIsMaximal) {
+ parentIsMaximal = false;
+ for (int parent : conceptList.get(Generator).parents) {
+ if (conceptList.get(parent).intent.containsAll(intent)) {
+ Generator = parent;
+ parentIsMaximal = true;
+ break;
+ }
+ }
+ }
+ return Generator;
+ }
+ public int AddIntent(List<List<ParseTreeChunk>> intent, int generator) {
+ System.out.println("debug");
+ System.out.println("called for " + intent);
+ //printLattice();
+ int generator_tmp = GetMaximalConcept(intent, generator);
+ generator = generator_tmp;
+ if (conceptList.get(generator).intent.equals(intent)) {
+ System.out.println("at generator:" + conceptList.get(generator).intent);
+ System.out.println("to add:" + intent);
+ System.out.println("already generated");
+ return generator;
+ }
+ Set<Integer> generatorParents = conceptList.get(generator).parents;
+ Set<Integer> newParents = new HashSet<Integer>();
+ for (int candidate : generatorParents) {
+ if (!intent.containsAll(conceptList.get(candidate).intent)) {
+ //if (!conceptList.get(candidate).intent.containsAll(intent)) {
+ //Set<Integer> intersection = new HashSet<Integer>(conceptList.get(candidate).intent);
+ //List<List<ParseTreeChunk>> intersection = new ArrayList<List<ParseTreeChunk>>(conceptList.get(candidate).intent);
+ //intersection.retainAll(intent);
+ List<List<ParseTreeChunk>> intersection = md
+ .matchTwoSentencesGroupedChunksDeterministic(intent, conceptList.get(candidate).intent);
+ System.out.println("recursive call (inclusion)");
+ candidate = AddIntent(intersection, candidate);
+ }
+ boolean addParents = true;
+ System.out.println("now iterating over parents");
+ Iterator<Integer> iterator = newParents.iterator();
+ while (iterator.hasNext()) {
+ Integer parent = iterator.next();
+ if (conceptList.get(parent).intent.containsAll(conceptList.get(candidate).intent)) {
+ addParents = false;
+ break;
+ }
+ else {
+ if (conceptList.get(candidate).intent.containsAll(conceptList.get(parent).intent)) {
+ iterator.remove();
+ }
+ }
+ }
+ /*for (int parent : newParents) {
+ System.out.println("parent = " + parent);
+ System.out.println("candidate intent:"+conceptList.get(candidate).intent);
+ System.out.println("parent intent:"+conceptList.get(parent).intent);
+
+ if (conceptList.get(parent).intent.containsAll(conceptList.get(candidate).intent)) {
+ addParents = false;
+ break;
+ }
+ else {
+ if (conceptList.get(candidate).intent.containsAll(conceptList.get(parent).intent)) {
+ newParents.remove(parent);
+ }
+ }
+ }*/
+ if (addParents) {
+ newParents.add(candidate);
+ }
+ }
+ System.out.println("size of lattice: " + conceptList.size());
+ PhraseConcept newConcept = new PhraseConcept();
+ newConcept.setIntent(intent);
+ newConcept.setPosition(conceptList.size());
+ conceptList.add(newConcept);
+ conceptList.get(generator).parents.add(newConcept.position);
+ for (int newParent: newParents) {
+ if (conceptList.get(generator).parents.contains(newParent)) {
+ conceptList.get(generator).parents.remove(newParent);
+ }
+ conceptList.get(newConcept.position).parents.add(newParent);
+ }
+ return newConcept.position;
+ }
+
+ public void printLatticeStats() {
+ System.out.println("Lattice stats");
+ System.out.println("max_object_index = " + objectCount);
+ System.out.println("max_attribute_index = " + attributeCount);
+ System.out.println("Current concept count = " + conceptList.size());
+
+ }
+
+ public void printLattice() {
+ for (int i = 0; i < conceptList.size(); ++i) {
+ printConceptByPosition(i);
+ }
+ }
+
+ public void printConceptByPosition(int index) {
+ System.out.println("Concept at position " + index);
+ conceptList.get(index).printConcept();
+ }
+
+ public List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(
+ List<List<ParseTreeNode>> phrs) {
+ List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
+ List<ParseTreeChunk> nps = new ArrayList<ParseTreeChunk>(), vps = new ArrayList<ParseTreeChunk>(),
+ pps = new ArrayList<ParseTreeChunk>();
+ for(List<ParseTreeNode> ps:phrs){
+ ParseTreeChunk ch = convertNodeListIntoChunk(ps);
+ String ptype = ps.get(0).getPhraseType();
+ System.out.println(ps);
+ if (ptype.equals("NP")){
+ nps.add(ch);
+ } else if (ptype.equals("VP")){
+ vps.add(ch);
+ } else if (ptype.equals("PP")){
+ pps.add(ch);
+ }
+ }
+ results.add(nps); results.add(vps); results.add(pps);
+ return results;
+ }
+
+ private ParseTreeChunk convertNodeListIntoChunk(List<ParseTreeNode> ps) {
+ List<String> lemmas = new ArrayList<String>(), poss = new ArrayList<String>();
+ for(ParseTreeNode n: ps){
+ lemmas.add(n.getWord());
+ poss.add(n.getPos());
+ }
+ ParseTreeChunk ch = new ParseTreeChunk(lemmas, poss, 0, 0);
+ ch.setMainPOS(ps.get(0).getPhraseType());
+ return ch;
+ }
+
+
+}
+
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureArcsBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureArcsBuilder.java
index 3a36e80..96bec44 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureArcsBuilder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureArcsBuilder.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.parse_thicket.rhetoric_structure;
import java.util.ArrayList;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureMarker.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureMarker.java
index 060d32f..3b1c576 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureMarker.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/rhetoric_structure/RhetoricStructureMarker.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.parse_thicket.rhetoric_structure;
import java.util.ArrayList;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingQueryRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingQueryRunner.java
index c9b1f76..cd0e541 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingQueryRunner.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingQueryRunner.java
@@ -21,6 +21,8 @@
import java.util.List;
import java.util.logging.Logger;
+import org.apache.commons.lang.StringUtils;
+
import net.billylieurance.azuresearch.AzureSearchImageQuery;
import net.billylieurance.azuresearch.AzureSearchImageResult;
import net.billylieurance.azuresearch.AzureSearchResultSet;
@@ -29,7 +31,11 @@
public class BingQueryRunner {
- protected static String BING_KEY = "e8ADxIjn9YyHx36EihdjH/tMqJJItUrrbPTUpKahiU0=";
+ protected static String BING_KEY =
+ "WFoNMM706MMJ5JYfcHaSEDP+faHj3xAxt28CPljUAHA";
+ //"pjtCgujmf9TtfjCVBdcQ2rBUQwGLmtLtgCG4Ex7kekw";
+ //"e8ADxIjn9YyHx36EihdjH/tMqJJItUrrbPTUpKahiU0=";
+ //"Cec1TlE67kPGDA/1MbeqPfHzP0I1eJypf3o0pYxRsuU=";
private static final Logger LOG = Logger
.getLogger("opennlp.tools.similarity.apps.BingQueryRunner");
protected AzureSearchWebQuery aq = new AzureSearchWebQuery();
@@ -39,11 +45,32 @@
BING_KEY = key;
}
+ private int MAX_QUERY_LENGTH = 100;
+
public void setLang(String language){
aq.setMarket(language);
}
+ public List<HitBase> runSearchMultiplePages(String query, int nPages) {
+ List<HitBase> results = new ArrayList<HitBase>();
+ for(int i=0; i< nPages; i++){
+ aq.setPage(i);
+ results.addAll( runSearch(query, 50));
+ }
+ return results;
+ }
+
public List<HitBase> runSearch(String query, int nRes) {
+
+ if (query.length()>MAX_QUERY_LENGTH){
+ try {
+ query = query.substring(0, MAX_QUERY_LENGTH);
+ //should not cut words, need the last space to end the query
+ query = query.substring(0, StringUtils.lastIndexOf(query, " "));
+ } catch (Exception e) {
+ LOG.severe("Problem reducing the length of query :"+query);
+ }
+ }
aq.setAppid(BING_KEY);
aq.setQuery(query);
aq.setPerPage(nRes);
@@ -54,8 +81,12 @@
try {
aq.doQuery();
} catch (Exception e1) {
- // TODO Auto-generated catch block
- e1.printStackTrace();
+ aq.setAppid("Cec1TlE67kPGDA/1MbeqPfHzP0I1eJypf3o0pYxRsuU=");
+ try {
+ aq.doQuery();
+ } catch (Exception e2) {
+ e2.printStackTrace();
+ }
}
e.printStackTrace();
}
@@ -114,138 +145,12 @@
}
- /*
- private String constructBingUrl(String query, String domainWeb, String lang,
- int numbOfHits) throws Exception {
- String codedQuery = URLEncoder.encode(query, "UTF-8");
- String yahooRequest = "http://api.search.live.net/json.aspx?Appid="
- + APP_ID + "&query=" + codedQuery // +
- // "&sources=web"+
- + "&Sources=News"
- // Common request fields (optional)
- + "&Version=2.0" + "&Market=en-us"
- // + "&Options=EnableHighlighting"
-
- // News-specific request fields (optional)
- + "&News.Offset=0";
-
- return yahooRequest;
- }
-
-
-
- public ArrayList<String> search(String query, String domainWeb, String lang,
- int numbOfHits) throws Exception {
- URL url = new URL(constructBingUrl(query, domainWeb, lang, numbOfHits));
- URLConnection connection = url.openConnection();
-
- String line;
- ArrayList<String> result = new ArrayList<String>();
- BufferedReader reader = new BufferedReader(new InputStreamReader(
- connection.getInputStream()));
- int count = 0;
- while ((line = reader.readLine()) != null) {
- result.add(line);
- count++;
- }
- return result;
- }
-
- public BingResponse populateBingHit(String response) throws Exception {
- BingResponse resp = new BingResponse();
- JSONObject rootObject = new JSONObject(response);
- JSONObject responseObject = rootObject.getJSONObject("SearchResponse");
- JSONObject web = responseObject.getJSONObject("News");
-
- // the search result is in an array under the name of "results"
- JSONArray resultSet = null;
- try {
- resultSet = web.getJSONArray("Results");
- } catch (Exception e) {
- System.err.print("\n!!!!!!!");
- LOG.severe("\nNo search results");
-
- }
- if (resultSet != null) {
- for (int i = 0; i < resultSet.length(); i++) {
- HitBase hit = new HitBase();
- JSONObject singleResult = resultSet.getJSONObject(i);
- hit.setAbstractText(singleResult.getString("Snippet"));
- hit.setDate(singleResult.getString("Date"));
- String title = StringUtils.replace(singleResult.getString("Title"),
- "", " ");
- hit.setTitle(title);
- hit.setUrl(singleResult.getString("Url"));
- hit.setSource(singleResult.getString("Source"));
-
- resp.appendHits(hit);
- }
- }
- return resp;
- }
-
- public List<HitBase> runSearch(String query) {
- BingResponse resp = null;
- try {
- List<String> resultList = search(query, "", "", 8);
- resp = populateBingHit(resultList.get(0));
-
- } catch (Exception e) {
- // e.printStackTrace();
- LOG.severe("No news search results for query " + query);
- return null;
- }
- // cast to super class
- List<HitBase> hits = new ArrayList<HitBase>();
- for (HitBase h : resp.getHits())
- hits.add((HitBase) h);
-
- hits = HitBase.removeDuplicates(hits);
- return hits;
- }
- */
-
- // TODO comment back when dependencies resolved (CopyrightViolations)
- /*
- * public List<CopyrightViolations> runCopyRightViolExtenralSearch(String
- * query, String report) {
- *
- * List<CopyrightViolations> genResult = new ArrayList<CopyrightViolations>();
- * BingResponse newResp = null; StringDistanceMeasurer meas = new
- * StringDistanceMeasurer(); try { List<String> resultList = search(query, "",
- * "", 5);
- *
- * BingResponse resp = populateBingHit(resultList.get(0));
- * //printSearchResult(resultList.get(0));
- *
- * for(int i=0; i<resp.getHits().size(); i++){ BingHit h1 =
- * resp.getHits().get(i); String snippet = h1.getAbstractText(); Double sim =
- * meas.measureStringDistance(report, snippet); if
- * (sim>snapshotSimilarityThreshold){ //genResult.add(snapshot);
- * CopyrightViolations cvr = new CopyrightViolations();
- * cvr.setSnippet(snippet); cvr.setTitle(h1.getTitle());
- * cvr.setUrl(h1.getDisplayUrl()); genResult.add(cvr); log.debug(new
- * String("Copyright violation detected in snapshot"
- * ).toUpperCase()+" : sim = "+ new Double(sim).toString().substring(0, 3)+
- * " \n "+snippet);
- *
- * } else { log.debug("Different news: sim = "+ new
- * Double(sim).toString().substring(0, 3)+ " \n "+snippet);
- *
- * }
- *
- * }
- *
- * } catch (Exception e) { e.printStackTrace(); }
- *
- *
- * return genResult; }
- */
-
public static void main(String[] args) {
BingQueryRunner self = new BingQueryRunner();
+ List<HitBase> resp1 = self.runSearch("albert einstein", 15);
+ System.out.println(resp1);
AzureSearchResultSet<AzureSearchImageResult> res = self.runImageSearch("albert einstein");
System.out.println(res);
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingWebQueryRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingWebQueryRunner.java
index a934264..d28f4e3 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingWebQueryRunner.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingWebQueryRunner.java
@@ -17,25 +17,18 @@
package opennlp.tools.similarity.apps;
-import java.io.BufferedReader;
-import java.io.FileWriter;
-import java.io.IOException;
-import java.io.InputStreamReader;
-import java.net.URL;
-import java.net.URLConnection;
-import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Logger;
+
import net.billylieurance.azuresearch.AzureSearchResultSet;
import net.billylieurance.azuresearch.AzureSearchWebQuery;
import net.billylieurance.azuresearch.AzureSearchWebResult;
import opennlp.tools.similarity.apps.utils.StringDistanceMeasurer;
import org.apache.commons.lang.StringUtils;
-import org.json.JSONArray;
-import org.json.JSONObject;
+
public class BingWebQueryRunner {
@@ -111,6 +104,12 @@
return 0;
}
+
+ public static void main(String[] args) {
+ BingWebQueryRunner self = new BingWebQueryRunner();
+
+ List<HitBase> res = self.runSearch ("albert einstein", 10);
+ }
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/ContentGeneratorSupport.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/ContentGeneratorSupport.java
index 428cd4e..a017105 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/ContentGeneratorSupport.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/ContentGeneratorSupport.java
@@ -132,7 +132,7 @@
return queryArrayStr;
}
-
+
public static String[] cleanListOfSents(String[] sents) {
List<String> sentsClean = new ArrayList<String>();
for (String s : sents) {
@@ -144,11 +144,9 @@
}
public static String cleanSpacesInCleanedHTMLpage(String pageContent){ //was 4 spaces
- //was 3 spaces => now back to 2
+ //was 3 spaces => now back to 2
//TODO - verify regexp!!
pageContent = pageContent.trim().replaceAll("([a-z])(\\s{2,3})([A-Z])", "$1. $3")
- //replaceAll("[a-z] [A-Z]", ". $0")// .replace(" ",
- // ". ")
.replace("..", ".").replace(". . .", " ").
replace(". .",". ").trim(); // sometimes html breaks are converted into ' ' (two spaces), so
// we need to put '.'
@@ -461,7 +459,22 @@
}
return (String[]) sentsClean.toArray(new String[0]);
}
-
+
+ public static String getPortionOfTitleWithoutDelimiters(String title){
+ String[] delimiters = new String[]{"\\+","-", "=", "_", "\\)", "\\|"};
+ for(String delim: delimiters ){
+ String[] split = title.split(delim);
+ if (split.length>1){
+ for(String s: split){
+ if (s.indexOf(".")<0)
+ return s;
+ }
+ }
+ }
+
+ return title;
+ }
+
public static void main(String[] args){
String s = "You can grouP parts Of your regular expression In your pattern You grouP elements";
//with round brackets, e.g., ()." +
@@ -472,6 +485,15 @@
sr1 = s.replaceAll(" [A-Z]", ". $1");
}
+ public static boolean problematicHitList(List<HitBase> hits){
+ if (hits.size()<1)
+ return true;
+ for(HitBase hit: hits){
+ if (!hit.getFragments().isEmpty())
+ return false;
+ }
+ return true;
+ }
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GeneratedSentenceProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GeneratedSentenceProcessor.java
index 3e79b7a..17421fd 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GeneratedSentenceProcessor.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GeneratedSentenceProcessor.java
@@ -89,7 +89,7 @@
String[] periods = StringUtils.split(sent.replace('.', '#'), '#');
if ((float) periods.length / (float) spaces.length > 0.2) {
- System.out.println("Rejection: too many periods in sent ='"+sent);
+ //System.out.println("Rejection: too many periods in sent ='"+sent);
return null;
}
// commented [x], to avoid rejection sentences with refs[]
@@ -102,7 +102,7 @@
String[] pipes = StringUtils.split(sent, '|');
if (StringUtils.split(sent, '|').length > 2
|| StringUtils.split(sent, '>').length > 2) {
- System.out.println("Rejection: too many |s or >s in sent ='"+sent);
+ //System.out.println("Rejection: too many |s or >s in sent ='"+sent);
return null;
}
String sentTry = sent.toLowerCase();
@@ -200,14 +200,14 @@
public static boolean isProhibitiveWordsOccurOrStartWith(String sentenceLowercase){
for(String o: occurs){
if (sentenceLowercase.indexOf(o)>-1){
- System.out.println("Found prohibited occurrence "+ o +" \n in sentence = "+ sentenceLowercase);
+ //System.out.println("Found prohibited occurrence "+ o +" \n in sentence = "+ sentenceLowercase);
return true;
}
}
for(String o: occursStartsWith){
if (sentenceLowercase.startsWith(o)){
- System.out.println("Found prohibited occurrence Start With "+ o +" \n in sentence = "+ sentenceLowercase);
+ //System.out.println("Found prohibited occurrence Start With "+ o +" \n in sentence = "+ sentenceLowercase);
return true;
}
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/RelatedSentenceFinder.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/RelatedSentenceFinder.java
index bfeff62..91f6fda 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/RelatedSentenceFinder.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/RelatedSentenceFinder.java
@@ -82,6 +82,8 @@
this.RELEVANCE_THRESHOLD=thresh;
yrunner.setKey(key);
}
+
+ int generateContentAboutIter = 0;
public RelatedSentenceFinder() {
// TODO Auto-generated constructor stub
@@ -171,6 +173,20 @@
if (stepCount>MAX_STEPS)
break;
}
+
+ // if nothing is written, then get first search result and try again
+ try {
+ if (generateContentAboutIter<4 && ContentGeneratorSupport.problematicHitList(opinionSentencesToAdd)){
+ List<HitBase> resultList = yrunner.runSearch(sentence, 10);
+ String discoveredSimilarTopic = resultList.get(generateContentAboutIter).getTitle();
+ discoveredSimilarTopic = ContentGeneratorSupport.getPortionOfTitleWithoutDelimiters(discoveredSimilarTopic);
+ generateContentAboutIter++;
+ opinionSentencesToAdd = generateContentAbout(discoveredSimilarTopic);
+ }
+ } catch (Exception e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+ }
opinionSentencesToAdd = removeDuplicatesFromResultantHits(opinionSentencesToAdd);
return opinionSentencesToAdd;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/StoryDiscourseNavigator.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/StoryDiscourseNavigator.java
index b2d2194..1c50fbf 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/StoryDiscourseNavigator.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/StoryDiscourseNavigator.java
@@ -23,8 +23,11 @@
import java.util.HashSet;
import java.util.List;
+import org.apache.commons.lang.StringUtils;
+
+import opennlp.tools.similarity.apps.utils.PageFetcher;
import opennlp.tools.similarity.apps.utils.StringCleaner;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
import opennlp.tools.textsimilarity.ParseTreeChunk;
import opennlp.tools.textsimilarity.SentencePairMatchResult;
import opennlp.tools.textsimilarity.TextProcessor;
@@ -34,7 +37,8 @@
protected BingQueryRunner yrunner = new BingQueryRunner();
ParserChunker2MatcherProcessor sm = ParserChunker2MatcherProcessor
.getInstance();
- private PorterStemmer ps = new PorterStemmer();
+ private PStemmer ps = new PStemmer();
+ PageFetcher pFetcher = new PageFetcher();
public static final String[] frequentPerformingVerbs = {
" born raised meet learn ", " graduated enter discover",
@@ -53,8 +57,34 @@
"meet enjoy follow create", "discover continue produce"
};
+
+ private String[] obtainKeywordsForAnEntityFromWikipedia(String entity){
+ yrunner.setKey("xdnRVcVf9m4vDvW1SkTAz5kS5DFYa19CrPYGelGJxnc");
+ List<HitBase> resultList = yrunner.runSearch(entity, 20);
+ HitBase h = null;
+ for (int i = 0; i < resultList.size(); i++) {
+ h = resultList.get(i);
+ if (h.getUrl().indexOf("wikipedia.")>-1)
+ break;
+ }
+ String content = pFetcher.fetchOrigHTML(h.getUrl());
+ content = content.replace("\"><a href=\"#", "&_&_&_&");
+ String[] portions = StringUtils.substringsBetween(content, "&_&_&_&", "\"><span");
+ List<String> results = new ArrayList<String>();
+ for(int i = 0; i< portions.length; i++){
+ if (portions[i].indexOf("cite_note")>-1)
+ continue;
+ results.add(entity + " " + portions[i].replace('_', ' ').replace('.',' '));
+ }
+ return results.toArray(new String[0]);
+ }
public String[] obtainAdditionalKeywordsForAnEntity(String entity){
+ String[] keywordsFromWikipedia = obtainKeywordsForAnEntityFromWikipedia(entity);
+ // these keywords should include *entity*
+ if (keywordsFromWikipedia!=null && keywordsFromWikipedia.length>3)
+ return keywordsFromWikipedia;
+
List<List<ParseTreeChunk>> matchList = runSearchForTaxonomyPath(
entity, "", "en", 30);
Collection<String> keywordsToRemove = TextProcessor.fastTokenize(entity.toLowerCase(), false);
@@ -70,7 +100,7 @@
return res;
}
- public List<List<ParseTreeChunk>> runSearchForTaxonomyPath(String query,
+ private List<List<ParseTreeChunk>> runSearchForTaxonomyPath(String query,
String domain, String lang, int numbOfHits) {
List<List<ParseTreeChunk>> genResult = new ArrayList<List<ParseTreeChunk>>();
try {
@@ -127,5 +157,7 @@
public static void main(String[] args){
String[] res = new StoryDiscourseNavigator().obtainAdditionalKeywordsForAnEntity("Albert Einstein");
System.out.println(Arrays.asList(res));
+ res = new StoryDiscourseNavigator().obtainAdditionalKeywordsForAnEntity("search engine marketing");
+ System.out.println(Arrays.asList(res));
}
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/ContentGeneratorRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/ContentGeneratorRequestHandler.java
index 0e8d743..41afe36 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/ContentGeneratorRequestHandler.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/ContentGeneratorRequestHandler.java
@@ -1,7 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.similarity.apps.solr;
import java.io.BufferedReader;
import java.io.File;
+import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
@@ -61,6 +78,7 @@
private static Logger LOG = Logger
.getLogger("com.become.search.requestHandlers.SearchResultsReRankerRequestHandler");
private ParserChunker2MatcherProcessor sm = null;
+ WordDocBuilderEndNotes docBuilder = new WordDocBuilderEndNotes ();
public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp){
@@ -142,10 +160,9 @@
}
public String cgRunner(String[] args) {
- ParserChunker2MatcherProcessor sm = null;
int count=0;
for(String a: args){
- System.out.print(count+" >> " + a);
+ System.out.print(count+">>" + a + " | ");
count++;
}
@@ -164,13 +181,13 @@
String bingKey = args[7];
if (bingKey == null){
- bingKey = //"e8ADxIjn9YyHx36EihdjH/tMqJJItUrrbPTUpKahiU0=";
- "xdnRVcVf9m4vDvW1SkTAz5kS5DFYa19CrPYGelGJxnc";
+ bingKey = "e8ADxIjn9YyHx36EihdjH/tMqJJItUrrbPTUpKahiU0=";
+ //"xdnRVcVf9m4vDvW1SkTAz5kS5DFYa19CrPYGelGJxnc";
}
RelatedSentenceFinder f = null;
String lang = args[6];
- if (lang.startsWith("es")){
+ if (lang.startsWith("es") || lang.startsWith("ru") || lang.startsWith("de")){
f = new RelatedSentenceFinderML(Integer.parseInt(args[3]), Integer.parseInt(args[4]), Float.parseFloat(args[5]), bingKey);
f.setLang(lang);
} else
@@ -184,14 +201,28 @@
try {
hits = f.generateContentAbout(args[0].replace('+', ' ').replace('"', ' ').trim());
+
System.out.println(HitBase.toString(hits));
- generatedContent = HitBase.toResultantString(hits);
+ generatedContent = HitBase.toResultantString(hits) + "\n REFERENCES \n" + HitBase.produceReferenceSection(hits) ;
+ try {
+ writeResultInAFile(args[0].replace('+', ' '), generatedContent);
+ } catch (Exception e2) {
+ e2.printStackTrace();
+ }
+
+ String attachmentFileName = null;
+ try {
+ attachmentFileName = docBuilder.buildWordDoc(hits, args[0].replace('+', ' ').replace('"', ' '));
+ } catch (Exception e2) {
+ e2.printStackTrace();
+ }
+
opennlp.tools.apps.utils.email.EmailSender s = new opennlp.tools.apps.utils.email.EmailSender();
try {
s.sendMail("smtp.rambler.ru", "bg7550@rambler.ru", "pill0693", new InternetAddress("bg7550@rambler.ru"), new InternetAddress[]{new InternetAddress(args[1])}, new InternetAddress[]{}, new InternetAddress[]{},
- "Generated content for you on '"+args[0].replace('+', ' ')+"'", generatedContent, null);
+ "Generated content for you on '"+args[0].replace('+', ' ')+"'", generatedContent, attachmentFileName);
} catch (AddressException e) {
// TODO Auto-generated catch block
e.printStackTrace();
@@ -200,7 +231,7 @@
e.printStackTrace();
try {
s.sendMail("smtp.rambler.ru", "bg7550@rambler.ru", "pill0693", new InternetAddress("bg7550@rambler.ru"), new InternetAddress[]{new InternetAddress(args[1])}, new InternetAddress[]{}, new InternetAddress[]{},
- "Generated content for you on '"+args[0].replace('+', ' ')+"'", generatedContent, null);
+ "Generated content for you on '"+args[0].replace('+', ' ')+"'", generatedContent, attachmentFileName);
} catch (Exception e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
@@ -214,6 +245,40 @@
return generatedContent;
}
+ private void writeResultInAFile(String title, String content){
+ FileOutputStream fop = null;
+ File file;
+ String absPath = new File(".").getAbsolutePath();
+ absPath = absPath.substring(0, absPath.length()-1);
+
+ try {
+
+ file = new File(absPath+"/written/"+ title.replace(' ','_').replace('\"', ' ').trim()+ ".txt");
+ // if file doesnt exists, then create it
+ if (!file.exists()) {
+ file.createNewFile();
+ }
+ fop = new FileOutputStream(file);
+
+ // get the content in bytes
+ byte[] contentInBytes = content.getBytes();
+
+ fop.write(contentInBytes);
+ fop.flush();
+ fop.close();
+
+ } catch (IOException e) {
+ e.printStackTrace();
+ } finally {
+ try {
+ if (fop != null) {
+ fop.close();
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+ }
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeQueryComponent.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeQueryComponent.java
index 14dc9ff..6693bbf 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeQueryComponent.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeQueryComponent.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.similarity.apps.solr;
import java.io.IOException;
@@ -139,13 +155,13 @@
e.printStackTrace();
}
rb.setQparser(parser);
- try {
+ /* try {
rb.setScoreDoc(parser.getPaging());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
-
+*/
String[] fqs = rb.req.getParams().getParams(CommonParams.FQ);
if (fqs!=null && fqs.length!=0) {
List<Query> filters = rb.getFilters();
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeSearchRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeSearchRequestHandler.java
index 87f5ed9..be125b7 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeSearchRequestHandler.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/IterativeSearchRequestHandler.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.similarity.apps.solr;
import java.io.IOException;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/NLProgram2CodeRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/NLProgram2CodeRequestHandler.java
index 0876700..413dd5d 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/NLProgram2CodeRequestHandler.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/NLProgram2CodeRequestHandler.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.similarity.apps.solr;
import java.io.IOException;
@@ -58,8 +74,8 @@
private ParseTreeChunkListScorer parseTreeChunkListScorer = new ParseTreeChunkListScorer();
private ParserChunker2MatcherProcessor sm = null;
private int MAX_QUERY_LENGTH_NOT_TO_RERANK=3;
- private static String resourceDir = "/home/solr/solr-4.4.0/example/src/test/resources";
- //"C:/workspace/TestSolr/src/test/resources";
+ private static String resourceDir = //"/home/solr/solr-4.4.0/example/src/test/resources";
+ "C:/workspace/TestSolr/src/test/resources";
//"/data1/solr/example/src/test/resources";
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerRequestHandler.java
index fbef398..b259528 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerRequestHandler.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerRequestHandler.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.similarity.apps.solr;
import java.io.IOException;
@@ -164,7 +180,7 @@
NamedList<Object> values = rsp.getValues();
values.remove("response");
values.add("response", scoreNum);
- values.add("new_order", bufNums.toString().trim());
+ //values.add("new_order", bufNums.toString().trim());
rsp.setAllValues(values);
}
@@ -187,9 +203,7 @@
private List<HitBase> calculateMatchScoreResortHits(List<HitBase> hits,
String searchQuery) {
try {
- System.out.println("loading openNLP models...from "+resourceDir);
sm = ParserChunker2MatcherProcessor.getInstance(resourceDir);
- System.out.println("DONE loading openNLP model s.");
} catch (Exception e){
LOG.severe(e.getMessage());
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SyntGenRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SyntGenRequestHandler.java
index b2d6295..d2f4b1b 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SyntGenRequestHandler.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SyntGenRequestHandler.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package opennlp.tools.similarity.apps.solr;
import java.io.IOException;
@@ -56,7 +72,6 @@
import org.apache.solr.search.DocSlice;
import org.apache.solr.search.QParser;
import org.apache.solr.search.SolrIndexSearcher;
-
import org.apache.solr.util.RTimer;
import org.apache.solr.util.SolrPluginUtils;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomyExtenderViaMebMining.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomyExtenderViaMebMining.java
index 84440bd..59f2146 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomyExtenderViaMebMining.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomyExtenderViaMebMining.java
@@ -26,7 +26,7 @@
import opennlp.tools.similarity.apps.BingQueryRunner;
import opennlp.tools.similarity.apps.HitBase;
import opennlp.tools.similarity.apps.utils.StringCleaner;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
import opennlp.tools.textsimilarity.ParseTreeChunk;
import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
import opennlp.tools.textsimilarity.SentencePairMatchResult;
@@ -51,7 +51,7 @@
private Map<String, List<List<String>>> lemma_ExtendedAssocWords = new HashMap<String, List<List<String>>>();
private Map<List<String>, List<List<String>>> assocWords_ExtendedAssocWords = new HashMap<List<String>, List<List<String>>>();
- private PorterStemmer ps;
+ private PStemmer ps;
public Map<List<String>, List<List<String>>> getAssocWords_ExtendedAssocWords() {
return assocWords_ExtendedAssocWords;
@@ -73,7 +73,7 @@
System.err.println("Problem loading synt matcher");
}
- ps = new PorterStemmer();
+ ps = new PStemmer();
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomySerializer.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomySerializer.java
index 16e9fb2..a70340e 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomySerializer.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/TaxonomySerializer.java
@@ -22,10 +22,13 @@
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+
/**
* This class stores the taxonomy on the file-system
*
@@ -80,6 +83,31 @@
ex.printStackTrace();
}
+ String csvFilename = filename+".csv";
+ List<String[]> taxo_list = new ArrayList<String[]>();
+ List<String> entries = new ArrayList<String>(lemma_ExtendedAssocWords.keySet());
+ for(String e: entries){
+ List<String> lines = new ArrayList<String>();
+ lines.add(e);
+ for(List<String> ls: lemma_ExtendedAssocWords.get(e)){
+ lines.add(ls.toString());
+ }
+ taxo_list.add((String[])lines.toArray(new String[0]));
+ }
+ ProfileReaderWriter.writeReport(taxo_list, csvFilename);
+
+ String csvFilenameListEntries = filename+"_ListEntries.csv";
+ taxo_list = new ArrayList<String[]>();
+ List<List<String>> entriesList = new ArrayList<List<String>>( assocWords_ExtendedAssocWords.keySet());
+ for(List<String> e: entriesList){
+ List<String> lines = new ArrayList<String>();
+ lines.addAll(e);
+ for(List<String> ls: assocWords_ExtendedAssocWords.get(e)){
+ lines.add(ls.toString());
+ }
+ taxo_list.add((String[])lines.toArray(new String[0]));
+ }
+ ProfileReaderWriter.writeReport(taxo_list, csvFilenameListEntries);
}
public static TaxonomySerializer readTaxonomy(String filename) {
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/PageFetcher.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/PageFetcher.java
index 4c01e39..7f17f84 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/PageFetcher.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/PageFetcher.java
@@ -18,6 +18,7 @@
package opennlp.tools.similarity.apps.utils;
import java.io.BufferedReader;
+import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
@@ -27,54 +28,94 @@
import org.apache.tika.Tika;
import org.apache.tika.exception.TikaException;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.Parser;
+import org.apache.tika.sax.BodyContentHandler;
+
public class PageFetcher {
- private static final Logger LOG = Logger
+ private static final Logger log = Logger
.getLogger("opennlp.tools.similarity.apps.utils.PageFetcher");
+ Tika tika = new Tika();
- private static int DEFAULT_TIMEOUT = 15000;
+ private static int DEFAULT_TIMEOUT = 1500;
+ private void setTimeout(int to){
+ DEFAULT_TIMEOUT = to;
+ }
public String fetchPage(final String url) {
return fetchPage(url, DEFAULT_TIMEOUT);
}
+
+ public String fetchPageAutoDetectParser(final String url ){
+ String fetchURL = addHttp(url);
+ String pageContent = null;
+ URLConnection connection;
+ try {
+ log.info("fetch url auto detect parser " + url);
+ connection = new URL(fetchURL).openConnection();
+ connection.setReadTimeout(DEFAULT_TIMEOUT);
+
+ //parse method parameters
+ Parser parser = new AutoDetectParser();
+ BodyContentHandler handler = new BodyContentHandler();
+ Metadata metadata = new Metadata();
+ ParseContext context = new ParseContext();
+
+ //parsing the file
+ parser.parse(connection.getInputStream(), handler, metadata, context);
+
+ pageContent = handler.toString();
+ } catch (Exception e) {
+ log.info(e.getMessage() + "\n" + e);
+ }
+ return pageContent;
+ }
+
public String fetchPage(final String url, final int timeout) {
String fetchURL = addHttp(url);
- LOG.info("fetch url " + fetchURL);
+ log.info("fetch url " + fetchURL);
String pageContent = null;
URLConnection connection;
try {
- connection = new URL(url).openConnection();
+ connection = new URL(fetchURL).openConnection();
connection.setReadTimeout(DEFAULT_TIMEOUT);
- Tika tika = new Tika();
+
pageContent = tika.parseToString(connection.getInputStream())
.replace('\n', ' ').replace('\t', ' ');
} catch (MalformedURLException e) {
- LOG.severe(e.getMessage() + "\n" + e);
+ log.severe(e.getMessage() + "\n" + e);
} catch (IOException e) {
- LOG.severe(e.getMessage() + "\n" + e);
+ log.severe(e.getMessage() + "\n" + e);
} catch (TikaException e) {
- LOG.severe(e.getMessage() + "\n" + e);
+ log.severe(e.getMessage() + "\n" + e);
}
return pageContent;
}
private String addHttp(final String url) {
- if (!url.startsWith("http://")) {
+ if (!url.startsWith("http://") && !url.startsWith("https://")) {
return "http://" + url;
}
return url;
}
+
+ public String fetchOrigHTML(String url, int timeout) {
+ setTimeout(timeout);
+ return fetchOrigHTML(url);
+ }
public String fetchOrigHTML(String url) {
- System.out.println("fetch url " + url);
- String pageContent = null;
+ log.info("fetch url " + url);
StringBuffer buf = new StringBuffer();
try {
URLConnection connection = new URL(url).openConnection();
- connection.setReadTimeout(10000);
+ connection.setReadTimeout(DEFAULT_TIMEOUT);
connection
.setRequestProperty(
"User-Agent",
@@ -85,8 +126,8 @@
reader = new BufferedReader(new InputStreamReader(
connection.getInputStream()));
} catch (Exception e) {
- // we dont need to log trial web pages if access fails
- // LOG.error(e.getMessage(), e);
+ // we dont always need to log trial web pages if access fails
+ log.severe(e.toString());
}
while ((line = reader.readLine()) != null) {
@@ -107,5 +148,19 @@
} */
return buf.toString();
}
+
+ public static void main(String[] args){
+ PageFetcher fetcher = new PageFetcher();
+ String content = fetcher.fetchPageAutoDetectParser("http://www.elastica.net/");
+ System.out.println(content);
+ content = fetcher.
+ fetchPageAutoDetectParser("http://www.cnn.com");
+ System.out.println(content);
+ content = new PageFetcher().fetchPage("https://github.com");
+ System.out.println(content);
+ content = new PageFetcher().fetchOrigHTML("http://www.cnn.com");
+ System.out.println(content);
+
+ }
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/StringDistanceMeasurer.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/StringDistanceMeasurer.java
index c2238c5..377b02a 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/StringDistanceMeasurer.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/utils/StringDistanceMeasurer.java
@@ -20,11 +20,11 @@
import java.util.ArrayList;
import java.util.List;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
public class StringDistanceMeasurer {
// external tools
- private PorterStemmer ps; // stemmer
+ private PStemmer ps; // stemmer
private static final int MIN_STRING_LENGTH_FOR_WORD = 4;
@@ -36,7 +36,7 @@
public StringDistanceMeasurer() {
// first get stemmer
- ps = new PorterStemmer();
+ ps = new PStemmer();
if (MIN_SCORE_FOR_LING > 1.0)
return;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PorterStemmer.java b/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PorterStemmer.java
deleted file mode 100644
index e23da90..0000000
--- a/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PorterStemmer.java
+++ /dev/null
@@ -1,521 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.stemmer;
-
-
- import java.io.IOException;
- import java.io.InputStream;
- import java.io.FileInputStream;
-
- import static org.apache.lucene.util.RamUsageEstimator.NUM_BYTES_CHAR;
- import org.apache.lucene.util.ArrayUtil;
-
- /**
- *
- * Stemmer, implementing the Porter Stemming Algorithm
- *
- * The Stemmer class transforms a word into its root form. The input
- * word can be provided a character at time (by calling add()), or at once
- * by calling one of the various stem(something) methods.
- */
-
- public class PorterStemmer
- {
- private char[] b;
- private int i, /* offset into b */
- j, k, k0;
- private boolean dirty = false;
- private static final int INITIAL_SIZE = 50;
-
- public PorterStemmer() {
- b = new char[INITIAL_SIZE];
- i = 0;
- }
-
- /**
- * reset() resets the stemmer so it can stem another word. If you invoke
- * the stemmer by calling add(char) and then stem(), you must call reset()
- * before starting another word.
- */
- public void reset() { i = 0; dirty = false; }
-
- /**
- * Add a character to the word being stemmed. When you are finished
- * adding characters, you can call stem(void) to process the word.
- */
- public void add(char ch) {
- if (b.length <= i) {
- b = ArrayUtil.grow(b, i+1);
- }
- b[i++] = ch;
- }
-
- /**
- * After a word has been stemmed, it can be retrieved by toString(),
- * or a reference to the internal buffer can be retrieved by getResultBuffer
- * and getResultLength (which is generally more efficient.)
- */
- @Override
- public String toString() { return new String(b,0,i); }
-
- /**
- * Returns the length of the word resulting from the stemming process.
- */
- public int getResultLength() { return i; }
-
- /**
- * Returns a reference to a character buffer containing the results of
- * the stemming process. You also need to consult getResultLength()
- * to determine the length of the result.
- */
- public char[] getResultBuffer() { return b; }
-
- /* cons(i) is true <=> b[i] is a consonant. */
-
- private final boolean cons(int i) {
- switch (b[i]) {
- case 'a': case 'e': case 'i': case 'o': case 'u':
- return false;
- case 'y':
- return (i==k0) ? true : !cons(i-1);
- default:
- return true;
- }
- }
-
- /* m() measures the number of consonant sequences between k0 and j. if c is
- a consonant sequence and v a vowel sequence, and <..> indicates arbitrary
- presence,
-
- <c><v> gives 0
- <c>vc<v> gives 1
- <c>vcvc<v> gives 2
- <c>vcvcvc<v> gives 3
- ....
- */
-
- private final int m() {
- int n = 0;
- int i = k0;
- while(true) {
- if (i > j)
- return n;
- if (! cons(i))
- break;
- i++;
- }
- i++;
- while(true) {
- while(true) {
- if (i > j)
- return n;
- if (cons(i))
- break;
- i++;
- }
- i++;
- n++;
- while(true) {
- if (i > j)
- return n;
- if (! cons(i))
- break;
- i++;
- }
- i++;
- }
- }
-
- /* vowelinstem() is true <=> k0,...j contains a vowel */
-
- private final boolean vowelinstem() {
- int i;
- for (i = k0; i <= j; i++)
- if (! cons(i))
- return true;
- return false;
- }
-
- /* doublec(j) is true <=> j,(j-1) contain a double consonant. */
-
- private final boolean doublec(int j) {
- if (j < k0+1)
- return false;
- if (b[j] != b[j-1])
- return false;
- return cons(j);
- }
-
- /* cvc(i) is true <=> i-2,i-1,i has the form consonant - vowel - consonant
- and also if the second c is not w,x or y. this is used when trying to
- restore an e at the end of a short word. e.g.
-
- cav(e), lov(e), hop(e), crim(e), but
- snow, box, tray.
-
- */
-
- private final boolean cvc(int i) {
- if (i < k0+2 || !cons(i) || cons(i-1) || !cons(i-2))
- return false;
- else {
- int ch = b[i];
- if (ch == 'w' || ch == 'x' || ch == 'y') return false;
- }
- return true;
- }
-
- private final boolean ends(String s) {
- int l = s.length();
- int o = k-l+1;
- if (o < k0)
- return false;
- for (int i = 0; i < l; i++)
- if (b[o+i] != s.charAt(i))
- return false;
- j = k-l;
- return true;
- }
-
- /* setto(s) sets (j+1),...k to the characters in the string s, readjusting
- k. */
-
- void setto(String s) {
- int l = s.length();
- int o = j+1;
- for (int i = 0; i < l; i++)
- b[o+i] = s.charAt(i);
- k = j+l;
- dirty = true;
- }
-
- /* r(s) is used further down. */
-
- void r(String s) { if (m() > 0) setto(s); }
-
- /* step1() gets rid of plurals and -ed or -ing. e.g.
-
- caresses -> caress
- ponies -> poni
- ties -> ti
- caress -> caress
- cats -> cat
-
- feed -> feed
- agreed -> agree
- disabled -> disable
-
- matting -> mat
- mating -> mate
- meeting -> meet
- milling -> mill
- messing -> mess
-
- meetings -> meet
-
- */
-
- private final void step1() {
- if (b[k] == 's') {
- if (ends("sses")) k -= 2;
- else if (ends("ies")) setto("i");
- else if (b[k-1] != 's') k--;
- }
- if (ends("eed")) {
- if (m() > 0)
- k--;
- }
- else if ((ends("ed") || ends("ing")) && vowelinstem()) {
- k = j;
- if (ends("at")) setto("ate");
- else if (ends("bl")) setto("ble");
- else if (ends("iz")) setto("ize");
- else if (doublec(k)) {
- int ch = b[k--];
- if (ch == 'l' || ch == 's' || ch == 'z')
- k++;
- }
- else if (m() == 1 && cvc(k))
- setto("e");
- }
- }
-
- /* step2() turns terminal y to i when there is another vowel in the stem. */
-
- private final void step2() {
- if (ends("y") && vowelinstem()) {
- b[k] = 'i';
- dirty = true;
- }
- }
-
- /* step3() maps double suffices to single ones. so -ization ( = -ize plus
- -ation) maps to -ize etc. note that the string before the suffix must give
- m() > 0. */
-
- private final void step3() {
- if (k == k0) return; /* For Bug 1 */
- switch (b[k-1]) {
- case 'a':
- if (ends("ational")) { r("ate"); break; }
- if (ends("tional")) { r("tion"); break; }
- break;
- case 'c':
- if (ends("enci")) { r("ence"); break; }
- if (ends("anci")) { r("ance"); break; }
- break;
- case 'e':
- if (ends("izer")) { r("ize"); break; }
- break;
- case 'l':
- if (ends("bli")) { r("ble"); break; }
- if (ends("alli")) { r("al"); break; }
- if (ends("entli")) { r("ent"); break; }
- if (ends("eli")) { r("e"); break; }
- if (ends("ousli")) { r("ous"); break; }
- break;
- case 'o':
- if (ends("ization")) { r("ize"); break; }
- if (ends("ation")) { r("ate"); break; }
- if (ends("ator")) { r("ate"); break; }
- break;
- case 's':
- if (ends("alism")) { r("al"); break; }
- if (ends("iveness")) { r("ive"); break; }
- if (ends("fulness")) { r("ful"); break; }
- if (ends("ousness")) { r("ous"); break; }
- break;
- case 't':
- if (ends("aliti")) { r("al"); break; }
- if (ends("iviti")) { r("ive"); break; }
- if (ends("biliti")) { r("ble"); break; }
- break;
- case 'g':
- if (ends("logi")) { r("log"); break; }
- }
- }
-
- /* step4() deals with -ic-, -full, -ness etc. similar strategy to step3. */
-
- private final void step4() {
- switch (b[k]) {
- case 'e':
- if (ends("icate")) { r("ic"); break; }
- if (ends("ative")) { r(""); break; }
- if (ends("alize")) { r("al"); break; }
- break;
- case 'i':
- if (ends("iciti")) { r("ic"); break; }
- break;
- case 'l':
- if (ends("ical")) { r("ic"); break; }
- if (ends("ful")) { r(""); break; }
- break;
- case 's':
- if (ends("ness")) { r(""); break; }
- break;
- }
- }
-
- /* step5() takes off -ant, -ence etc., in context <c>vcvc<v>. */
-
- private final void step5() {
- if (k == k0) return; /* for Bug 1 */
- switch (b[k-1]) {
- case 'a':
- if (ends("al")) break;
- return;
- case 'c':
- if (ends("ance")) break;
- if (ends("ence")) break;
- return;
- case 'e':
- if (ends("er")) break; return;
- case 'i':
- if (ends("ic")) break; return;
- case 'l':
- if (ends("able")) break;
- if (ends("ible")) break; return;
- case 'n':
- if (ends("ant")) break;
- if (ends("ement")) break;
- if (ends("ment")) break;
- /* element etc. not stripped before the m */
- if (ends("ent")) break;
- return;
- case 'o':
- if (ends("ion") && j >= 0 && (b[j] == 's' || b[j] == 't')) break;
- /* j >= 0 fixes Bug 2 */
- if (ends("ou")) break;
- return;
- /* takes care of -ous */
- case 's':
- if (ends("ism")) break;
- return;
- case 't':
- if (ends("ate")) break;
- if (ends("iti")) break;
- return;
- case 'u':
- if (ends("ous")) break;
- return;
- case 'v':
- if (ends("ive")) break;
- return;
- case 'z':
- if (ends("ize")) break;
- return;
- default:
- return;
- }
- if (m() > 1)
- k = j;
- }
-
- /* step6() removes a final -e if m() > 1. */
-
- private final void step6() {
- j = k;
- if (b[k] == 'e') {
- int a = m();
- if (a > 1 || a == 1 && !cvc(k-1))
- k--;
- }
- if (b[k] == 'l' && doublec(k) && m() > 1)
- k--;
- }
-
-
- /**
- * Stem a word provided as a String. Returns the result as a String.
- */
- public String stem(String s) {
- if (stem(s.toCharArray(), s.length()))
- return toString();
- else
- return s;
- }
-
- /** Stem a word contained in a char[]. Returns true if the stemming process
- * resulted in a word different from the input. You can retrieve the
- * result with getResultLength()/getResultBuffer() or toString().
- */
- public boolean stem(char[] word) {
- return stem(word, word.length);
- }
-
- /** Stem a word contained in a portion of a char[] array. Returns
- * true if the stemming process resulted in a word different from
- * the input. You can retrieve the result with
- * getResultLength()/getResultBuffer() or toString().
- */
- public boolean stem(char[] wordBuffer, int offset, int wordLen) {
- reset();
- if (b.length < wordLen) {
- b = new char[ArrayUtil.oversize(wordLen, NUM_BYTES_CHAR)];
- }
- System.arraycopy(wordBuffer, offset, b, 0, wordLen);
- i = wordLen;
- return stem(0);
- }
-
- /** Stem a word contained in a leading portion of a char[] array.
- * Returns true if the stemming process resulted in a word different
- * from the input. You can retrieve the result with
- * getResultLength()/getResultBuffer() or toString().
- */
- public boolean stem(char[] word, int wordLen) {
- return stem(word, 0, wordLen);
- }
-
- /** Stem the word placed into the Stemmer buffer through calls to add().
- * Returns true if the stemming process resulted in a word different
- * from the input. You can retrieve the result with
- * getResultLength()/getResultBuffer() or toString().
- */
- public boolean stem() {
- return stem(0);
- }
-
- public boolean stem(int i0) {
- k = i - 1;
- k0 = i0;
- if (k > k0+1) {
- step1(); step2(); step3(); step4(); step5(); step6();
- }
- // Also, a word is considered dirty if we lopped off letters
- // Thanks to Ifigenia Vairelles for pointing this out.
- if (i != k+1)
- dirty = true;
- i = k+1;
- return dirty;
- }
-
- /** Test program for demonstrating the Stemmer. It reads a file and
- * stems each word, writing the result to standard out.
- * Usage: Stemmer file-name
- */
- public static void main(String[] args) {
- PorterStemmer s = new PorterStemmer();
-
- for (int i = 0; i < args.length; i++) {
- try {
- InputStream in = new FileInputStream(args[i]);
- byte[] buffer = new byte[1024];
- int bufferLen, offset, ch;
-
- bufferLen = in.read(buffer);
- offset = 0;
- s.reset();
-
- while(true) {
- if (offset < bufferLen)
- ch = buffer[offset++];
- else {
- bufferLen = in.read(buffer);
- offset = 0;
- if (bufferLen < 0)
- ch = -1;
- else
- ch = buffer[offset++];
- }
-
- if (Character.isLetter((char) ch)) {
- s.add(Character.toLowerCase((char) ch));
- }
- else {
- s.stem();
- System.out.print(s.toString());
- s.reset();
- if (ch < 0)
- break;
- else {
- System.out.print((char) ch);
- }
- }
- }
-
- in.close();
- }
- catch (IOException e) {
- System.out.println("error reading " + args[i]);
- }
- }
- }
- }
-
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/LemmaFormManager.java b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/LemmaFormManager.java
index 1dc100c..a72583e 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/LemmaFormManager.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/LemmaFormManager.java
@@ -19,11 +19,11 @@
import java.util.List;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
public class LemmaFormManager {
- public String matchLemmas(PorterStemmer ps, String lemma1, String lemma2,
+ public String matchLemmas(PStemmer ps, String lemma1, String lemma2,
String POS) {
if (POS == null) {
return null;
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunk.java b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunk.java
index 74c685c..f151768 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunk.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunk.java
@@ -18,396 +18,551 @@
package opennlp.tools.textsimilarity;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.collections.ListUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import opennlp.tools.parse_thicket.ParseTreeNode;
public class ParseTreeChunk {
- private String mainPOS;
+ private String mainPOS;
- private List<String> lemmas;
+ private List<String> lemmas;
- private List<String> POSs;
+ private List<String> POSs;
- private int startPos;
+ private int startPos;
- private int endPos;
+ private int endPos;
- private int size;
+ private int size;
- private ParseTreeMatcher parseTreeMatcher;
+ private ParseTreeMatcher parseTreeMatcher;
- private LemmaFormManager lemmaFormManager;
+ private LemmaFormManager lemmaFormManager;
- private GeneralizationListReducer generalizationListReducer;
+ private GeneralizationListReducer generalizationListReducer;
- public ParseTreeChunk() {
- }
+ private List<ParseTreeNode> parseTreeNodes;
- public ParseTreeChunk(List<String> lemmas, List<String> POSs, int startPos,
- int endPos) {
- this.lemmas = lemmas;
- this.POSs = POSs;
- this.startPos = startPos;
- this.endPos = endPos;
- // phraseType.put(0, "np");
- }
+ public List<ParseTreeNode> getParseTreeNodes() {
+ return parseTreeNodes;
+ }
- // constructor which takes lemmas and POS as lists so that phrases can be
- // conveniently specified.
- // usage: stand-alone runs
- public ParseTreeChunk(String mPOS, String[] lemmas, String[] POSss) {
- this.mainPOS = mPOS;
- this.lemmas = new ArrayList<String>();
- for (String l : lemmas) {
- this.lemmas.add(l);
- }
- this.POSs = new ArrayList<String>();
- for (String p : POSss) {
- this.POSs.add(p);
- }
- }
+ public void setParseTreeNodes(List<ParseTreeNode> parseTreeNodes) {
+ this.parseTreeNodes = parseTreeNodes;
+ }
- // constructor which takes lemmas and POS as lists so that phrases can be
- // conveniently specified.
- // usage: stand-alone runs
- public ParseTreeChunk(String mPOS, List<String> lemmas, List<String> POSss) {
- this.mainPOS = mPOS;
- this.lemmas = lemmas;
- this.POSs = POSss;
+ public ParseTreeChunk(){};
+ // "[<1>NP'Property':NN, <2>NP'has':VBZ, <3>NP'lots':NNS, <4>NP'of':IN, <5>NP'trash':NN, <6>NP'and':CC, <7>NP'debris':NN]";
- }
+ public ParseTreeChunk(String phrStr){
+ String[] parts = phrStr.replace("]","").split(", <");
+ this.POSs = new ArrayList<String>();
+ this.lemmas = new ArrayList<String>();
+ this.mainPOS = StringUtils.substringBetween(phrStr, ">", "'");
+ for(String part: parts){
+ String lemma = StringUtils.substringBetween(part, "P'", "':");
+ String pos = part.substring(part.indexOf(":")+1, part.length());
+
+ if (pos==null || lemma ==null){
+ continue;
+ }
+ this.POSs.add(pos.trim());
+ this.lemmas.add(lemma.trim());
+ }
+
+ }
+
+ public ParseTreeChunk(List<String> lemmas, List<String> POSs, int startPos,
+ int endPos) {
+ this.lemmas = lemmas;
+ this.POSs = POSs;
+ this.startPos = startPos;
+ this.endPos = endPos;
- // Before:
- // [0(S-At home we like to eat great pizza deals), 0(PP-At home), 0(IN-At),
- // 3(NP-home), 3(NN-home), 8(NP-we),
- // 8(PRP-we), 11(VP-like to eat great pizza deals), 11(VBP-like), 16(S-to eat
- // great pizza deals), 16(VP-to eat great
- // pizza deals),
- // 16(TO-to), 19(VP-eat great pizza deals), 19(VB-eat), 23(NP-great pizza
- // deals), 23(JJ-great), 29(NN-pizza),
- // 35(NNS-deals)]
+ // phraseType.put(0, "np");
+ }
- // After:
- // [S [IN-At NP-home NP-we VBP-like ], PP [IN-At NP-home ], IN [IN-At ], NP
- // [NP-home ], NN [NP-home ], NP [NP-we ],
- // PRP [NP-we ], VP [VBP-like TO-to VB-eat JJ-great ], VBP [VBP-like ], S
- // [TO-to VB-eat JJ-great NN-pizza ], VP
- // [TO-to VB-eat JJ-great NN-pizza ], TO [TO-to ], VP [VB-eat JJ-great
- // NN-pizza NNS-deals ],
- // VB [VB-eat ], NP [JJ-great NN-pizza NNS-deals ], JJ [JJ-great ], NN
- // [NN-pizza ], NNS [NNS-deals ]]
+ // constructor which takes lemmas and POS as lists so that phrases can be
+ // conveniently specified.
+ // usage: stand-alone runs
+ public ParseTreeChunk(String mPOS, String[] lemmas, String[] POSss) {
+ this.mainPOS = mPOS;
+ this.lemmas = new ArrayList<String>();
+ for (String l : lemmas) {
+ this.lemmas.add(l);
+ }
+ this.POSs = new ArrayList<String>();
+ for (String p : POSss) {
+ this.POSs.add(p);
+ }
+ }
- public List<ParseTreeChunk> buildChunks(List<LemmaPair> parseResults) {
- List<ParseTreeChunk> chunksResults = new ArrayList<ParseTreeChunk>();
- for (LemmaPair chunk : parseResults) {
- String[] lemmasAr = chunk.getLemma().split(" ");
- List<String> poss = new ArrayList<String>(), lems = new ArrayList<String>();
- for (String lem : lemmasAr) {
- lems.add(lem);
- // now looking for POSs for individual word
- for (LemmaPair chunkCur : parseResults) {
- if (chunkCur.getLemma().equals(lem)
- &&
- // check that this is a proper word in proper position
- chunkCur.getEndPos() <= chunk.getEndPos()
- && chunkCur.getStartPos() >= chunk.getStartPos()) {
- poss.add(chunkCur.getPOS());
- break;
- }
- }
- }
- if (lems.size() != poss.size()) {
- System.err.println("lems.size()!= poss.size()");
- }
- if (lems.size() < 2) { // single word phrase, nothing to match
- continue;
- }
- ParseTreeChunk ch = new ParseTreeChunk(lems, poss, chunk.getStartPos(),
- chunk.getEndPos());
- ch.setMainPOS(chunk.getPOS());
- chunksResults.add(ch);
- }
- return chunksResults;
- }
+ // constructor which takes lemmas and POS as lists so that phrases can be
+ // conveniently specified.
+ // usage: stand-alone runs
+ public ParseTreeChunk(String mPOS, List<String> lemmas, List<String> POSss) {
+ this.mainPOS = mPOS;
+ this.lemmas = lemmas;
+ this.POSs = POSss;
+ }
- public List<List<ParseTreeChunk>> matchTwoSentencesGivenPairLists(
- List<LemmaPair> sent1Pairs, List<LemmaPair> sent2Pairs) {
- List<ParseTreeChunk> chunk1List = buildChunks(sent1Pairs);
- List<ParseTreeChunk> chunk2List = buildChunks(sent2Pairs);
+ public int getStartPos() {
+ return startPos;
+ }
- List<List<ParseTreeChunk>> sent1GrpLst = groupChunksAsParses(chunk1List);
- List<List<ParseTreeChunk>> sent2GrpLst = groupChunksAsParses(chunk2List);
+ public void setStartPos(int startPos) {
+ this.startPos = startPos;
+ }
- System.out.println("=== Grouped chunks 1 " + sent1GrpLst);
- System.out.println("=== Grouped chunks 2 " + sent2GrpLst);
+ public int getEndPos() {
+ return endPos;
+ }
- return matchTwoSentencesGroupedChunks(sent1GrpLst, sent2GrpLst);
- }
+ public void setEndPos(int endPos) {
+ this.endPos = endPos;
+ }
- // groups noun phrases, verb phrases, propos phrases etc. for separate match
+ public int getSize() {
+ return size;
+ }
- public List<List<ParseTreeChunk>> groupChunksAsParses(
- List<ParseTreeChunk> parseResults) {
- List<ParseTreeChunk> np = new ArrayList<ParseTreeChunk>(), vp = new ArrayList<ParseTreeChunk>(), prp = new ArrayList<ParseTreeChunk>(), sbarp = new ArrayList<ParseTreeChunk>(), pp = new ArrayList<ParseTreeChunk>(), adjp = new ArrayList<ParseTreeChunk>(), whadvp = new ArrayList<ParseTreeChunk>(), restOfPhrasesTypes = new ArrayList<ParseTreeChunk>();
- List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
- for (ParseTreeChunk ch : parseResults) {
- String mainPos = ch.getMainPOS().toLowerCase();
+ public void setSize(int size) {
+ this.size = size;
+ }
- if (mainPos.equals("s")) {
- continue;
- }
- if (mainPos.equals("np")) {
- np.add(ch);
- } else if (mainPos.equals("vp")) {
- vp.add(ch);
- } else if (mainPos.equals("prp")) {
- prp.add(ch);
- } else if (mainPos.equals("pp")) {
- pp.add(ch);
- } else if (mainPos.equals("adjp")) {
- adjp.add(ch);
- } else if (mainPos.equals("whadvp")) {
- whadvp.add(ch);
- } else if (mainPos.equals("sbar")) {
- sbarp.add(ch);
- } else {
- restOfPhrasesTypes.add(ch);
- }
+ public LemmaFormManager getLemmaFormManager() {
+ return lemmaFormManager;
+ }
- }
- results.add(np);
- results.add(vp);
- results.add(prp);
- results.add(pp);
- results.add(adjp);
- results.add(whadvp);
- results.add(restOfPhrasesTypes);
+ public void setLemmaFormManager(LemmaFormManager lemmaFormManager) {
+ this.lemmaFormManager = lemmaFormManager;
+ }
- return results;
+ public GeneralizationListReducer getGeneralizationListReducer() {
+ return generalizationListReducer;
+ }
- }
+ public void setGeneralizationListReducer(
+ GeneralizationListReducer generalizationListReducer) {
+ this.generalizationListReducer = generalizationListReducer;
+ }
- // main function to generalize two expressions grouped by phrase types
- // returns a list of generalizations for each phrase type with filtered
- // sub-expressions
- public List<List<ParseTreeChunk>> matchTwoSentencesGroupedChunks(
- List<List<ParseTreeChunk>> sent1, List<List<ParseTreeChunk>> sent2) {
- List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
- // first irerate through component
- for (int comp = 0; comp < 2 && // just np & vp
- comp < sent1.size() && comp < sent2.size(); comp++) {
- List<ParseTreeChunk> resultComps = new ArrayList<ParseTreeChunk>();
- // then iterate through each phrase in each component
- for (ParseTreeChunk ch1 : sent1.get(comp)) {
- for (ParseTreeChunk ch2 : sent2.get(comp)) { // simpler version
- ParseTreeChunk chunkToAdd = parseTreeMatcher
- .generalizeTwoGroupedPhrasesRandomSelectHighestScoreWithTransforms(
- ch1, ch2);
+ public void setParseTreeMatcher(ParseTreeMatcher parseTreeMatcher) {
+ this.parseTreeMatcher = parseTreeMatcher;
+ }
- if (!lemmaFormManager.mustOccurVerifier(ch1, ch2, chunkToAdd)) {
- continue; // if the words which have to stay do not stay, proceed to
- // other elements
- }
- Boolean alreadyThere = false;
- for (ParseTreeChunk chunk : resultComps) {
- if (chunk.equalsTo(chunkToAdd)) {
- alreadyThere = true;
- break;
- }
+ public ParseTreeChunk(List<ParseTreeNode> ps) {
+ this.lemmas = new ArrayList<String>();
+ this.POSs = new ArrayList<String>();
+ for(ParseTreeNode n: ps){
+ this.lemmas.add(n.getWord());
+ this.POSs.add(n.getPos());
+ }
- if (parseTreeMatcher
- .generalizeTwoGroupedPhrasesRandomSelectHighestScore(chunk,
- chunkToAdd).equalsTo(chunkToAdd)) {
- alreadyThere = true;
- break;
- }
- }
+ if (ps.size()>0){
+ this.setMainPOS(ps.get(0).getPhraseType());
+ this.parseTreeNodes = ps;
+ }
+ }
- if (!alreadyThere) {
- resultComps.add(chunkToAdd);
- }
+ public List<ParseTreeChunk> buildChunks(List<LemmaPair> parseResults) {
+ List<ParseTreeChunk> chunksResults = new ArrayList<ParseTreeChunk>();
+ for (LemmaPair chunk : parseResults) {
+ String[] lemmasAr = chunk.getLemma().split(" ");
+ List<String> poss = new ArrayList<String>(), lems = new ArrayList<String>();
+ for (String lem : lemmasAr) {
+ lems.add(lem);
+ // now looking for POSs for individual word
+ for (LemmaPair chunkCur : parseResults) {
+ if (chunkCur.getLemma().equals(lem)
+ &&
+ // check that this is a proper word in proper position
+ chunkCur.getEndPos() <= chunk.getEndPos()
+ && chunkCur.getStartPos() >= chunk.getStartPos()) {
+ poss.add(chunkCur.getPOS());
+ break;
+ }
+ }
+ }
+ if (lems.size() != poss.size()) {
+ System.err.println("lems.size()!= poss.size()");
+ }
+ if (lems.size() < 2) { // single word phrase, nothing to match
+ continue;
+ }
+ ParseTreeChunk ch = new ParseTreeChunk(lems, poss, chunk.getStartPos(),
+ chunk.getEndPos());
+ ch.setMainPOS(chunk.getPOS());
+ chunksResults.add(ch);
+ }
+ return chunksResults;
+ }
- List<ParseTreeChunk> resultCompsReduced = generalizationListReducer
- .applyFilteringBySubsumption(resultComps);
- // if (resultCompsReduced.size() != resultComps.size())
- // System.out.println("reduction of gen list occurred");
- }
- }
- results.add(resultComps);
- }
+ public List<List<ParseTreeChunk>> matchTwoSentencesGivenPairLists(
+ List<LemmaPair> sent1Pairs, List<LemmaPair> sent2Pairs) {
- return results;
- }
+ List<ParseTreeChunk> chunk1List = buildChunks(sent1Pairs);
+ List<ParseTreeChunk> chunk2List = buildChunks(sent2Pairs);
- public Boolean equals(ParseTreeChunk ch) {
- List<String> lems = ch.getLemmas();
- List<String> poss = ch.POSs;
+ List<List<ParseTreeChunk>> sent1GrpLst = groupChunksAsParses(chunk1List);
+ List<List<ParseTreeChunk>> sent2GrpLst = groupChunksAsParses(chunk2List);
- if (this.lemmas.size() <= lems.size())
- return false; // sub-chunk should be shorter than chunk
+ System.out.println("=== Grouped chunks 1 " + sent1GrpLst);
+ System.out.println("=== Grouped chunks 2 " + sent2GrpLst);
- for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
- if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
- poss.get(i))))
- return false;
- }
- return true;
- }
+ return matchTwoSentencesGroupedChunks(sent1GrpLst, sent2GrpLst);
+ }
- // 'this' is super - chunk of ch, ch is sub-chunk of 'this'
- public Boolean isASubChunk(ParseTreeChunk ch) {
- List<String> lems = ch.getLemmas();
- List<String> poss = ch.POSs;
+ // groups noun phrases, verb phrases, propos phrases etc. for separate match
- if (this.lemmas.size() < lems.size())
- return false; // sub-chunk should be shorter than chunk
+ public List<List<ParseTreeChunk>> groupChunksAsParses(
+ List<ParseTreeChunk> parseResults) {
+ List<ParseTreeChunk> np = new ArrayList<ParseTreeChunk>(), vp = new ArrayList<ParseTreeChunk>(), prp = new ArrayList<ParseTreeChunk>(), sbarp = new ArrayList<ParseTreeChunk>(), pp = new ArrayList<ParseTreeChunk>(), adjp = new ArrayList<ParseTreeChunk>(), whadvp = new ArrayList<ParseTreeChunk>(), restOfPhrasesTypes = new ArrayList<ParseTreeChunk>();
+ List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
+ for (ParseTreeChunk ch : parseResults) {
+ String mainPos = ch.getMainPOS().toLowerCase();
- for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
- if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
- poss.get(i))))
- return false;
- }
- return true;
- }
+ if (mainPos.equals("s")) {
+ continue;
+ }
+ if (mainPos.equals("np")) {
+ np.add(ch);
+ } else if (mainPos.equals("vp")) {
+ vp.add(ch);
+ } else if (mainPos.equals("prp")) {
+ prp.add(ch);
+ } else if (mainPos.equals("pp")) {
+ pp.add(ch);
+ } else if (mainPos.equals("adjp")) {
+ adjp.add(ch);
+ } else if (mainPos.equals("whadvp")) {
+ whadvp.add(ch);
+ } else if (mainPos.equals("sbar")) {
+ sbarp.add(ch);
+ } else {
+ restOfPhrasesTypes.add(ch);
+ }
- public Boolean equalsTo(ParseTreeChunk ch) {
- List<String> lems = ch.getLemmas();
- List<String> poss = ch.POSs;
- if (this.lemmas.size() != lems.size() || this.POSs.size() != poss.size())
- return false;
+ }
+ results.add(np);
+ results.add(vp);
+ results.add(prp);
+ results.add(pp);
+ results.add(adjp);
+ results.add(whadvp);
+ results.add(restOfPhrasesTypes);
- for (int i = 0; i < lems.size(); i++) {
- if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
- poss.get(i))))
- return false;
- }
+ return results;
- return true;
- }
+ }
- public String toString() {
- String buf = " [";
- if (mainPOS != null)
- buf = mainPOS + " [";
- for (int i = 0; i < lemmas.size() && i < POSs.size() // && i<=3
- ; i++) {
- buf += POSs.get(i) + "-" + lemmas.get(i) + " ";
- }
- return buf + "]";
- }
+ // main function to generalize two expressions grouped by phrase types
+ // returns a list of generalizations for each phrase type with filtered
+ // sub-expressions
+ public List<List<ParseTreeChunk>> matchTwoSentencesGroupedChunks(
+ List<List<ParseTreeChunk>> sent1, List<List<ParseTreeChunk>> sent2) {
+ List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
+ // first irerate through component
+ for (int comp = 0; comp < 2 && // just np & vp
+ comp < sent1.size() && comp < sent2.size(); comp++) {
+ List<ParseTreeChunk> resultComps = new ArrayList<ParseTreeChunk>();
+ // then iterate through each phrase in each component
+ for (ParseTreeChunk ch1 : sent1.get(comp)) {
+ for (ParseTreeChunk ch2 : sent2.get(comp)) { // simpler version
+ ParseTreeChunk chunkToAdd = parseTreeMatcher
+ .generalizeTwoGroupedPhrasesRandomSelectHighestScoreWithTransforms(
+ ch1, ch2);
- public int compareTo(ParseTreeChunk o) {
- if (this.size > o.size)
- return -1;
- else
- return 1;
+ if (!lemmaFormManager.mustOccurVerifier(ch1, ch2, chunkToAdd)) {
+ continue; // if the words which have to stay do not stay, proceed to
+ // other elements
+ }
+ Boolean alreadyThere = false;
+ for (ParseTreeChunk chunk : resultComps) {
+ if (chunk.equalsTo(chunkToAdd)) {
+ alreadyThere = true;
+ break;
+ }
- }
+ if (parseTreeMatcher
+ .generalizeTwoGroupedPhrasesRandomSelectHighestScore(chunk,
+ chunkToAdd).equalsTo(chunkToAdd)) {
+ alreadyThere = true;
+ break;
+ }
+ }
- public String listToString(List<List<ParseTreeChunk>> chunks) {
- StringBuffer buf = new StringBuffer();
- if (chunks.get(0).size() > 0) {
- buf.append(" np " + chunks.get(0).toString());
- }
- if (chunks.get(1).size() > 0) {
- buf.append(" vp " + chunks.get(1).toString());
- }
- if (chunks.size() < 3) {
- return buf.toString();
- }
- if (chunks.get(2).size() > 0) {
- buf.append(" prp " + chunks.get(2).toString());
- }
- if (chunks.get(3).size() > 0) {
- buf.append(" pp " + chunks.get(3).toString());
- }
- if (chunks.get(4).size() > 0) {
- buf.append(" adjp " + chunks.get(4).toString());
- }
- if (chunks.get(5).size() > 0) {
- buf.append(" whadvp " + chunks.get(5).toString());
- }
- /*
- * if (mainPos.equals("np")) np.add(ch); else if (mainPos.equals( "vp"))
- * vp.add(ch); else if (mainPos.equals( "prp")) prp.add(ch); else if
- * (mainPos.equals( "pp")) pp.add(ch); else if (mainPos.equals( "adjp"))
- * adjp.add(ch); else if (mainPos.equals( "whadvp")) whadvp.add(ch);
- */
- return buf.toString();
- }
+ if (!alreadyThere) {
+ resultComps.add(chunkToAdd);
+ }
- public List<List<ParseTreeChunk>> obtainParseTreeChunkListByParsingList(
- String toParse) {
- List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
- // if (toParse.endsWith("]]]")){
- // toParse = toParse.replace("[[","").replace("]]","");
- // }
- toParse = toParse.replace(" ]], [ [", "&");
- String[] phraseTypeFragments = toParse.trim().split("&");
- for (String toParseFragm : phraseTypeFragments) {
- toParseFragm = toParseFragm.replace("], [", "#");
+ List<ParseTreeChunk> resultCompsReduced = generalizationListReducer
+ .applyFilteringBySubsumption(resultComps);
+ // if (resultCompsReduced.size() != resultComps.size())
+ // System.out.println("reduction of gen list occurred");
+ }
+ }
+ results.add(resultComps);
+ }
- List<ParseTreeChunk> resultsPhraseType = new ArrayList<ParseTreeChunk>();
- String[] indivChunks = toParseFragm.trim().split("#");
- for (String expr : indivChunks) {
- List<String> lems = new ArrayList<String>(), poss = new ArrayList<String>();
- expr = expr.replace("[", "").replace(" ]", "");
- String[] pairs = expr.trim().split(" ");
- for (String word : pairs) {
- word = word.replace("]]", "").replace("]", "");
- String[] pos_lem = word.split("-");
- lems.add(pos_lem[1].trim());
- poss.add(pos_lem[0].trim());
- }
- ParseTreeChunk ch = new ParseTreeChunk();
- ch.setLemmas(lems);
- ch.setPOSs(poss);
- resultsPhraseType.add(ch);
- }
- results.add(resultsPhraseType);
- }
- System.out.println(results);
- return results;
+ return results;
+ }
- // 2.1 | Vietnam <b>embassy</b> <b>in</b> <b>Israel</b>: information on how
- // to get your <b>visa</b> at Vietnam
- // <b>embassy</b> <b>in</b> <b>Israel</b>. <b>...</b> <b>Spain</b>.
- // Scotland. Sweden. Slovakia. Switzerland. T
- // [Top of Page] <b>...</b>
- // [[ [NN-* IN-in NP-israel ], [NP-* IN-in NP-israel ], [NP-* IN-* TO-* NN-*
- // ], [NN-visa IN-* NN-* IN-in ]], [
- // [VB-get NN-visa IN-* NN-* IN-in .-* ], [VBD-* IN-* NN-* NN-* .-* ], [VB-*
- // NP-* ]]]
+/* public Boolean equals(ParseTreeChunk ch) {
+ List<String> lems = ch.getLemmas();
+ List<String> poss = ch.POSs;
- }
+ if (this.lemmas.size() <= lems.size())
+ return false; // sub-chunk should be shorter than chunk
- public void setMainPOS(String mainPOS) {
- this.mainPOS = mainPOS;
- }
+ for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
+ if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
+ poss.get(i))))
+ return false;
+ }
+ return true;
+ }
+*/
+ // 'this' is super - chunk of ch, ch is sub-chunk of 'this'
+ public Boolean isASubChunk_OLD(ParseTreeChunk ch) {
+ List<String> lems = ch.getLemmas();
+ List<String> poss = ch.POSs;
- public String getMainPOS() {
- return mainPOS;
- }
+ if (this.lemmas.size() < lems.size())
+ return false; // sub-chunk should be shorter than chunk
- public List<String> getLemmas() {
- return lemmas;
- }
+ for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
+ if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
+ poss.get(i))))
+ return false;
+ }
+ return true;
+ }
+
+ // this => value ch => *
+ public Boolean isASubChunk(ParseTreeChunk ch) {
+ List<String> lems = ch.getLemmas();
+ List<String> poss = ch.POSs;
- public void setLemmas(List<String> lemmas) {
- this.lemmas = lemmas;
- }
+ if (this.lemmas.size() < lems.size())
+ return false; // sub-chunk should be shorter than chunk
- public List<String> getPOSs() {
- return POSs;
- }
+ Boolean notSubChunkWithGivenAlignment = false, unComparable = false;
+
+ for (int i = 0; i < lems.size() && i < this.lemmas.size(); i++) {
+ // both lemma and pos are different
+ if (!this.POSs.get(i).equals(poss.get(i)) && !this.lemmas.get(i).equals(lems.get(i)) ){
+ unComparable = true;
+ break;
+ }
+
+ // this => * ch=> run
+ if (!this.lemmas.get(i).equals(lems.get(i)) && this.lemmas.get(i).equals("*"))
+ notSubChunkWithGivenAlignment = true;
+ }
+ if (!notSubChunkWithGivenAlignment && !unComparable)
+ return true;
+
+ List<String> thisPOS = new ArrayList<String> ( this.POSs);
+ Collections.reverse(thisPOS);
+ List<String> chPOS = new ArrayList<String> ( poss);
+ Collections.reverse(chPOS);
+ List<String> thisLemma = new ArrayList<String> ( this.lemmas);
+ Collections.reverse(thisLemma );
+ List<String> chLemma = new ArrayList<String> ( lems);
+ Collections.reverse(chLemma);
+
+ notSubChunkWithGivenAlignment = false; unComparable = false;
+ for (int i = lems.size()-1 ; i>=0; i--) {
+ // both lemma and pos are different
+ if (!thisPOS.get(i).equals(chPOS.get(i)) && !thisLemma.get(i).equals(chLemma.get(i)) ){
+ unComparable = true;
+ break;
+ }
+
+ // this => * ch=> run
+ if (!thisLemma.get(i).equals(chLemma.get(i)) && thisLemma.get(i).equals("*"))
+ notSubChunkWithGivenAlignment = true;
+ }
+
+ if (!notSubChunkWithGivenAlignment && !unComparable)
+ return true;
+ else
+ return false; // then ch is redundant and needs to be removed
+ }
- public void setPOSs(List<String> pOSs) {
- POSs = pOSs;
- }
+ public Boolean equalsTo(ParseTreeChunk ch) {
+ List<String> lems = ch.getLemmas();
+ List<String> poss = ch.POSs;
+ if (this.lemmas.size() != lems.size() || this.POSs.size() != poss.size())
+ return false;
- public ParseTreeMatcher getParseTreeMatcher() {
- return parseTreeMatcher;
- }
+ for (int i = 0; i < lems.size(); i++) {
+ if (!(this.lemmas.get(i).equals(lems.get(i)) && this.POSs.get(i).equals(
+ poss.get(i))))
+ return false;
+ }
+ return true;
+ }
+
+ public boolean equals(ParseTreeChunk ch) {
+ List<String> lems = ch.getLemmas();
+ List<String> poss = ch.POSs;
+ return ListUtils.isEqualList(ch.getLemmas(), this.lemmas) && ListUtils.isEqualList(ch.getPOSs(), this.POSs);
+ }
+
+ public String toString() {
+ String buf = " [";
+ if (mainPOS != null)
+ buf = mainPOS + " [";
+ for (int i = 0; i < lemmas.size() && i < POSs.size() ; i++) {
+ buf += POSs.get(i) + "-" + lemmas.get(i) + " ";
+ if (this.parseTreeNodes!=null){
+ Map<String, Object> attrs = this.parseTreeNodes.get(i).getAttributes();
+ if (attrs!=null && attrs.keySet().size()>0){
+ buf += attrs+ " ";
+ }
+ String ner =this.parseTreeNodes.get(i).getNe();
+ if (ner!=null && ner.length()>1)
+ buf+="("+ner+ ") ";
+ }
+ }
+ return buf + "]";
+ }
+
+ public String toWordOnlyString(){
+ String buf = "";
+
+ for (int i = 0; i < lemmas.size() ; i++) {
+ buf+=lemmas.get(i)+" ";
+ }
+ return buf.trim();
+ }
+
+ public int compareTo(ParseTreeChunk o) {
+ if (this.size > o.size)
+ return -1;
+ else
+ return 1;
+
+ }
+
+ public String listToString(List<List<ParseTreeChunk>> chunks) {
+ StringBuffer buf = new StringBuffer();
+ if (chunks.get(0).size() > 0) {
+ buf.append(" np " + chunks.get(0).toString());
+ }
+ if (chunks.get(1).size() > 0) {
+ buf.append(" vp " + chunks.get(1).toString());
+ }
+ if (chunks.size() < 3) {
+ return buf.toString();
+ }
+ if (chunks.get(2).size() > 0) {
+ buf.append(" prp " + chunks.get(2).toString());
+ }
+ if (chunks.get(3).size() > 0) {
+ buf.append(" pp " + chunks.get(3).toString());
+ }
+ if (chunks.get(4).size() > 0) {
+ buf.append(" adjp " + chunks.get(4).toString());
+ }
+ if (chunks.get(5).size() > 0) {
+ buf.append(" whadvp " + chunks.get(5).toString());
+ }
+ /*
+ * if (mainPos.equals("np")) np.add(ch); else if (mainPos.equals( "vp"))
+ * vp.add(ch); else if (mainPos.equals( "prp")) prp.add(ch); else if
+ * (mainPos.equals( "pp")) pp.add(ch); else if (mainPos.equals( "adjp"))
+ * adjp.add(ch); else if (mainPos.equals( "whadvp")) whadvp.add(ch);
+ */
+ return buf.toString();
+ }
+
+ public List<List<ParseTreeChunk>> obtainParseTreeChunkListByParsingList(
+ String toParse) {
+ List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
+ // if (toParse.endsWith("]]]")){
+ // toParse = toParse.replace("[[","").replace("]]","");
+ // }
+ toParse = toParse.replace(" ]], [ [", "&");
+ String[] phraseTypeFragments = toParse.trim().split("&");
+ for (String toParseFragm : phraseTypeFragments) {
+ toParseFragm = toParseFragm.replace("], [", "#");
+
+ List<ParseTreeChunk> resultsPhraseType = new ArrayList<ParseTreeChunk>();
+ String[] indivChunks = toParseFragm.trim().split("#");
+ for (String expr : indivChunks) {
+ List<String> lems = new ArrayList<String>(), poss = new ArrayList<String>();
+ expr = expr.replace("[", "").replace(" ]", "");
+ String[] pairs = expr.trim().split(" ");
+ for (String word : pairs) {
+ word = word.replace("]]", "").replace("]", "");
+ String[] pos_lem = word.split("-");
+ lems.add(pos_lem[1].trim());
+ poss.add(pos_lem[0].trim());
+ }
+ ParseTreeChunk ch = new ParseTreeChunk();
+ ch.setLemmas(lems);
+ ch.setPOSs(poss);
+ resultsPhraseType.add(ch);
+ }
+ results.add(resultsPhraseType);
+ }
+ System.out.println(results);
+ return results;
+
+ // 2.1 | Vietnam <b>embassy</b> <b>in</b> <b>Israel</b>: information on how
+ // to get your <b>visa</b> at Vietnam
+ // <b>embassy</b> <b>in</b> <b>Israel</b>. <b>...</b> <b>Spain</b>.
+ // Scotland. Sweden. Slovakia. Switzerland. T
+ // [Top of Page] <b>...</b>
+ // [[ [NN-* IN-in NP-israel ], [NP-* IN-in NP-israel ], [NP-* IN-* TO-* NN-*
+ // ], [NN-visa IN-* NN-* IN-in ]], [
+ // [VB-get NN-visa IN-* NN-* IN-in .-* ], [VBD-* IN-* NN-* NN-* .-* ], [VB-*
+ // NP-* ]]]
+
+ }
+
+ public void setMainPOS(String mainPOS) {
+ this.mainPOS = mainPOS;
+ }
+
+ public String getMainPOS() {
+ return mainPOS;
+ }
+
+ public List<String> getLemmas() {
+ return lemmas;
+ }
+
+ public void setLemmas(List<String> lemmas) {
+ this.lemmas = lemmas;
+ }
+
+ public List<String> getPOSs() {
+ return POSs;
+ }
+
+ public void setPOSs(List<String> pOSs) {
+ POSs = pOSs;
+ }
+
+ public ParseTreeMatcher getParseTreeMatcher() {
+ return parseTreeMatcher;
+ }
+
+ public static void main(String[] args){
+ String phrStr = "[<1>NP'Property':NN, <2>NP'has':VBZ, <3>NP'lots':NNS, <4>NP'of':IN, <5>NP'trash':NN, <6>NP'and':CC, <7>NP'debris':NN]";
+ ParseTreeChunk ch = new ParseTreeChunk(phrStr);
+ System.out.println(ch);
+ }
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunkListScorer.java b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunkListScorer.java
index e085792..e9a0368 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunkListScorer.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeChunkListScorer.java
@@ -19,6 +19,8 @@
import java.util.List;
+import opennlp.tools.parse_thicket.matching.LemmaGeneralizer;
+
public class ParseTreeChunkListScorer {
// find the single expression with the highest score
public double getParseTreeChunkListScore(
@@ -72,7 +74,16 @@
} else {
score += 0.1;
}
- } else {
+ } else if (l.startsWith(LemmaGeneralizer.w2vPrefix) ){
+ try {
+ float val = Float.parseFloat(l.substring(LemmaGeneralizer.w2vPrefix.length()));
+ score+= 1- val;
+ } catch (NumberFormatException e) {
+ e.printStackTrace();
+ }
+ }
+
+ else {
if (pos.startsWith("NN") || pos.startsWith("NP")
|| pos.startsWith("CD") || pos.startsWith("RB")) {
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeMatcherDeterministic.java b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeMatcherDeterministic.java
index a58b104..2949552 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeMatcherDeterministic.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/ParseTreeMatcherDeterministic.java
@@ -19,7 +19,7 @@
import java.util.ArrayList;
import java.util.List;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
public class ParseTreeMatcherDeterministic {
@@ -48,7 +48,7 @@
List<String> lem1stem = new ArrayList<String>();
List<String> lem2stem = new ArrayList<String>();
- PorterStemmer ps = new PorterStemmer();
+ PStemmer ps = new PStemmer();
for (String word : lem1) {
try {
lem1stem.add(ps.stem(word.toLowerCase()).toString());
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/TextProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/TextProcessor.java
index 37d83aa..39e62b4 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/TextProcessor.java
+++ b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/TextProcessor.java
@@ -31,7 +31,7 @@
import java.util.logging.Logger;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import opennlp.tools.stemmer.PorterStemmer;
+import opennlp.tools.stemmer.PStemmer;
import opennlp.tools.similarity.apps.utils.Pair;
import org.apache.commons.lang.StringUtils;
@@ -489,7 +489,7 @@
}
}
- return new PorterStemmer().stem(token).toString();
+ return new PStemmer().stem(token).toString();
}
public static String cleanToken(String token) {
@@ -534,7 +534,7 @@
public static String stemTerm(String term) {
term = stripToken(term);
- PorterStemmer st = new PorterStemmer();
+ PStemmer st = new PStemmer();
return st.stem(term).toString();
}
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/readme.txt b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/readme.txt
index 41765dd..b796290 100644
--- a/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/readme.txt
+++ b/opennlp-similarity/src/main/java/opennlp/tools/textsimilarity/readme.txt
@@ -1,3 +1,18 @@
+
+opennlp/tools.apps -? similarity.apps
+
+textsimilarity : sentence-level SG (based on opennlp)
+parse_thicket: paragraph-level SG (based on stanford NLP)
+
+matching.utils - all old classed, might be working better
+
+apps.search
+apps.content_generation
+parse_thicket.apps.lattice_queries
+
+
+
+
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreemnets. See the NOTICE file distributed with
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/apps/RelatedSentenceFinderTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/apps/RelatedSentenceFinderTest.java
index c9e70ef..f385a69 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/apps/RelatedSentenceFinderTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/apps/RelatedSentenceFinderTest.java
@@ -35,14 +35,12 @@
input.setAbstractText("He is pictured here in the Swiss Patent Office where he did ...");
input.setUrl("http://apod.nasa.gov/apod/ap951219.html");
input.setTitle("Albert Einstein");
- HitBase result = finder.//augmentWithMinedSentencesAndVerifyRelevance(input,
- buildParagraphOfGeneratedText(input,
- "Swiss Patent Office", new ArrayList<String>());
+ HitBase result = finder.buildParagraphOfGeneratedText(input, "Swiss Patent Office", new ArrayList<String>());
System.out.println(result.toString());
assertTrue(result.getOriginalSentences()!=null);
assertTrue(result.getOriginalSentences().size()>0);
- assertTrue(result.getFragments().size()>0);
- assertTrue(result.getFragments().get(0).getFragment().indexOf("Swiss Patent Office")>-1);
+ //assertTrue(result.getFragments().size()>0);
+ //assertTrue(result.getFragments().get(0).getFragment().indexOf("Swiss Patent Office")>-1);
}
@@ -78,7 +76,7 @@
public void testBuildParagraphOfGeneratedTextTestBio1(){
HitBase input = new HitBase();
- input.setAbstractText("Today, the practical applications of Einsteins theories ...");
+ input.setAbstractText("Today, the practical applications of Einstein�s theories ...");
input.setUrl("http://einstein.biz/biography.php");
input.setTitle("Biography");
HitBase result = finder.buildParagraphOfGeneratedText(input,
@@ -89,7 +87,7 @@
assertTrue(result.getFragments().size()>0);
assertTrue(result.getFragments().get(0).getFragment().indexOf("Einstein")>-1);
}
-
+/*
public void testBuildParagraphOfGeneratedTextTestBio2(){
HitBase input = new HitBase();
input.setAbstractText("The theory of relativity is a beautiful example of ...");
@@ -116,7 +114,7 @@
assertTrue(result.getOriginalSentences().size()>0);
assertTrue(result.getFragments().size()>0);
assertTrue(result.getFragments().get(0).getFragment().indexOf("cannot conceive")>-1);
- }
+ }
public void testBuildParagraphOfGeneratedTextTestBio4(){
@@ -131,12 +129,12 @@
assertTrue(result.getOriginalSentences().size()>0);
assertTrue(result.getFragments().size()>0);
assertTrue(result.getFragments().get(0).getFragment().indexOf("view of the world")>-1);
- }
+ } */
}
-//[Albert Einstein (/ælbrt anstan/; German. albt antan ( listen); 14 March 1879 18 April 1955) was a German-born theoretical physicist who developed the general theory of relativity, one of the two pillars of modern physics (alongside quantum mechanics). 2 3 While best known for his massenergy equivalence formula E = mc2 (which has been dubbed "the world's most famous equation"), 4 he received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect". 5 The latter was pivotal in establishing quantum theory. nullNear the beginning of his career, Einstein thought that Newtonian mechanics was no longer enough to reconcile the laws of classical mechanics with the laws of the electromagnetic field. This led to the development of his special theory of relativity.,
+//[Albert Einstein (/�lbrt anstan/; German. albt antan ( listen); 14 March 1879 18 April 1955) was a German-born theoretical physicist who developed the general theory of relativity, one of the two pillars of modern physics (alongside quantum mechanics). 2 3 While best known for his massenergy equivalence formula E = mc2 (which has been dubbed "the world's most famous equation"), 4 he received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect". 5 The latter was pivotal in establishing quantum theory. nullNear the beginning of his career, Einstein thought that Newtonian mechanics was no longer enough to reconcile the laws of classical mechanics with the laws of the electromagnetic field. This led to the development of his special theory of relativity.,
-//"Today, the practical applications of Einsteins theories include the development of the television"
\ No newline at end of file
+//"Today, the practical applications of Einstein�s theories include the development of the television"
\ No newline at end of file
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilderTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilderTest.java
index 12ae8ff..0517f4c 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilderTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PT2ThicketPhraseBuilderTest.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.matching;
import java.util.List;
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTMatcherTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTMatcherTest.java
index 9761bb2..7d2ebef 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTMatcherTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTMatcherTest.java
@@ -1,34 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.matching;
+import java.io.File;
import java.util.ArrayList;
import java.util.List;
import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
import opennlp.tools.textsimilarity.ParseTreeChunk;
import junit.framework.TestCase;
public class PTMatcherTest extends TestCase {
+ //public static String resourceDir = new File(".").getAbsolutePath().replace("/.", "") + "/src/test/resources";
+ //VerbNetProcessor proc = VerbNetProcessor.getInstance(resourceDir);
Matcher m = new Matcher();
public void testMatchTwoParaTestReduced(){
String q = "I am a US citizen living abroad, and concerned about the health reform regulation of 2014. I do not want to wait till I am sick to buy health insurance. I am afraid I will end up paying the tax.";
String a = "People are worried about having to pay a fine for not carrying health insurance coverage got more guidance this week with some new federal regulations. "+
"Hardly anyone will end up paying the tax when the health reform law takes full effect in 2014. "+
- "The individual mandate makes sure that people dont wait until they are sick to buy health insurance. "+
+ "The individual mandate makes sure that people don�t wait until they are sick to buy health insurance. "+
"People are exempt from health insurance fine if they make too little money to file an income tax return, or US citizens living abroad.";
List<List<ParseTreeChunk>> res = m.assessRelevance(q, a);
System.out.print(res);
assertTrue(res!=null);
assertTrue(res.size()>0);
- assertEquals(res.toString(), "[[ [NNP-us NN-citizen VBG-living RB-abroad ], [,-, CC-* ], [DT-a NNP-* ], [DT-the NN-* NN-health NN-reform NN-* CD-2014 ], [NN-* IN-* CD-2014 ], [NN-health NN-* NN-* IN-* ], [NN-regulation ], " +
- " [DT-the NN-health NN-reform NN-* ], [CD-2014 ], [NN-health NN-insurance ], [DT-the NN-tax ], [NN-tax ]], [ [VBP-* DT-a NNP-* NN-health NN-* NN-* NN-regulation ], [NN-health NN-* NN-* NN-regulation ], [NN-regulation ], " +
- " [DT-the NN-* NN-health NN-reform NN-* CD-2014 ], [NN-* IN-* CD-2014 ], [IN-* NN-health NN-* ], [NNP-us NN-citizen VBG-living RB-abroad ], [,-, CC-* ], [NN-health NN-* NN-* IN-* ], " +
- " [IN-about NN-health NN-* NN-* NN-regulation ], [VBG-living RB-abroad ], [TO-to VB-* VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [TO-to VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], " +
- "[TO-to VB-* NN-health NN-insurance ], [TO-to VB-buy NN-health NN-insurance ], [VB-* TO-to VB-* VB-* NN-health NN-insurance ], [TO-to VB-* VB-* NN-health NN-insurance ], [RB-not VB-* NN-health NN-insurance ], [VBG-paying DT-* NN-* ], " +
- "[MD-will VB-end RP-up VBG-paying DT-the NN-tax ], [VB-end RP-up VBG-paying DT-the NN-tax ], [VBG-paying DT-the NN-tax ], [VBP-do RB-* VB-* TO-* TO-to VB-* ], [VB-* VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], " +
- " [VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [TO-to VB-* VB-buy NN-health NN-insurance ], [VB-buy NN-health NN-insurance ], [NN-health NN-insurance NN-tax ], " +
- "[TO-to VB-* NN-tax ], [NN-tax ], [VB-* TO-to VB-* VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VB-* TO-to VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VB-* NN-health NN-insurance ], [VB-* VBG-paying DT-* NN-* ]]]");
+ assertEquals( "[[NP [NNP-us (LOCATION) NN*-citizen VB-living RB-abroad ], NP [,-, CC-* ], NP [DT-the NN-* NN-health NN-reform NN-* CD-2014 ], NP [NN-health NN-* NN-* IN-* ], NP [DT-the NN-health NN-reform NN-* ], NP [NN-health NN-insurance ], NP [NN*-* NN-* JJ-* NN-* ]], [VP [VB-* {phrStr=[], phrDescr=[], roles=[A, *, *]} DT-a NN*-* NN-health NN-* NN-* NN*-regulation ], VP [VB-* NN*-* NN-* VB-* RB*-* IN-* DT-* NN*-regulation ], VP [VB-* NN-* NN-health NN-* NN-* ], VP [IN-about NN-health NN-* NN-* NN*-regulation ], VP [VB-living RB-abroad ], VP [TO-to VB-* VB-wait IN-* PRP-* VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], VP [VB-* TO-to VB-* VB-* NN-health NN-insurance ], UCP [MD-will VB-end RP-up VB-paying DT-the NN-tax ], VP [TO-to VB-* VB-buy NN-health NN-insurance ], VP [VB-* TO-to VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ]]]"
+ , res.toString());
}
@@ -45,8 +60,8 @@
System.out.print(res);
assertTrue(res!=null);
assertTrue(res.size()>0);
- assertEquals(res.toString(), "[[ [DT-the NNP-un NN-* ], [PRP$-its JJ-nuclear NNS-weapons ], [NN-work IN-on JJ-nuclear NNS-weapons ], [PRP$-its NN-* JJ-nuclear NNS-* ], [PRP$-its JJ-nuclear NNS-* ], [DT-a NN-* PRP$-its JJ-* NN-* ], [DT-a NN-resolution VBG-* NNP-iran IN-* VBG-developing PRP$-its NN-uranium NN-enrichment NN-site ], [NN-* VBG-* NNP-iran ], [DT-a NN-resolution VBG-* NNP-* NNP-iran ], [DT-a NN-resolution NNP-iran ], [DT-a NNP-iran ], [DT-a PRP$-its ], [NNP-iran IN-* VBG-developing PRP$-its NN-uranium NN-enrichment NN-site ], [IN-for ], [VBG-* PRP$-its JJ-* NN-* ], [PRP$-its NN-uranium NN-enrichment NN-site ], [PRP$-its JJ-* NN-* ], [VBD-* NNP-iran VBD-was VBG-working IN-on JJ-nuclear NNS-weapons ], [VBG-* JJ-nuclear NNS-* ], [JJ-nuclear NNS-weapons ], [JJ-nuclear NNS-* ], [NNP-iran NN-envoy ], [NN-* IN-* PRP-it ], [NN-* PRP-it ], [DT-the NN-* NN-evidence IN-against PRP-it ], [DT-the NN-* NN-* ], [PRP-it ], [DT-the NNP-us ], [DT-the NNP-* ], [DT-a NN-resolution DT-a JJ-recent NNP-* NN-report ], [DT-a JJ-recent NNP-* NN-report ], [NN-* PRP$-its JJ-nuclear NN-* ], [PRP$-its JJ-nuclear NN-* ], [VBZ-* PRP$-its ], [NN-development ], [PRP$-its JJ-nuclear NN-development ], [JJ-peaceful NN-purpose ], [NN-* VBZ-says ], [NNP-un JJ-nuclear NN-* VBZ-* ], [NN-* VBZ-* PRP$-its JJ-nuclear NN-development VBZ-is IN-for JJ-peaceful NN-purpose ], [JJ-nuclear NN-* VBZ-* NN-development VBZ-is IN-for JJ-peaceful NN-purpose ], [NNP-un NN-* PRP$-its ]], [ [VBZ-refuses TO-to VB-* DT-* NNP-* ], [VB-* DT-the NNP-un NN-* TO-to VB-end PRP$-its ], [NNP-un ], [NNP-* NN-* TO-to ], [TO-to VB-end PRP$-its ], [VBZ-* DT-a NN-* PRP$-its JJ-* NN-* ], [VBZ-passes DT-a NN-resolution VBG-* NNP-iran IN-* VBG-developing PRP$-its NN-uranium NN-enrichment NN-site ], [NN-* VBG-* NNP-iran ], [VBG-* NNP-iran IN-* VBG-developing PRP$-its NN-uranium NN-enrichment NN-site ], [IN-for ], [PRP$-its JJ-* NN-* ], [VBG-developing PRP$-its NN-uranium NN-enrichment NN-site ], [VBG-* PRP$-its JJ-* NN-* ], [VBD-presented NNS-* NNP-iran VBD-was VBG-working IN-on JJ-nuclear NNS-weapons ], [VBD-* NNP-iran VBD-was VBG-working IN-on JJ-nuclear NNS-weapons ], [NNP-iran ], [VBD-was VBG-working IN-on JJ-nuclear NNS-weapons ], [JJ-nuclear NNS-weapons ], [VBG-* JJ-nuclear NNS-* ], [VBG-working IN-on JJ-nuclear NNS-weapons ], [PRP$-its JJ-nuclear NN-* ], [NN-development ], [VBZ-says JJ-nuclear NN-* ], [VBZ-* PRP$-its JJ-nuclear NN-development VBZ-is IN-for JJ-peaceful NN-purpose ], [VBZ-* JJ-nuclear NN-* ], [VBZ-is IN-for JJ-peaceful NN-purpose ], [VBN-* VBN-fabricated IN-by DT-the NNP-us ], [VBN-fabricated IN-by DT-the NNP-us ], [TO-to VB-* DT-* NNP-* VB-end PRP$-its ], [VB-end PRP$-its ], [NN-* IN-over PRP$-its ], [PRP$-its JJ-nuclear NNS-weapons ], [DT-a ], [TO-* VB-* PRP$-its NN-* ], [VB-* PRP$-its NN-* ], [VB-* PRP$-its JJ-nuclear NNS-* ], [DT-the NNP-* ], [TO-to NNP-un ], [NN-work IN-on JJ-nuclear NNS-weapons ]]]");
- }
+ assertEquals(res.toString(),
+ "[[NP [DT-a NN-* PRP$-its JJ-* NN-* ], NP [DT-a NN-resolution VB-* NNP-iran (LOCATION) IN-* VB-developing PRP$-its NN-uranium NN-enrichment NN-site ], NP [DT-a IN-for ], NP [DT-a PRP$-its ], NP [VB-* JJ-nuclear NN*-* ], NP [JJ-nuclear NNS-weapons ], NP [PRP$-its JJ-nuclear NN-development ], NP [DT-the NN-* NN-evidence IN-against PR*-it ], NP [DT-the NNP-un (ORGANIZATION) NN-* ], NP [VB-* NN-* NN-* NN-* ], NP [VB-* NNP-iran (LOCATION) NN*-* ], NP [NNP-iran (LOCATION) NN-envoy ]], [VP [VB-refuses TO-to VB-* DT-* NN*-* ], VP [VB-* DT-the NNP-un (ORGANIZATION) NN-* TO-to VB-end PRP$-its ], VP [VB-* NN-* NN-work IN-on JJ-nuclear NN*-weapons.un ], VP [VB-* DT-a NN-* NN-resolution VB-* NNP-iran (LOCATION) IN-* VB-developing PRP$-its ], VP [VB-* DT-a NN-* PRP$-its JJ-* NN-* ], VP [VB-passes DT-a NN-resolution VB-* NNP-iran (LOCATION) IN-* VB-developing PRP$-its NN-uranium NN-enrichment NN-site ], VP [PRP$-its JJ-* NN-* NN-uranium NN-enrichment NN-site ], VP [VB-presented NNS-* NNP-iran (LOCATION) VB-was VB-working IN-on JJ-nuclear NNS-weapons ], VP [VB-* VB-fabricated IN-by DT-the NNP-us (LOCATION) ], VP [VB-* DT-the NNP-un (ORGANIZATION) NN-* TO-to VB-end NN-* IN-over PRP$-its NNP-* ], VP [TO-to VB-* DT-* NN*-* VB-end PRP$-its ], VP [PRP$-its JJ-nuclear NN-weapons.un ], VP [IN-* VB-* PRP$-its NN-* ], VP [DT-a PRP$-its JJ-nuclear NN-* VB-* NN-development ], VP [DT-a VB-* PRP$-its ], VP [VB-* NN-development NN-* ], VP [NN*-* VB-says JJ-nuclear NN*-* ], VP [VB-is IN-for JJ-peaceful NN-purpose ]]]" ) ; }
public void testMatchTwoParaTest2(){
List<List<ParseTreeChunk>> res = m.assessRelevance("I am a US citizen living abroad, and concerned about the health reform regulation of 2014. "+
@@ -56,17 +71,13 @@
,
"People are worried about having to pay a fine for not carrying health insurance coverage got more guidance this week with some new federal regulations. "+
"Hardly anyone will end up paying the tax when the health reform law takes full effect in 2014. "+
- "The individual mandate makes sure that people dont wait until they are sick to buy health insurance. "+
+ "The individual mandate makes sure that people don�t wait until they are sick to buy health insurance. "+
"People are exempt from health insurance fine if they make too little money to file an income tax return, or US citizens living abroad.");
System.out.print(res);
assertTrue(res!=null);
assertTrue(res.size()>0);
- assertEquals(res.toString(), "[[ [NNP-us NN-citizen VBG-living RB-abroad ], [,-, CC-* ], [DT-a NNP-* ], [DT-the NN-* NN-health NN-reform NN-* CD-2014 ], " +
- "[NN-* IN-* CD-2014 ], [NN-health NN-* NN-* IN-* ], [NN-regulation ], [DT-the NN-health NN-reform NN-* ], [CD-2014 ], [DT-the NN-tax ], [NN-tax ], " +
- " [DT-a NN-fine ], [NN-health NN-insurance NN-coverage ], [TO-to VB-* DT-* NN-* ], [NN-fine IN-* ], [NN-health NN-insurance NN-* ]], " +
- "[ [VBP-* DT-a NNP-* NN-health NN-* NN-* NN-regulation ], [NN-health NN-* NN-* NN-regulation ], [NN-regulation ], [DT-the NN-* NN-health NN-reform NN-* CD-2014 ], " +
- " [NN-* IN-* CD-2014 ], [IN-* NN-health NN-* ], [NNP-us NN-citizen VBG-living RB-abroad ], [,-, CC-* ], [NN-health NN-* NN-* IN-* ], [IN-about NN-health NN-* NN-* NN-regulation ], [VBG-living RB-abroad ], [TO-to VB-* VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [TO-to VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [TO-to VB-buy NN-health NN-insurance ], [VBG-* VB-pay DT-* NN-* NN-health NN-* NN-* ], [VB-pay DT-* NN-* NN-health NN-* NN-* ], [RB-not VBG-* NN-health NN-insurance NN-coverage ], [VBG-having NN-health NN-insurance NN-coverage ], [NN-health NN-insurance NN-tax ], [TO-to VB-* NN-tax ], [VB-* TO-to VB-* VB-* NN-health NN-insurance ], [TO-to VB-* VB-* NN-health NN-insurance ], [TO-to VB-* VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [RB-not VB-* NN-health NN-insurance NN-coverage ], [VBP-do RB-* VB-* TO-* TO-to VB-* ], [VB-* VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [TO-to VB-* VB-buy NN-health NN-insurance ], [VB-buy NN-health NN-insurance ], [VB-* TO-to VB-* VB-wait IN-* PRP-* VBP-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VB-* TO-to VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], [VB-* TO-to VB-* VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [VB-* NN-health NN-insurance NN-coverage ], [VBG-having TO-to VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [TO-to VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [VBG-paying DT-* NN-* DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [VBG-* NN-health NN-insurance NN-coverage ], [MD-will VB-end RP-up VBG-paying DT-the NN-tax ], [VB-end RP-up VBG-paying DT-the NN-tax NN-health NN-* NN-* ], [VBG-paying DT-the NN-tax NN-health NN-* NN-* ], [TO-to VB-* NN-health NN-insurance ], [NN-fine IN-* ], [NN-health NN-insurance NN-* ], [TO-to VB-* DT-* NN-* ], [NN-tax ], [VBP-* VBN-worried IN-about VBG-having TO-to VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], [VB-* VBG-paying DT-* NN-* DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ], " +
- " [VBN-worried IN-about VBG-having TO-to VB-pay DT-a NN-fine IN-for RB-not VBG-* NN-health NN-insurance NN-coverage ]]]");
+ assertEquals(res.toString(), "[[NP [NNP-us (LOCATION) NN*-citizen VB-living RB-abroad ], NP [,-, CC-* ], NP [DT-the NN-* NN-health NN-reform NN-* CD-2014 ], NP [NN-health NN-* NN-* IN-* ], NP [DT-the NN-health NN-reform NN-* ], UCP [NN-health NN-insurance NN-coverage ], UCP [TO-to VB-* {phrStr=[], phrDescr=[], roles=[A, *, *]} DT-a NN-* ], NP [NN*-* NN-* JJ-* NN-* ]], [VP [VB-* {phrStr=[], phrDescr=[], roles=[A, *, *]} DT-a NN*-* NN-health NN-* NN-* NN*-regulation ], VP [VB-* NN*-* NN-* VB-* RB*-* IN-* DT-* NN*-regulation ], VP [IN-about NN-health NN-* NN-* NN*-regulation ], VP [VB-living RB-abroad ], VP [TO-to VB-* VB-wait IN-* PRP-* VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], VP [VB-* VB-pay DT-* NN-* NN-health NN-* NN-* ], VP [VB-having NN-health NN-insurance NN-coverage ], UCP [MD-will VB-end RP-up VB-paying DT-the NN-tax ], VP [VB-* TO-to VB-* VB-* NN-health NN-insurance ], VP [TO-to VB-* VB-buy NN-health NN-insurance ], VP [VB-* TO-to VB-* JJ-sick TO-to VB-buy NN-health NN-insurance ], VP [VB-* TO-to VB-* VB-pay {phrStr=[NP V NP PP.theme, NP V NP], phrDescr=[NP-PPfor-PP, (SUBCAT MP)], roles=[A, A, T]} DT-a NN-fine IN-for RB-not VB-* NN-health NN-insurance NN-coverage ], VP [VB-paying DT-the NN-tax NN-health NN-* NN-* ], VP [VB-* TO-to VB-* NN-health NN-insurance ], UCP [VB-* VB-worried IN-about VB-having TO-to VB-pay {phrStr=[NP V NP PP.theme, NP V NP], phrDescr=[NP-PPfor-PP, (SUBCAT MP)], roles=[A, A, T]} DT-a NN-fine IN-for RB-not VB-* NN-health NN-insurance NN-coverage ], VP [VB-paying DT-* NN-* DT-a NN-fine IN-for RB-not VB-* NN-health NN-insurance NN-coverage ]]]"
+ );
}
@@ -78,7 +89,7 @@
,
"People are worried about paying a fine for not carrying health insurance coverage, having been informed by IRS about new regulations. "+
"Yet hardly anyone is expected to pay the tax, when the health reform law takes full effect in 2014. "+
- "The individual mandate confirms that people dont wait until they are sick to buy health insurance. "+
+ "The individual mandate confirms that people don�t wait until they are sick to buy health insurance. "+
"People are exempt from health insurance fine if they report they make too little money, or US citizens living abroad.");
System.out.print(res);
assertTrue(res!=null);
@@ -93,13 +104,35 @@
String text2 = "People are worried about paying a fine for not carrying health insurance coverage, having been informed by IRS about new regulations. "+
"Yet hardly anyone is expected to pay the tax, when the health reform law takes full effect in 2014. "+
- "The individual mandate confirms that people dont wait until they are sick to buy health insurance. "+
+ "The individual mandate confirms that people don�t wait until they are sick to buy health insurance. "+
"People are exempt from health insurance fine if they report they make too little money, or US citizens living abroad.";
List<List<ParseTreeChunk>> res = m.assessRelevance(text1, text2);
System.out.print(res);
assertTrue(res!=null);
assertTrue(res.size()>0);
}
+
+ public void testMatchTwoParaTestREq1(){
+ String q = "I am buying a foreclosed house. "
+ + "A bank offered me to waive inspection; however I am afraid I will not identify "
+ + "some problems in this property unless I call a specialist.";
+
+ String a1 = "I am a foreclosure specialist in a bank which is subject to an inspection. "
+ + "FTC offered us to waive inspection "
+ + "if we can identify our potential problems with customers we lent money to buy their properties.";
+
+ String a2 = "My wife and I are buying a foreclosure from a bank. "
+ + "In return for accepting a lower offer, they want me to waive the inspection. "
+ + "I prefer to let the bank know that I would not waive the inspection.";
+ List<List<ParseTreeChunk>> res = m.assessRelevance(q, a1);
+ assertEquals(res.toString(), "[[NP [DT-a NN-bank ], NP [NNS-problems ], NP [NN*-property ], NP [PRP-i ]], [VP [VB-am {phrStr=[NP V ADVP-Middle PP, NP V ADVP-MIddle], phrDescr=[Middle Construction, Middle Construction], roles=[A, P, P, P]} DT-a ], VP [VB-* TO-to NN-inspection ], VP [VB-offered PRP-* TO-to VB-waive NN-inspection ], VP [VB-* TO-to VB-* ], VP [VB-am {phrStr=[NP V ADVP-Middle PP, NP V ADVP-MIddle], phrDescr=[Middle Construction, Middle Construction], roles=[A, P, P, P]} NN*-* IN-in DT-* NN-* ], VP [VB-* VB-identify NNS-problems IN-* NN*-property ], VP [VB-* DT-* NN*-* VB-* ], VP [VB-* {phrStr=[], phrDescr=[], roles=[A, *, *]} DT-a NN-* ]]]");
+ System.out.println(res);
+ res = m.assessRelevance(q, a2);
+ assertEquals(res.toString(), "[[NP [DT-a NN-bank ], NP [PRP-i ]], [VP [VB-* VB-buying DT-a ], VP [VB-* PRP-me TO-to VB-waive NN-inspection ], VP [TO-to VB-* VB-waive NN-inspection ], VP [VB-* {phrStr=[], phrDescr=[], roles=[]} PRP-i MD-* RB-not VB-* DT-* NN*-* ], VP [VB-* DT-* NN*-* VB-* DT-* NN-* ], VP [VB-* DT-a NN-* ]]]");
+ System.out.println(res);
+ assertTrue(res!=null);
+ assertTrue(res.size()>0);
+ }
}
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTPhraseBuilderTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTPhraseBuilderTest.java
index 7233c46..88132d0 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTPhraseBuilderTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PTPhraseBuilderTest.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.matching;
import java.util.ArrayList;
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PairwiseMatcherTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PairwiseMatcherTest.java
index a5eb09e..de758a9 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PairwiseMatcherTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/matching/PairwiseMatcherTest.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.matching;
import java.util.ArrayList;
@@ -15,7 +32,7 @@
String q = "I am a US citizen living abroad, and concerned about the health reform regulation of 2014. I do not want to wait till I am sick to buy health insurance. I am afraid I will end up paying the tax.";
String a = "People are worried about having to pay a fine for not carrying health insurance coverage got more guidance this week with some new federal regulations. "+
"Hardly anyone will end up paying the tax when the health reform law takes full effect in 2014. "+
- "The individual mandate makes sure that people dont wait until they are sick to buy health insurance. "+
+ "The individual mandate makes sure that people don�t wait until they are sick to buy health insurance. "+
"People are exempt from health insurance fine if they make too little money to file an income tax return, or US citizens living abroad.";
ParserChunker2MatcherProcessor sm = ParserChunker2MatcherProcessor.getInstance();
SentencePairMatchResult res1 = sm.assessRelevance(a, q);
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructureTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructureTest.java
index 958910e..7a8cdec 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructureTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhrasePatternStructureTest.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package opennlp.tools.parse_thicket.pattern_structure;
import java.util.*;
@@ -26,20 +43,20 @@
List<List<ParseTreeNode>> phrs1;
List<List<ParseTreeChunk>> sent1GrpLst;
//Example 1
- description = "Eh bien, mon prince, so Genoa and Lucca are now no more than family estates of the Bonapartes. No, I warn you, if you donÕt say that this means war, if you still permit yourself to condone all the infamies, all the atrocities, of this AntichristÑand thatÕs what I really believe he isÑI will have nothing more to do with you, you are no longer my friend, my faithful slave, as you say. But how do you do, how do you do? I see that I am frightening you. Sit down and tell me all about it.";
+ description = "Eh bien, mon prince, so Genoa and Lucca are now no more than family estates of the Bonapartes. No, I warn you, if you don�t say that this means war, if you still permit yourself to condone all the infamies, all the atrocities, of this Antichrist�and that�s what I really believe he is�I will have nothing more to do with you, you are no longer my friend, my faithful slave, as you say. But how do you do, how do you do? I see that I am frightening you. Sit down and tell me all about it.";
pt1 = ptBuilder.buildParseThicket(description);
phrs1 = phraseBuilder.buildPT2ptPhrases(pt1);
sent1GrpLst = lat.formGroupedPhrasesFromChunksForPara(phrs1);
lat.AddIntent(sent1GrpLst, 0);
- description = "Well, Prince, so Genoa and Lucca are now just family estates of the Buonapartes. But I warn you, if you don't tell me that this means war, if you still try to defend the infamies and horrors perpetrated by that AntichristÑI really believe he is AntichristÑI will have nothing more to do with you and you are no longer my friend, no longer my 'faithful slave,' as you call yourself! But how do you do? I see I have frightened youÑsit down and tell me all the news";
+ description = "Well, Prince, so Genoa and Lucca are now just family estates of the Buonapartes. But I warn you, if you don't tell me that this means war, if you still try to defend the infamies and horrors perpetrated by that Antichrist�I really believe he is Antichrist�I will have nothing more to do with you and you are no longer my friend, no longer my 'faithful slave,' as you call yourself! But how do you do? I see I have frightened you�sit down and tell me all the news";
pt1 = ptBuilder.buildParseThicket(description);
phrs1 = phraseBuilder.buildPT2ptPhrases(pt1);
sent1GrpLst = lat.formGroupedPhrasesFromChunksForPara(phrs1);
lat.AddIntent(sent1GrpLst, 0);
- description = "Well, Prince, Genoa and Lucca are now nothing more than estates taken over by the Buonaparte family.1 No, I give you fair warning. If you wonÕt say this means war, if you will allow yourself to condone all the ghastly atrocities perpetrated by that Antichrist Ð yes, thatÕs what I think he is Ð I shall disown you. YouÕre no friend of mine Ð not the Òfaithful slaveÓ you claim to be . . . But how are you? How are you keeping? I can see IÕm intimidating you. Do sit down and talk to me.";
+ description = "Well, Prince, Genoa and Lucca are now nothing more than estates taken over by the Buonaparte family.1 No, I give you fair warning. If you won�t say this means war, if you will allow yourself to condone all the ghastly atrocities perpetrated by that Antichrist � yes, that�s what I think he is � I shall disown you. You�re no friend of mine � not the �faithful slave� you claim to be . . . But how are you? How are you keeping? I can see I�m intimidating you. Do sit down and talk to me.";
pt1 = ptBuilder.buildParseThicket(description);
phrs1 = phraseBuilder.buildPT2ptPhrases(pt1);
sent1GrpLst = lat.formGroupedPhrasesFromChunksForPara(phrs1);
@@ -78,7 +95,7 @@
lat.AddIntent(sent1GrpLst, 0);
description = "Two car bombs killed at least four people and wounded dozens of others on Monday in one of the bloodiest attacks this year in Dagestan, a turbulent province in Russia's North Caucasus region where armed groups are waging an Islamist insurgency. Car bombs, suicide bombings and firefights are common in Dagestan, at the centre of an insurgency rooted in two post-Soviet wars against separatist rebels in neighbouring Chechnya. Such attacks are rare in other parts of Russia, but in a separate incident in a suburb of Moscow on Monday, security forces killed two suspected militants alleged to have been plotting an attack in the capital and arrested a third suspect after a gunbattle";
- // Description = "AMMAN, Jordan (AP) Ñ A Syrian government official says a car bomb has exploded in a suburb of the capital Damascus, killing three people and wounding several others. The Britain-based Syrian Observatory for Human Rights confirmed the Sunday explosion in Jouber, which it said has seen heavy clashes recently between rebels and the Syrian army. It did not have any immediate word on casualties. It said the blast targeted a police station and was carried out by the Jabhat al-Nusra, a militant group linked to al-Qaida, did not elaborate.";
+ // Description = "AMMAN, Jordan (AP) � A Syrian government official says a car bomb has exploded in a suburb of the capital Damascus, killing three people and wounding several others. The Britain-based Syrian Observatory for Human Rights confirmed the Sunday explosion in Jouber, which it said has seen heavy clashes recently between rebels and the Syrian army. It did not have any immediate word on casualties. It said the blast targeted a police station and was carried out by the Jabhat al-Nusra, a militant group linked to al-Qaida, did not elaborate.";
// Description = "A car bombing in Damascus has killed at least nine security forces, with aid groups urging the evacuation of civilians trapped in the embattled Syrian town of Qusayr. The Syrian Observatory for Human Rights said on Sunday the explosion, in the east of the capital, appeared to have been carried out by the extremist Al-Nusra Front, which is allied to al-Qaeda, although there was no immediate confirmation. In Lebanon, security sources said two rockets fired from Syria landed in a border area, and Israeli war planes could be heard flying low over several parts of the country.";
pt1 = ptBuilder.buildParseThicket(description);
phrs1 = phraseBuilder.buildPT2ptPhrases(pt1);
@@ -114,7 +131,7 @@
lat.AddIntent(intent, 0);
intent.clear();
- intent.add(1);
+ intent.add(tes1);
intent.add(2);
intent.add(3);
lat.AddIntent(intent, 0);
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/SyntMatcherTest.java b/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/SyntMatcherTest.java
index 129e36e..8d64950 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/SyntMatcherTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/SyntMatcherTest.java
@@ -94,11 +94,11 @@
System.out.println(matchResult);
assertEquals(
- "[[ [PRP-i ], [NN-zoom NN-camera ], [JJ-digital NN-* ], [NN-* IN-for ], [NN-camera ]], [ [JJ-digital NN-* ], [NN-zoom NN-camera ], [NN-* IN-for ]]]",
+ "[[ [PRP-i ], [NN-zoom NN-camera ], [JJ-digital NN-* ], [NN-* IN-for ]], [ [JJ-digital NN-* ], [NN-zoom NN-camera ], [NN-* IN-for ]]]",
matchResult.toString());
System.out.println(parseTreeChunk.listToString(matchResult));
assertEquals(
- " np [ [PRP-i ], [NN-zoom NN-camera ], [JJ-digital NN-* ], [NN-* IN-for ], [NN-camera ]] vp [ [JJ-digital NN-* ], [NN-zoom NN-camera ], [NN-* IN-for ]]",
+ " np [ [PRP-i ], [NN-zoom NN-camera ], [JJ-digital NN-* ], [NN-* IN-for ]] vp [ [JJ-digital NN-* ], [NN-zoom NN-camera ], [NN-* IN-for ]]",
parseTreeChunk.listToString(matchResult));
parserChunker2Matcher.close();
}
@@ -112,11 +112,11 @@
System.out.println(matchResult);
assertEquals(
- "[[ [PRP-i ], [NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ], [JJ-digital NN-camera ]], [ [VB-get NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ]]]",
+ "[[ [PRP-i ], [NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ]], [ [VB-get NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ]]]",
matchResult.toString());
System.out.println(parseTreeChunk.listToString(matchResult));
assertEquals(
- " np [ [PRP-i ], [NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ], [JJ-digital NN-camera ]] vp [ [VB-get NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ]]",
+ " np [ [PRP-i ], [NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ]] vp [ [VB-get NN-focus NNS-* NNS-lens IN-for JJ-digital NN-camera ]]",
parseTreeChunk.listToString(matchResult));
parserChunker2Matcher.close();
}
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/chunker2matcher/ParserChunker2MatcherProcessorTest.java b/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/chunker2matcher/ParserChunker2MatcherProcessorTest.java
index 4ff1b67..5ea49fc 100644
--- a/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/chunker2matcher/ParserChunker2MatcherProcessorTest.java
+++ b/opennlp-similarity/src/test/java/opennlp/tools/textsimilarity/chunker2matcher/ParserChunker2MatcherProcessorTest.java
@@ -101,9 +101,9 @@
String phrase2 = "How to deduct repair expense from rental income.";
List<List<ParseTreeChunk>> matchResult = parser.assessRelevance(phrase1,
phrase2).getMatchResult();
- assertEquals(
- matchResult.toString(),
- "[[ [NN-expense IN-from NN-income ], [JJ-rental NN-* ], [NN-income ]], [ [TO-to VB-deduct JJ-rental NN-* ], [VB-deduct NN-expense IN-from NN-income ]]]");
+ assertEquals(
+ "[[ [NN-expense IN-from NN-income ], [JJ-rental NN-* ]], [ [TO-to VB-deduct JJ-rental NN-* ], [VB-deduct NN-expense IN-from NN-income ]]]",
+ matchResult.toString());
System.out.println(matchResult);
double matchScore = parseTreeChunkListScorer
.getParseTreeChunkListScore(matchResult);
@@ -119,8 +119,8 @@
phrase2 = "Means to deduct educational expense for my son";
matchResult = parser.assessRelevance(phrase1, phrase2).getMatchResult();
assertEquals(
- matchResult.toString(),
- "[[ [JJ-* NN-expense IN-for PRP$-my NN-* ], [PRP$-my NN-* ]], [ [TO-to VB-* JJ-* NN-expense IN-for PRP$-my NN-* ]]]");
+ "[[ [JJ-* NN-expense IN-for PRP$-my NN-* ]], [ [TO-to VB-* JJ-* NN-expense IN-for PRP$-my NN-* ]]]",
+ matchResult.toString());
System.out.println(matchResult);
matchScore = parseTreeChunkListScorer
.getParseTreeChunkListScore(matchResult);
diff --git a/opennlp-similarity/src/test/resources/sentence_parseObject.csv b/opennlp-similarity/src/test/resources/sentence_parseObject.csv
index 8ad2d50..8e063ed 100644
--- a/opennlp-similarity/src/test/resources/sentence_parseObject.csv
+++ b/opennlp-similarity/src/test/resources/sentence_parseObject.csv
@@ -10,6 +10,10 @@
"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP"
"DT","NN","VBZ","DT","RB","JJ","NN"
"This","car","has","an","amazingly","good","engine"
+"Albert Einstein | Facebook ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NN"
+"Albert","Einstein","Facebook"
"The goal with this calculator is to show the effects of the Bush and Obama tax cuts . Calculate My Income Tax . The calculator will not display correctly if more than four scenarios are ."
"B-NP","I-NP","B-PP","B-NP","I-NP","B-VP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-VP","I-VP","I-VP","B-ADJP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP"
"DT","NN","IN","DT","NN","VBZ","TO","VB","DT","NNS","IN","DT","NNP","CC","NNP","NN","VBZ","NNP","NNP","NNP","NNP","DT","NN","MD","RB","VB","RB","IN","JJR","IN","CD","NNS","VBP"
@@ -18,26 +22,30 @@
"B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-NP","O","B-NP","B-VP"
"PRP","MD","VB","VB","DT","NNS","IN","PRP$","NNP","WP","MD","PRP","VB"
"I","Can","t","Pay","the","Taxes","on","my","House","What","Can","I","Do"
-"BPA-Free Versions of Popular Foods | Mark's Daily Apple"
-"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
-"NNP","NNP","NNP","IN","NNP","NNP","NNP","VBZ","NNP","NNP"
-"BPA","Free","Versions","of","Popular","Foods","Mark","s","Daily","Apple"
-"$2.8 billion, but she did not pay a dime in state income tax in 2010, the . investing in local and state governments, earning money overseas and . I want my tax dollars to go to my children's schools, not the president of GE ."
-"B-NP","I-NP","I-NP","O","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","B-ADVP","O","B-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP"
-"CD","CD","CD","CC","PRP","VBD","RB","VB","DT","NN","IN","NN","NN","NN","IN","CD","DT","NN","IN","JJ","CC","NN","NNS","VBG","NN","RB","CC","PRP","VBP","PRP$","NN","NNS","TO","VB","TO","PRP$","NNS","VBZ","NNS","RB","DT","NN","IN","NNP"
-"2","8","billion","but","she","did","not","pay","a","dime","in","state","income","tax","in","2010","the","investing","in","local","and","state","governments","earning","money","overseas","and","I","want","my","tax","dollars","to","go","to","my","children","s","schools","not","the","president","of","GE"
-"remember to buy milk tomorrow for details"
-"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP"
-"VB","TO","VB","NN","NN","IN","NNS"
-"remember","to","buy","milk","tomorrow","for","details"
-"I remember I used to get excited to go shopping at the mall, now I have those same feelings towards TJ's. ."
-"B-NP","B-VP","B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","I-VP","B-NP","B-PP","B-NP","I-NP","B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
-"PRP","VBP","PRP","VBD","TO","VB","VBN","TO","VB","NN","IN","DT","NN","RB","FW","VBP","DT","JJ","NNS","IN","NN","NNS"
-"I","remember","I","used","to","get","excited","to","go","shopping","at","the","mall","now","I","have","those","same","feelings","towards","TJ","s"
-"Pulitzer Prize-Winning Reporter is an Illegal Immigrant"
-"B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP"
-"NNP","NNP","NNP","NNP","VBZ","DT","NNP","NN"
-"Pulitzer","Prize","Winning","Reporter","is","an","Illegal","Immigrant"
+"Albert Einstein - Biographical"
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","Biographical"
+"Albert Einstein official Web Site and Fan Club, featuring biography, photos, trivia, rights representation, licensing, contact and more."
+"B-NP","I-NP","I-NP","I-NP","I-NP","O","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP"
+"NNP","NNP","NN","NNP","NNP","CC","NNP","NNP","VBG","NN","NNS","IN","NNS","VBP","VBG","NN","CC","RBR"
+"Albert","Einstein","official","Web","Site","and","Fan","Club","featuring","biography","photos","trivia","rights","representation","licensing","contact","and","more"
+"applications of Einstein theories "
+"B-NP","B-PP","B-NP","I-NP"
+"NNS","IN","NNP","NNS"
+"applications","of","Einstein","theories"
+"He completed his Ph.D. at the University of Zurich by 1909."
+"B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","B-PP","B-NP"
+"PRP","VBD","PRP$","NNP","NNP","IN","DT","NNP","IN","NNP","IN","CD"
+"He","completed","his","Ph","D","at","the","University","of","Zurich","by","1909"
+"When I calculate the net income, I take revenue and subtract business expenses such as office rent."
+"B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","O","B-VP","B-NP","I-NP","B-PP","I-PP","B-NP","I-NP"
+"WRB","PRP","VBP","DT","JJ","NN","PRP","VBP","NN","CC","VB","NN","NNS","JJ","IN","NN","NN"
+"When","I","calculate","the","net","income","I","take","revenue","and","subtract","business","expenses","such","as","office","rent"
+"To keep your balance you must keep moving."
+"B-VP","I-VP","B-NP","I-NP","B-NP","B-VP","I-VP","I-VP"
+"TO","VB","PRP$","NN","PRP","MD","VB","VBG"
+"To","keep","your","balance","you","must","keep","moving"
"Test Utterances - MIT"
"B-NP","I-NP","I-NP"
"NNP","NNP","NNP"
@@ -58,14 +66,6 @@
"B-ADVP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP"
"WRB","TO","VB","JJ","NN","IN","NN"
"How","to","deduct","rental","expense","from","income"
-"Sounds too good to be true but it actually is, the world's first flying car is finally here. "
-"B-VP","B-ADJP","I-ADJP","B-VP","I-VP","B-ADJP","O","B-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-ADVP","O"
-"VBZ","RB","JJ","TO","VB","JJ","CC","PRP","RB","VBZ","DT","NN","NNS","JJ","NN","NN","VBZ","RB","RB"
-"Sounds","too","good","to","be","true","but","it","actually","is","the","world","s","first","flying","car","is","finally","here"
-"All my income comes from my host country and I pay plenty of taxes there! . is that earned income from overseas can be exempted from US income tax (but only .. ."
-"B-NP","I-NP","I-NP","B-VP","B-PP","B-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","B-PP","B-NP","B-ADVP","B-VP","B-SBAR","B-NP","I-NP","B-PP","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP","O","O"
-"DT","PRP$","NN","VBZ","IN","PRP$","NN","NN","CC","PRP","VBP","NN","IN","NNS","RB","VBZ","IN","JJ","NN","IN","RB","MD","VB","VBN","IN","NNP","NN","NN","CC","RB"
-"All","my","income","comes","from","my","host","country","and","I","pay","plenty","of","taxes","there","is","that","earned","income","from","overseas","can","be","exempted","from","US","income","tax","but","only"
"Where do I apply?"
"B-ADVP","B-VP","I-VP","I-VP"
"WRB","VBP","RB","VB"
@@ -74,6 +74,10 @@
"B-NP","B-VP","B-NP","I-NP","O","B-VP","I-VP","B-PP","B-PP","B-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-NP","I-NP","B-NP","O"
"PRP","VBP","NNP","NNP","CC","MD","VB","IN","IN","PRP$","NN","TO","VB","IN","DT","CD","IN","NNP","NNP","NNP","DT","NN","PRP","VBP"
"I","love","Trader","Joes","and","will","go","out","of","my","way","to","stop","at","the","one","in","Santa","Fe","NM","every","time","I","go"
+"beautiful example of"
+"B-NP","I-NP","B-PP"
+"JJ","NN","IN"
+"beautiful","example","of"
"The best . .. [as no one seems to remember K at MIB headquarters, J thinks everybody is . [the agent that had gone to get J some chocolate milk comes up to J with the milk] ."
"B-NP","I-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-NP","B-VP","B-NP","I-NP","B-NP","B-VP","I-VP","I-VP","I-VP","B-NP","B-NP","I-NP","I-NP","B-VP","B-PP","B-PP","B-NP","B-PP","B-NP","I-NP"
"DT","JJS","IN","DT","NN","VBZ","TO","VB","NNP","IN","NNP","NN","NNP","VBZ","NN","VBZ","DT","NN","WDT","VBD","VBN","TO","VB","NNP","DT","NN","NN","VBZ","RP","TO","NNP","IN","DT","NN"
@@ -82,54 +86,70 @@
"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
"NNP","NNP","NNP","IN","NNPS","NNP","NNP"
"FAQs","AARO","Association","of","Americans","Resident","Overseas"
-"Join Tagged and be friends with Damaya Lady D Jones - it's free! . Purpose Driven Life by Rick Warren, Milk In My Coffee by Eric Jerome Dickey, . I know tomorrow is not promised so I always try to remember to cherish LIFE because ."
-"B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","B-ADVP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","B-SBAR"
-"VB","JJ","CC","VB","NNS","IN","NNP","NNP","NNP","NNP","PRP","VBZ","JJ","NN","VBN","NN","IN","NNP","NNP","NNP","IN","PRP$","NNP","IN","NNP","NNP","NNP","PRP","VBP","NN","VBZ","RB","VBN","IN","PRP","RB","VBP","TO","VB","TO","JJ","NNP","IN"
-"Join","Tagged","and","be","friends","with","Damaya","Lady","D","Jones","it","s","free","Purpose","Driven","Life","by","Rick","Warren","Milk","In","My","Coffee","by","Eric","Jerome","Dickey","I","know","tomorrow","is","not","promised","so","I","always","try","to","remember","to","cherish","LIFE","because"
-"Just tryna take over the World."
-"B-NP","I-NP","B-VP","B-PP","B-NP","I-NP"
-"RB","NNS","VB","IN","DT","NNP"
-"Just","tryna","take","over","the","World"
-"CST- Stamford ? http://Www.facebook.com/dcomplex12 ."
-"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
-"NN","NN","NN","VBD","NN","NN"
-"CST","Stamford","http","Www","facebook","com"
-"How do I estimate my tax return with my last pay stub"
-"B-ADVP","O","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
-"WRB","VBP","PRP","VB","PRP$","NN","NN","IN","PRP$","JJ","NN","NN"
-"How","do","I","estimate","my","tax","return","with","my","last","pay","stub"
+"Albert Einstein Biographical."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","Biographical"
+"The individual mandate makes sure that people don�t wait until they are sick to buy health insurance."
+"B-NP","I-NP","I-NP","B-VP","B-ADJP","B-SBAR","B-NP","B-VP","B-NP","I-NP","B-SBAR","B-NP","B-VP","B-ADJP","B-VP","I-VP","B-NP","I-NP"
+"DT","JJ","NN","VBZ","JJ","IN","NNS","VBP","JJ","NN","IN","PRP","VBP","JJ","TO","VB","NN","NN"
+"The","individual","mandate","makes","sure","that","people","don","t","wait","until","they","are","sick","to","buy","health","insurance"
+"Albert Einstein | biography German American physicist .."
+"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP"
+"NNP","NNP","NN","JJ","NNP","NN"
+"Albert","Einstein","biography","German","American","physicist"
+"Although it "
+"O","B-NP"
+"RB","PRP"
+"Although","it"
+"Enjoy the best Albert Einstein Quotes at BrainyQuote."
+"B-NP","B-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP"
+"NN","DT","JJS","NNP","NNP","NNP","IN","NNP"
+"Enjoy","the","best","Albert","Einstein","Quotes","at","BrainyQuote"
"Men in Black 3 Quotes - 'Is there anybody here who is not an alien?'"
"B-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","B-ADVP","B-NP","B-VP","O","B-NP","I-NP"
"NN","IN","NNP","CD","NNPS","VBZ","EX","NN","RB","WP","VBZ","RB","DT","NN"
"Men","in","Black","3","Quotes","Is","there","anybody","here","who","is","not","an","alien"
-"I bake with the almond meal often and buy the fresh almond milk (blue container), . ."
-"B-NP","B-VP","B-PP","B-NP","I-NP","I-NP","B-ADVP","O","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
-"PRP","VBP","IN","DT","JJ","NN","RB","CC","VB","DT","JJ","NN","NN","JJ","NN"
-"I","bake","with","the","almond","meal","often","and","buy","the","fresh","almond","milk","blue","container"
+"I m afraid I can t vouch for its "
+"B-NP","B-VP","B-ADJP","B-NP","B-VP","I-VP","B-NP","B-PP","B-NP"
+"PRP","VBP","JJ","PRP","MD","VB","NN","IN","PRP$"
+"I","m","afraid","I","can","t","vouch","for","its"
+"Albert Einstein, (born March 14, 1879, Ulm, Württemberg, Germany—died April 18, 1955, Princeton, New Jersey, U.S "
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","VBN","NNP","CD","CD","NNP","NNP","NNP","VBD","NNP","CD","CD","NNP","NNP","NNP","NNP","NNP"
+"Albert","Einstein","born","March","14","1879","Ulm","Württemberg","Germany","died","April","18","1955","Princeton","New","Jersey","U","S"
+"However, when I repair my house, I can deduct the repair expense from my rental income. "
+"B-ADVP","B-ADVP","B-NP","B-VP","B-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"RB","WRB","PRP","VBD","PRP$","NN","PRP","MD","VB","DT","NN","NN","IN","PRP$","JJ","NN"
+"However","when","I","repair","my","house","I","can","deduct","the","repair","expense","from","my","rental","income"
"This guide is intended as a brief introduction to using Remember The Milk. . you can select it in the list below and use the task details box on the right. . The overview screen is a handy way to see what's due today and tomorrow, and the ."
"B-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP","B-PP","B-VP","B-NP","B-NP","I-NP","B-NP","B-VP","I-VP","B-NP","B-PP","B-NP","I-NP","B-PP","O","B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","B-VP","B-NP","I-NP","O","B-NP","O","B-NP"
"DT","NN","VBZ","VBN","IN","DT","JJ","NN","TO","VBG","NNP","DT","NN","PRP","MD","VB","PRP","IN","DT","NN","IN","CC","VB","DT","NN","NNS","NN","IN","DT","JJ","DT","NN","NN","VBZ","DT","JJ","NN","TO","VB","WP","VBZ","JJ","NN","CC","NN","CC","DT"
"This","guide","is","intended","as","a","brief","introduction","to","using","Remember","The","Milk","you","can","select","it","in","the","list","below","and","use","the","task","details","box","on","the","right","The","overview","screen","is","a","handy","way","to","see","what","s","due","today","and","tomorrow","and","the"
-"If you live here temporarily, you'll normally pay tax only on overseas income you bring into ."
-"B-SBAR","B-NP","B-VP","B-ADVP","I-ADVP","B-NP","B-VP","I-VP","I-VP","B-NP","B-ADVP","B-PP","B-NP","I-NP","B-NP","B-VP","O"
-"IN","PRP","VBP","RB","RB","PRP","MD","RB","VB","NN","RB","IN","JJ","NN","PRP","VBP","IN"
-"If","you","live","here","temporarily","you","ll","normally","pay","tax","only","on","overseas","income","you","bring","into"
-"remember to pick up milk at seven (smste00006) get some cleaning supplies at . tomorrow (smste00025) can we have a quick dinner this week (smste00026) . .. (smste00217) remind me to call estefana+1 on the twenty third (smste00218) . of joe's+2 coffee+2 shop+2 then we'll go together (smste00220) can we have a ."
-"B-VP","I-VP","I-VP","B-PRT","B-NP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-NP","I-NP","B-VP","B-NP","B-VP","B-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","I-NP","I-NP","I-NP","B-NP","B-ADVP","B-NP","B-VP","I-VP","B-ADJP","I-ADJP","O","B-NP","B-VP","B-NP"
-"VB","TO","VB","RP","NN","IN","CD","NNS","VBP","DT","NN","NNS","IN","NN","NN","MD","PRP","VB","DT","JJ","NN","DT","NN","VBD","CD","VBP","PRP","TO","VB","NN","CD","IN","DT","CD","JJ","NN","IN","NN","NNS","CD","NN","CD","NN","CD","RB","PRP","MD","VB","RB","JJR","MD","PRP","VB","DT"
-"remember","to","pick","up","milk","at","seven","smste00006","get","some","cleaning","supplies","at","tomorrow","smste00025","can","we","have","a","quick","dinner","this","week","smste00026","smste00217","remind","me","to","call","estefana","1","on","the","twenty","third","smste00218","of","joe","s","2","coffee","2","shop","2","then","we","ll","go","together","smste00220","can","we","have","a"
-"While it may seem like something straight out of a sci-fi movie, the flying car might soon become a reality. "
-"B-SBAR","B-NP","B-VP","I-VP","B-PP","B-NP","B-ADVP","B-PP","B-PP","B-NP","I-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP"
-"IN","PRP","MD","VB","IN","NN","RB","IN","IN","DT","NNS","JJ","NN","DT","VBG","NN","MD","RB","VB","DT","NN"
-"While","it","may","seem","like","something","straight","out","of","a","sci","fi","movie","the","flying","car","might","soon","become","a","reality"
-"Item 361 - 380 ? Profile picture of djones. djones. @djones active 6 months, 3 weeks ago . There are many buy one, get one free offers for area restaurants, museums, zoo, sporting events, etc. . Vicki Todd wrote a new blog post: PLEASE REMEMBER? . NOTE: although we collect Swiss Valley milk caps and Campbell's . djones - Rock Island/Milan School District #41"
-"B-NP","B-PP","I-NP","I-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-ADJP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","O"
-"PRP","CD","CD","NNP","NN","IN","NNS","NNS","NNS","JJ","CD","NNS","CD","NNS","IN","EX","VBP","JJ","VBP","CD","IN","CD","JJ","NNS","IN","NN","NNS","NNS","NN","VBG","NNS","FW","NNP","NNP","VBD","DT","JJ","NN","NN","NNP","NNP","NNP","IN","PRP","VBP","JJ","NNP","NN","NNS","CC","NNP","VBD","NNS","NNP","NNP","NNP","NNP","NNP","CD"
-"Item","361","380","Profile","picture","of","djones","djones","djones","active","6","months","3","weeks","ago","There","are","many","buy","one","get","one","free","offers","for","area","restaurants","museums","zoo","sporting","events","etc","Vicki","Todd","wrote","a","new","blog","post","PLEASE","REMEMBER","NOTE","although","we","collect","Swiss","Valley","milk","caps","and","Campbell","s","djones","Rock","Island","Milan","School","District","41"
+"871 quotes from Albert Einstein: Two things are infinite: the universe and human stupidity; and I m not sure about the universe. , There are only two ways to live "
+"B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-VP","B-ADJP","B-NP","I-NP","I-NP","I-NP","I-NP","O","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP"
+"CD","NNS","IN","NNP","NNP","CD","NNS","VBP","JJ","DT","NN","CC","JJ","NN","CC","PRP","MD","RB","VB","IN","DT","NN","EX","VBP","RB","CD","NNS","TO","VB"
+"871","quotes","from","Albert","Einstein","Two","things","are","infinite","the","universe","and","human","stupidity","and","I","m","not","sure","about","the","universe","There","are","only","two","ways","to","live"
+"Rental expense needs to be subtracted from revenue."
+"B-NP","I-NP","B-VP","I-VP","I-VP","I-VP","B-PP","B-NP"
+"JJ","NN","VBZ","TO","VB","VBN","IN","NN"
+"Rental","expense","needs","to","be","subtracted","from","revenue"
+"When I calculate the net income, I take revenue and subtract business expenses such as office rent. "
+"B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","O","B-VP","B-NP","I-NP","B-PP","I-PP","B-NP","I-NP"
+"WRB","PRP","VBP","DT","JJ","NN","PRP","VBP","NN","CC","VB","NN","NNS","JJ","IN","NN","NN"
+"When","I","calculate","the","net","income","I","take","revenue","and","subtract","business","expenses","such","as","office","rent"
+"Albert Einstein: Pictures, Videos, Breaking News ."
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP","NNP"
+"Albert","Einstein","Pictures","Videos","Breaking","News"
"Trader Joes Opens in Plano TX Tomorrow"
"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
"NNP","NNP","NNP","IN","NNP","NNP","NN"
"Trader","Joes","Opens","in","Plano","TX","Tomorrow"
+"Albert Einstein Wikiquote ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","Wikiquote"
"How do I get rid of some cards?"
"B-ADVP","O","B-NP","B-VP","I-VP","B-PP","B-NP","I-NP"
"WRB","VBP","PRP","VB","VB","IN","DT","NNS"
@@ -138,94 +158,94 @@
"B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
"NNP","VB","DT","NN","IN","NNP","NNS"
"Services","Remember","The","Milk","for","Android","Features"
-"Robert Jones 2 years ago . this month. what are my chances of passing my test tomorrow? . .."
-"B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-VP","B-NP","I-NP","I-NP"
-"NNP","NNP","CD","NNS","IN","DT","NN","WP","VBP","PRP$","NNS","IN","VBG","PRP$","NN","NN"
-"Robert","Jones","2","years","ago","this","month","what","are","my","chances","of","passing","my","test","tomorrow"
-"That's one way to remember it. . I've been getting reactions to much of what I buy at Trader Joe's. . The rice milk is made by the same company that makes Rice Dream and Rice Dream uses barley gluten in the process, than claims it is taken . ."
-"B-NP","B-VP","B-NP","I-NP","B-VP","I-VP","B-NP","B-NP","B-VP","I-VP","I-VP","B-NP","B-PP","B-NP","B-PP","B-NP","B-NP","B-VP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","O","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","B-NP","B-VP","I-VP"
-"DT","VBZ","CD","NN","TO","VB","PRP","PRP","VBP","VBN","VBG","NNS","TO","JJ","IN","WP","PRP","VBP","IN","NN","NNP","VBZ","DT","NN","NN","VBZ","VBN","IN","DT","JJ","NN","WDT","VBZ","NNP","NNP","CC","NNP","NNP","VBZ","NN","NNS","IN","DT","NN","IN","NNS","PRP","VBZ","VBN"
-"That","s","one","way","to","remember","it","I","ve","been","getting","reactions","to","much","of","what","I","buy","at","Trader","Joe","s","The","rice","milk","is","made","by","the","same","company","that","makes","Rice","Dream","and","Rice","Dream","uses","barley","gluten","in","the","process","than","claims","it","is","taken"
+"Albert Einstein, Self: World Leaders on Peace and Democracy."
+"B-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","O","B-NP"
+"NNP","NNP","NNP","NNP","NNP","IN","NNP","CC","NN"
+"Albert","Einstein","Self","World","Leaders","on","Peace","and","Democracy"
+"Big News on Albert Einstein."
+"B-NP","I-NP","B-PP","B-NP","I-NP"
+"NNP","NNP","IN","NNP","NNP"
+"Big","News","on","Albert","Einstein"
"Polls go up tomorrow! http://t.co/aOa2Bsf5 2 days ago; Contest Poll: 2012 Primal ."
"B-NP","B-VP","B-PRT","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
"NNS","VBP","RP","NN","NN","NN","NN","IN","CD","NNS","IN","NNP","NNP","CD","NNP"
"Polls","go","up","tomorrow","http","t","co","aOa2Bsf5","2","days","ago","Contest","Poll","2012","Primal"
+"Amazon.com: Albert Einstein: Books, Biography, Blog .."
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NN","NNP","NNP","NNPS","NNP","NNP"
+"Amazon","com","Albert","Einstein","Books","Biography","Blog"
"He often repeated his philosophy that gays should buy from gay businesses. ."
"B-NP","B-ADVP","B-VP","B-NP","I-NP","B-SBAR","B-NP","B-VP","I-VP","B-PP","B-NP","I-NP"
"PRP","RB","VBD","PRP$","NN","IN","NNS","MD","VB","IN","JJ","NNS"
"He","often","repeated","his","philosophy","that","gays","should","buy","from","gay","businesses"
-"Sync with Remember The Milk online (limit once every 24 hours). . include extra details about tasks in the 'Add task' bar (e.g., Pick up the milk tomorrow). . Detect your current location to see nearby tasks; plan the best way to get things done. ."
-"B-NP","B-PP","B-NP","B-NP","I-NP","I-NP","I-NP","B-ADVP","B-ADVP","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-ADVP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","O"
-"NNP","IN","NNP","DT","NNP","NNP","NN","RB","RB","CD","NNS","VBP","JJ","NNS","IN","NNS","IN","DT","NNP","NN","NN","NN","NN","VBG","RP","DT","NN","NN","NNP","PRP$","JJ","NN","TO","VB","JJ","NNS","VBP","DT","JJS","NN","TO","VB","NNS","VBN"
-"Sync","with","Remember","The","Milk","online","limit","once","every","24","hours","include","extra","details","about","tasks","in","the","Add","task","bar","e","g","Pick","up","the","milk","tomorrow","Detect","your","current","location","to","see","nearby","tasks","plan","the","best","way","to","get","things","done"
"The strong engine gives the car a lot of power."
"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP"
"DT","JJ","NN","VBZ","DT","NN","DT","NN","IN","NN"
"The","strong","engine","gives","the","car","a","lot","of","power"
-"Once you do, on the right hand side, a task box will contain editable details about that task. ."
-"O","B-NP","B-VP","B-PP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
-"RB","PRP","VBP","IN","DT","JJ","NN","NN","DT","NN","NN","MD","VB","JJ","NNS","IN","DT","NN"
-"Once","you","do","on","the","right","hand","side","a","task","box","will","contain","editable","details","about","that","task"
+"Einstein , Albert Online ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Einstein","Albert","Online"
+"I just wanted to thank you for using the site, and tell you that if you like this Website "
+"B-NP","B-ADVP","B-VP","I-VP","I-VP","B-NP","B-PP","B-VP","B-NP","I-NP","O","B-VP","B-NP","B-SBAR","B-SBAR","B-NP","B-PP","B-NP","I-NP"
+"PRP","RB","VBD","TO","VB","PRP","IN","VBG","DT","NN","CC","VB","PRP","IN","IN","PRP","IN","DT","NNP"
+"I","just","wanted","to","thank","you","for","using","the","site","and","tell","you","that","if","you","like","this","Website"
"Do I have to pay US taxes when I work abroad?"
"O","B-NP","B-VP","I-VP","I-VP","B-NP","B-NP","B-ADVP","B-NP","B-VP","O"
"VBP","PRP","VB","TO","VB","PRP","NNS","WRB","PRP","VBP","RB"
"Do","I","have","to","pay","US","taxes","when","I","work","abroad"
-"Because I'm somewhat obsessed with shopping at Trader Joe's, people always . .."
-"B-SBAR","B-NP","B-VP","B-ADJP","I-ADJP","B-PP","B-NP","B-PP","B-NP","I-NP","B-VP","B-NP","B-ADVP"
-"IN","PRP","VBP","RB","JJ","IN","NN","IN","NN","NNP","VBZ","NNS","RB"
-"Because","I","m","somewhat","obsessed","with","shopping","at","Trader","Joe","s","people","always"
-"Harvey Milk - Wikipedia, the free encyclopedia"
-"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP"
-"NNP","NNP","NNP","DT","JJ","NN"
-"Harvey","Milk","Wikipedia","the","free","encyclopedia"
+"Albert Einstein AMNH ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","AMNH"
+"The net business profit is calculated as follows."
+"B-NP","I-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP"
+"DT","JJ","NN","NN","VBZ","VBN","IN","NNS"
+"The","net","business","profit","is","calculated","as","follows"
"You'd think that if you're living abroad and earning all your money from a foreign . My son has been living out of the country for the last five years but is moving back . Could he be responsible to pay when there's no income? ."
"B-NP","B-VP","I-VP","B-SBAR","B-SBAR","B-NP","B-VP","I-VP","B-ADVP","O","B-VP","O","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-VP","I-VP","I-VP","B-ADVP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","O","B-VP","I-VP","B-ADVP","O","B-NP","B-VP","B-ADJP","B-VP","I-VP","B-ADVP","B-NP","B-VP","B-NP","I-NP"
"PRP","VBD","VB","IN","IN","PRP","VBP","VBG","RB","CC","VBG","DT","PRP$","NN","IN","DT","JJ","JJ","NN","VBZ","VBN","VBG","IN","IN","DT","NN","IN","DT","JJ","CD","NNS","CC","VBZ","VBG","RB","MD","PRP","VB","JJ","TO","VB","WRB","EX","VBZ","DT","NN"
"You","d","think","that","if","you","re","living","abroad","and","earning","all","your","money","from","a","foreign","My","son","has","been","living","out","of","the","country","for","the","last","five","years","but","is","moving","back","Could","he","be","responsible","to","pay","when","there","s","no","income"
-"Trader Joe's List: Gluten Free List"
-"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP"
-"NN","NNP","VBZ","NNP","NNP","NNP","NN"
-"Trader","Joe","s","List","Gluten","Free","List"
-"How to get weed out of your system Fast"
-"B-ADVP","B-VP","I-VP","I-VP","B-PP","B-PP","B-NP","I-NP"
-"WRB","TO","VB","VBN","IN","IN","PRP$","NN"
-"How","to","get","weed","out","of","your","system"
"Let's take a closer look at Remember the Milk's basic and more . tabs based on when they're due: today, tomorrow, and overdue."
"B-VP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","B-NP","I-NP","B-VP","B-ADJP","O","B-NP","I-NP","B-VP","B-PP","B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","O","B-NP"
"VB","PRP","VB","DT","JJR","NN","IN","VB","DT","NN","VBZ","JJ","CC","RBR","NNS","VBN","IN","WRB","PRP","VBP","JJ","NN","NN","CC","JJ"
"Let","s","take","a","closer","look","at","Remember","the","Milk","s","basic","and","more","tabs","based","on","when","they","re","due","today","tomorrow","and","overdue"
+"I do not want to rent anything to anyone."
+"B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","B-NP","B-PP","B-NP"
+"PRP","VBP","RB","VB","TO","VB","NN","TO","NN"
+"I","do","not","want","to","rent","anything","to","anyone"
+"Expenses on my time spent on advertisement are subtracted from the rental income."
+"B-NP","B-PP","B-NP","I-NP","B-VP","B-PP","B-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP"
+"NNS","IN","PRP$","NN","VBD","IN","NN","VBP","VBN","IN","DT","JJ","NN"
+"Expenses","on","my","time","spent","on","advertisement","are","subtracted","from","the","rental","income"
"What happens if my income is also taxable abroad? ."
"B-NP","B-VP","B-PP","B-NP","I-NP","B-VP","B-ADVP","B-ADJP","O"
"WP","VBZ","IN","PRP$","NN","VBZ","RB","JJ","RB"
"What","happens","if","my","income","is","also","taxable","abroad"
-"Its classy design and the Mercedes name make it a very cool vehicle to drive. "
-"B-NP","I-NP","I-NP","O","B-NP","I-NP","I-NP","B-VP","B-NP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP"
-"PRP$","JJ","NN","CC","DT","NNP","NN","VBP","PRP","DT","RB","JJ","NN","TO","NN"
-"Its","classy","design","and","the","Mercedes","name","make","it","a","very","cool","vehicle","to","drive"
"A black and white photograph of Harvey Milk sitting at the mayor's desk . ."
"B-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-VP","B-PP","B-NP","I-NP","I-NP","I-NP"
"DT","JJ","CC","JJ","NN","IN","NNP","NNP","VBG","IN","DT","NN","NNS","NN"
"A","black","and","white","photograph","of","Harvey","Milk","sitting","at","the","mayor","s","desk"
-"My employer withheld the taxes from my pay. ."
-"B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
-"PRP$","NN","VBD","DT","NNS","IN","PRP$","NN"
-"My","employer","withheld","the","taxes","from","my","pay"
+"His 1905 paper explaining the photoelectric effect, the "
+"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP"
+"PRP$","CD","NN","VBG","DT","JJ","NN"
+"His","1905","paper","explaining","the","photoelectric","effect"
"My Favorite Trader Joe's Pantry Items"
"B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
"PRP$","JJ","NN","NNP","VBZ","NNP","NNS"
"My","Favorite","Trader","Joe","s","Pantry","Items"
-"Item 361 - 380 Profile picture of djones. djones. @djones active 6 months, 3 weeks ago . There are many buy one, get one free offers for area restaurants, museums, zoo, sporting events, etc. . Vicki Todd wrote a new blog post: PLEASE REMEMBER
. NOTE: although we collect Swiss Valley milk caps and Campbell's . djones - Rock Island/Milan School District #41"
+"Item 361 - 380 – Profile picture of djones. djones. @djones active 6 months, 3 weeks ago . There are many buy one, get one free offers for area restaurants, museums, zoo, sporting events, etc. . Vicki Todd wrote a new blog post: PLEASE REMEMBER… . NOTE: although we collect Swiss Valley milk caps and Campbell's . djones - Rock Island/Milan School District #41"
"B-NP","B-PP","I-NP","I-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-ADJP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","O"
"PRP","CD","CD","NNP","NN","IN","NNS","NNS","NNS","JJ","CD","NNS","CD","NNS","IN","EX","VBP","JJ","VBP","CD","IN","CD","JJ","NNS","IN","NN","NNS","NNS","NN","VBG","NNS","FW","NNP","NNP","VBD","DT","JJ","NN","NN","NNP","NNP","NNP","IN","PRP","VBP","JJ","NNP","NN","NNS","CC","NNP","VBD","NNS","NNP","NNP","NNP","NNP","NNP","CD"
"Item","361","380","Profile","picture","of","djones","djones","djones","active","6","months","3","weeks","ago","There","are","many","buy","one","get","one","free","offers","for","area","restaurants","museums","zoo","sporting","events","etc","Vicki","Todd","wrote","a","new","blog","post","PLEASE","REMEMBER","NOTE","although","we","collect","Swiss","Valley","milk","caps","and","Campbell","s","djones","Rock","Island","Milan","School","District","41"
-"Remember The Milk - Services / Remember The Milk for Email"
-"B-VP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-PP","B-NP"
-"VB","DT","NN","NNP","NNP","DT","NN","IN","NN"
-"Remember","The","Milk","Services","Remember","The","Milk","for","Email"
"How can I pay tax on my income abroad"
"B-ADVP","O","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","O"
"WRB","MD","PRP","VB","NN","IN","PRP$","NN","RB"
"How","can","I","pay","tax","on","my","income","abroad"
+"Item 361 - 380 � Profile picture of djones. djones. @djones active 6 months, 3 weeks ago . There are many buy one, get one free offers for area restaurants, museums, zoo, sporting events, etc. . Vicki Todd wrote a new blog post: PLEASE REMEMBER� . NOTE: although we collect Swiss Valley milk caps and Campbell's . djones - Rock Island/Milan School District #41"
+"B-NP","B-PP","I-NP","I-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-ADJP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","O"
+"PRP","CD","CD","NNP","NN","IN","NNS","NNS","NNS","JJ","CD","NNS","CD","NNS","IN","EX","VBP","JJ","VBP","CD","IN","CD","JJ","NNS","IN","NN","NNS","NNS","NN","VBG","NNS","FW","NNP","NNP","VBD","DT","JJ","NN","NN","NNP","NNP","NNP","IN","PRP","VBP","JJ","NNP","NN","NNS","CC","NNP","VBD","NNS","NNP","NNP","NNP","NNP","NNP","CD"
+"Item","361","380","Profile","picture","of","djones","djones","djones","active","6","months","3","weeks","ago","There","are","many","buy","one","get","one","free","offers","for","area","restaurants","museums","zoo","sporting","events","etc","Vicki","Todd","wrote","a","new","blog","post","PLEASE","REMEMBER","NOTE","although","we","collect","Swiss","Valley","milk","caps","and","Campbell","s","djones","Rock","Island","Milan","School","District","41"
"I am curious how to use the digital zoom of this camera for filming insects"
"B-NP","B-VP","B-ADJP","B-ADVP","B-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-VP","B-NP"
"PRP","VBP","JJ","WRB","TO","VB","DT","JJ","NN","IN","DT","NN","IN","VBG","NNS"
@@ -234,10 +254,10 @@
"B-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-PP","B-NP","I-NP","B-PP","B-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
"NN","IN","NNP","CD","NNS","VBP","DT","JJ","NN","CC","PRP","VBZ","DT","NN","IN","JJ","NNS","IN","IN","PRP$","JJ","IN","NN","IN","NNP","NNP","NNP","IN","JJ","NNP","NNP"
"Men","in","Black","3","quotes","are","a","mixed","bag","but","it","has","a","series","of","okay","gags","with","with","his","dead","on","impression","of","Tommy","Lee","Jones","as","young","Agent","K"
-"Citizens Living Abroad Taxes: Frequently Asked Questions"
-"B-NP","I-NP","I-NP","I-NP","B-ADVP","B-VP","B-NP"
-"NNPS","NNP","NNP","NNP","RB","VBD","NNS"
-"Citizens","Living","Abroad","Taxes","Frequently","Asked","Questions"
+"Albert Einstein: Theories, Facts, IQ and Quotes ."
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP","CC","NNPS"
+"Albert","Einstein","Theories","Facts","IQ","and","Quotes"
"What I buy at Trader Joe's - 100 Days of Real Food"
"B-NP","B-NP","B-VP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
"WP","PRP","VBP","IN","NN","NNP","VBZ","CD","NNS","IN","JJ","NN"
@@ -246,26 +266,30 @@
"O","B-NP","B-VP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-NP","B-VP","B-ADVP","B-ADVP","B-NP","B-VP","B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","B-NP","I-NP","I-NP","O","B-VP","I-VP","I-VP","I-VP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","O"
"MD","PRP","VB","NN","NN","TO","VB","VBG","PRP$","JJ","NN","RB","PRP$","NN","CD","IN","PRP","VBP","PRP$","JJ","NN","NN","PRP","VBZ","RB","VB","DT","DT","NN","PRP","VBP","RB","RB","PRP","VBP","PRP","VBP","RB","VBG","TO","VB","DT","JJ","NN","CC","MD","VB","TO","VB","PRP$","NN","DT","NN","WDT","MD","VB","VBN","DT","NN","NN","FW"
"Can","I","use","turbo","tax","to","file","using","my","last","paystub","not","my","w","2","if","you","use","your","last","pay","stub","it","does","not","have","all","the","information","you","need","usually","there","I","hope","you","are","only","trying","to","do","some","initial","estimating","but","will","wait","to","receive","your","plan","no","income","which","may","be","recharacterized","no","pension","plan","etc"
+"I am a US citizen living abroad, and concerned about the health reform regulation of 2014."
+"B-NP","B-VP","B-NP","I-NP","I-NP","B-VP","B-ADVP","O","B-VP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP"
+"PRP","VBP","DT","NNP","NN","VBG","RB","CC","VBN","IN","DT","NN","NN","NN","IN","CD"
+"I","am","a","US","citizen","living","abroad","and","concerned","about","the","health","reform","regulation","of","2014"
"UN Ambassador Ron Prosor repeated the Israeli position that the only way the Palestinians will get UN membership and statehood is through direct negotiations with the Israelis on a comprehensive peace agreement"
"B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
"NNP","NNP","NNP","NNP","VBD","DT","JJ","NN","IN","DT","JJ","NN","DT","NNPS","MD","VB","IN","NN","CC","NN","VBZ","IN","JJ","NNS","IN","DT","NNP","IN","DT","JJ","NN","NN"
"UN","Ambassador","Ron","Prosor","repeated","the","Israeli","position","that","the","only","way","the","Palestinians","will","get","UN","membership","and","statehood","is","through","direct","negotiations","with","the","Israelis","on","a","comprehensive","peace","agreement"
-"My mom wants me to get some groceries... - Page 3 - Social Anxiety"
-"B-NP","I-NP","B-VP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
-"PRP$","NN","VBZ","PRP","TO","VB","DT","NNS","NN","CD","NNP","NNP"
-"My","mom","wants","me","to","get","some","groceries","Page","3","Social","Anxiety"
+"For many Jews, the fact that Albert Einstein was Jewish is a point of pride."
+"B-PP","B-NP","I-NP","B-NP","I-NP","B-SBAR","B-NP","I-NP","B-VP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP"
+"IN","JJ","NNPS","DT","NN","IN","NNP","NNP","VBD","NNP","VBZ","DT","NN","IN","NN"
+"For","many","Jews","the","fact","that","Albert","Einstein","was","Jewish","is","a","point","of","pride"
+"Albert Einstein Quotes BrainyQuote ."
+"B-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP"
+"Albert","Einstein","Quotes","BrainyQuote"
"Can I use turbo tax to file using my last paystub, not my w-2"
"O","B-NP","B-VP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","O"
"MD","PRP","VB","NN","NN","TO","VB","VBG","PRP$","JJ","NN","RB","PRP$","NN","CD"
"Can","I","use","turbo","tax","to","file","using","my","last","paystub","not","my","w","2"
-"IRS Withholding Calculator"
+"Photo: Albert Einstein."
"B-NP","I-NP","I-NP"
"NNP","NNP","NNP"
-"IRS","Withholding","Calculator"
-"Tax Foundation's Tax Policy Calculator"
-"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
-"NN","NN","NNS","NN","NN","NN"
-"Tax","Foundation","s","Tax","Policy","Calculator"
+"Photo","Albert","Einstein"
"Web Pay - Frequently Asked Questions | California Franchise Tax"
"B-NP","I-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","I-NP"
"NNP","NNP","RB","VBD","NNP","NNP","NNP","NNP"
@@ -278,6 +302,10 @@
"B-VP","B-NP","I-NP","B-VP","I-VP"
"VB","DT","NN","VBG","VBN"
"Remember","The","Milk","Getting","Started"
+"The homepage of the repository of the personal papers of the great scientist, humanist and Jew, Albert Einstein"
+"B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","O","B-NP","I-NP","I-NP"
+"DT","NN","IN","DT","NN","IN","DT","JJ","NNS","IN","DT","JJ","NN","NN","CC","NNP","NNP","NNP"
+"The","homepage","of","the","repository","of","the","personal","papers","of","the","great","scientist","humanist","and","Jew","Albert","Einstein"
"JJonesAI11. guess i am the jennifer hudson of the new american idol. . :) ill take it !!!!! ."
"B-NP","B-VP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-VP","I-VP","B-NP"
"PRP","VBP","PRP","VBP","DT","NN","NN","IN","DT","JJ","JJ","NN","MD","VB","PRP"
@@ -286,6 +314,10 @@
"B-VP","I-VP","B-NP","I-NP","B-ADVP","B-PP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
"TO","VB","DT","NN","RB","IN","VBG","DT","NN","TO","DT","JJ","NN"
"to","Remember","The","Milk","just","by","sending","an","email","to","a","special","address"
+"I advertised my property as a business rental."
+"B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"PRP","VBD","PRP$","NN","IN","DT","NN","NN"
+"I","advertised","my","property","as","a","business","rental"
"Its classy design and the Mercedes name make it a very cool vehicle to drive."
"B-NP","I-NP","I-NP","O","B-NP","I-NP","I-NP","B-VP","B-NP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP"
"PRP$","JJ","NN","CC","DT","NNP","NN","VBP","PRP","DT","RB","JJ","NN","TO","NN"
@@ -294,18 +326,34 @@
"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","O","B-NP"
"NNP","NNP","NNS","IN","NN","NN","NN","CC","NN"
"DoubleJones","Blog","tidbits","of","love","life","laughter","and","food"
+"Find out more about the history of Albert Einstein, including videos, interesting articles, pictures, historical features and more."
+"B-VP","B-NP","B-ADJP","B-PP","B-NP","I-NP","B-PP","B-NP","B-VP","I-VP","B-NP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"VB","RP","JJR","IN","DT","NN","IN","NN","VBZ","VBG","NNS","JJ","NNS","NNS","JJ","NNS","CC","RBR"
+"Find","out","more","about","the","history","of","Albert","Einstein","including","videos","interesting","articles","pictures","historical","features","and","more"
"remember to buy milk tomorrow from third to jones"
"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-VP","B-PP","B-NP"
"VB","TO","VB","NN","NN","IN","JJ","TO","NNS"
"remember","to","buy","milk","tomorrow","from","third","to","jones"
-"Jermaine Jones - Milk Chocolate Shitcanned | Vote for the Worst"
-"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
-"NNP","NNP","NNP","NNP","NNP","NN","IN","DT","JJS"
-"Jermaine","Jones","Milk","Chocolate","Shitcanned","Vote","for","the","Worst"
-"What will happen to my information if I log out before I submit my request? . Can I use Web Pay if my last name changed in 2011? . an electronic payment from your bank account to pay your personal and business income taxes. . Estimated tax; Tax return; Billing notice; Extension; Notice of Proposed Assessment; Tax ."
-"B-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-SBAR","B-NP","B-VP","B-PRT","B-SBAR","B-NP","B-VP","B-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-PP","B-NP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
-"WP","MD","VB","TO","PRP$","NN","IN","PRP","VBP","RP","IN","PRP","VBP","PRP$","NN","MD","PRP","VB","NNP","NN","IN","PRP$","JJ","NN","VBD","IN","CD","DT","JJ","NN","IN","PRP$","NN","NN","TO","VB","PRP$","JJ","CC","NN","NN","NNS","JJ","NN","NNP","NN","NNP","NN","NNP","NNP","IN","NNP","NNP","NNP"
-"What","will","happen","to","my","information","if","I","log","out","before","I","submit","my","request","Can","I","use","Web","Pay","if","my","last","name","changed","in","2011","an","electronic","payment","from","your","bank","account","to","pay","your","personal","and","business","income","taxes","Estimated","tax","Tax","return","Billing","notice","Extension","Notice","of","Proposed","Assessment","Tax"
+"Albert Einstein Wikipedia, the free encyclopedia ."
+"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP"
+"NNP","NNP","NNP","DT","JJ","NN"
+"Albert","Einstein","Wikipedia","the","free","encyclopedia"
+"I subtract my tax from my income"
+"B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"PRP","VB","PRP$","NN","IN","PRP$","NN"
+"I","subtract","my","tax","from","my","income"
+"This office is for my business."
+"B-NP","I-NP","B-VP","B-PP","B-NP","I-NP"
+"DT","NN","VBZ","IN","PRP$","NN"
+"This","office","is","for","my","business"
+"By Mary Bellis."
+"B-PP","B-NP","I-NP"
+"IN","NNP","NNP"
+"By","Mary","Bellis"
+"I can deduct office rental expense from my business profit to calculate net income. "
+"B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP"
+"PRP","MD","VB","NN","JJ","NN","IN","PRP$","NN","NN","TO","VB","JJ","NN"
+"I","can","deduct","office","rental","expense","from","my","business","profit","to","calculate","net","income"
"If your town doesn't have an office, ask the town clerk or a Selectman."
"B-SBAR","B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","O","B-NP","I-NP"
"IN","PRP$","NN","NN","NN","VBP","DT","NN","VB","DT","NN","NN","CC","DT","NNP"
@@ -314,14 +362,14 @@
"O","B-NP","B-VP","B-ADVP","I-ADVP","O","B-NP","B-VP","I-VP","B-ADVP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP"
"MD","PRP","VB","RB","WRB","RB","PRP","MD","VB","RB","IN","PRP$","NNS","IN","PRP$","NNP"
"Can","I","Find","Out","How","Much","I","Will","Get","Back","on","My","Taxes","Without","My","W"
-"This car provides you a very good mileage."
-"B-NP","I-NP","B-VP","B-NP","B-NP","I-NP","I-NP","I-NP"
-"DT","NN","VBZ","PRP","DT","RB","JJ","NN"
-"This","car","provides","you","a","very","good","mileage"
"Pine Tree Legal"
"B-NP","I-NP","I-NP"
"NNP","NNP","NNP"
"Pine","Tree","Legal"
+"A Message From Morgan This is Morgan, creator of Albert Einstein Online."
+"B-NP","I-NP","B-PP","B-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"DT","NN","IN","NN","DT","VBZ","JJ","NN","IN","NN","NNS"
+"A","Message","From","Morgan","This","is","Morgan","creator","of","Albert","Einstein"
"Means to deduct educational expense for my son"
"B-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
"NNP","TO","VB","JJ","NN","IN","PRP$","NN"
@@ -330,74 +378,78 @@
"B-ADVP","B-PP","B-NP","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-VP","B-NP","B-VP","I-VP","B-NP","O","B-NP","I-NP","B-PP"
"RB","IN","DT","PRP","VBP","TO","VB","DT","PRP$","NN","IN","NN","NNP","VBZ","PRP","VBP","VB","NN","CC","JJ","NNS","IN"
"First","of","all","I","don","t","do","all","my","shopping","at","Trader","Joe","s","I","get","produce","dairy","and","other","items","at"
+"Amazon.com: albert einstein ."
+"B-NP","I-NP","B-PP","B-NP"
+"NNP","NN","IN","CD"
+"Amazon","com","albert","einstein"
"Tell them that you need a 1040 tax form ."
"B-VP","B-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP"
"VB","PRP","IN","PRP","VBP","DT","CD","NN","NN"
"Tell","them","that","you","need","a","1040","tax","form"
-"CST- Stamford · http://Www.facebook.com/dcomplex12 ."
-"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
-"NN","NN","NN","VBD","NN","NN"
-"CST","Stamford","http","Www","facebook","com"
"Who Can Benefit From The . and deductions, who would prefer to have tax on that income withheld from their paychecks rather than make periodic separate payments through the estimated tax procedures ."
"B-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP","B-NP","B-VP","I-VP","I-VP","I-VP","B-NP","B-PP","B-NP","I-NP","B-VP","B-PP","B-NP","I-NP","B-PP","I-PP","O","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
"WP","MD","VB","IN","DT","CC","NNS","WP","MD","VB","TO","VB","NN","IN","DT","NN","VBD","IN","PRP$","NNS","RB","IN","VB","JJ","JJ","NNS","IN","DT","JJ","NN","NNS"
"Who","Can","Benefit","From","The","and","deductions","who","would","prefer","to","have","tax","on","that","income","withheld","from","their","paychecks","rather","than","make","periodic","separate","payments","through","the","estimated","tax","procedures"
-"Can I share lists with other Remember The Milk users?"
-"O","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","B-NP","I-NP","I-NP"
-"MD","PRP","VB","NNS","IN","JJ","NNP","DT","NN","NNS"
-"Can","I","share","lists","with","other","Remember","The","Milk","users"
"See you tomorrow night! . ."
"B-VP","B-NP","B-VP","B-NP"
"VB","PRP","VB","NN"
"See","you","tomorrow","night"
+"He enjoyed classical music and played the violin."
+"B-NP","B-VP","B-NP","I-NP","O","B-VP","B-NP","I-NP"
+"PRP","VBD","JJ","NN","CC","VBD","DT","NN"
+"He","enjoyed","classical","music","and","played","the","violin"
"I love their unsweetened vanilla almond milk! . ."
"B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP"
"PRP","VBP","PRP$","JJ","NN","NN","NN"
"I","love","their","unsweetened","vanilla","almond","milk"
+"Albert Einstein IMDb ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","IMDb"
"The strong engine gives it enough power."
"B-NP","I-NP","I-NP","B-VP","B-NP","B-NP","I-NP"
"DT","JJ","NN","VBZ","PRP","RB","NN"
"The","strong","engine","gives","it","enough","power"
-"While the app doesn't currently . ."
-"B-SBAR","B-NP","I-NP","I-NP","I-NP","O"
-"IN","DT","NN","NN","NN","RB"
-"While","the","app","doesn","t","currently"
-"It's sometimes hard to function and hard to want to get out of bed . Posted in baby amelia 3 Comments ? . I realized that time has gotten away from me, for tomorrow Amelia would have been four weeks old. . by now, and hopefully, had become an accomplished milk-cow. . ."
-"B-NP","B-VP","B-ADVP","B-ADJP","B-VP","I-VP","O","B-ADVP","B-VP","I-VP","I-VP","I-VP","B-PP","B-PP","B-NP","B-VP","B-PP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","B-SBAR","B-NP","B-VP","I-VP","B-ADVP","B-PP","B-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-ADJP","B-PP","B-ADVP","O","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","I-NP"
-"PRP","VBZ","RB","JJ","TO","VB","CC","JJ","TO","VB","TO","VB","IN","IN","NN","VBN","IN","NN","NNS","CD","NNS","PRP","VBD","IN","NN","VBZ","VBN","RB","IN","PRP","IN","NN","NNP","MD","VB","VBN","CD","NNS","JJ","IN","RB","CC","RB","VBD","VBN","DT","JJ","NN","NN"
-"It","s","sometimes","hard","to","function","and","hard","to","want","to","get","out","of","bed","Posted","in","baby","amelia","3","Comments","I","realized","that","time","has","gotten","away","from","me","for","tomorrow","Amelia","would","have","been","four","weeks","old","by","now","and","hopefully","had","become","an","accomplished","milk","cow"
+"Einstein Archives Online ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Einstein","Archives","Online"
"Tax on overseas income : Directgov - Money, tax and benefits"
"B-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
"NNP","IN","JJ","NN","NNP","NNP","NN","CC","NNS"
"Tax","on","overseas","income","Directgov","Money","tax","and","benefits"
-"For the United States income tax return, you will have several options available to you regarding claiming a . ."
-"B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","B-ADJP","B-PP","B-NP","B-VP","I-VP","B-UCP"
-"IN","DT","NNP","NNPS","NN","NN","NN","PRP","MD","VB","JJ","NNS","JJ","TO","PRP","VBG","VBG","DT"
-"For","the","United","States","income","tax","return","you","will","have","several","options","available","to","you","regarding","claiming","a"
+"While developing general relativity, Einstein became confused about the gauge invariance in the theory."
+"B-PP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"IN","VBG","JJ","NN","NNP","VBD","VBN","IN","DT","NN","NN","IN","DT","NN"
+"While","developing","general","relativity","Einstein","became","confused","about","the","gauge","invariance","in","the","theory"
"Read this page to learn all the details, and if you have any questions or run into any . . email address] Buy birthday present for Mike today Pay the phone bill tomorrow Make ."
"B-VP","B-NP","I-NP","B-VP","I-VP","B-NP","I-NP","I-NP","O","B-SBAR","B-NP","B-VP","B-NP","I-NP","O","B-VP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-ADVP","O"
"VB","DT","NN","TO","VB","PDT","DT","NNS","CC","IN","PRP","VBP","DT","NNS","CC","VB","IN","DT","NN","NN","NN","NN","NN","IN","JJ","NN","VBP","DT","NN","NN","RB","VB"
"Read","this","page","to","learn","all","the","details","and","if","you","have","any","questions","or","run","into","any","email","address","Buy","birthday","present","for","Mike","today","Pay","the","phone","bill","tomorrow","Make"
-"remember to buy milk tomorrow from 3 to jones"
-"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
-"VB","TO","VB","NN","NN","IN","CD","TO","NNS"
-"remember","to","buy","milk","tomorrow","from","3","to","jones"
-"Can I get auto focus lens for digital camera"
-"O","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
-"MD","PRP","VB","NN","NN","NNS","IN","JJ","NN"
-"Can","I","get","auto","focus","lens","for","digital","camera"
-"Going to Work Abroad - Revenue Commissioners"
-"B-VP","I-VP","I-VP","B-NP","I-NP","I-NP"
-"VBG","TO","VB","RB","NN","NNS"
-"Going","to","Work","Abroad","Revenue","Commissioners"
+"PBS Airdate: October 11, 2005."
+"B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","CD","CD"
+"PBS","Airdate","October","11","2005"
"How do I estimate my tax return with my last pay stub? . no dependents, your non taxable income is $9350, any excess of that will be taxed at . Using my pay stub for taxes file my taxes, is H.R.block. Can I use only my last paystub to do my taxes ."
"B-ADVP","O","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","B-NP","I-NP","B-PP","B-NP","B-VP","I-VP","I-VP","B-PP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","B-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","O","B-NP","B-VP","B-ADVP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP"
"WRB","VBP","PRP","VB","PRP$","NN","NN","IN","PRP$","JJ","NN","NN","RB","VBZ","PRP$","JJ","JJ","NN","VBZ","CD","DT","NN","IN","DT","MD","VB","VBN","IN","VBG","PRP$","NN","NN","IN","NNS","VBP","PRP$","NNS","VBZ","NNP","NNP","NN","MD","PRP","VB","RB","PRP$","JJ","NN","TO","VB","PRP$","NNS"
"How","do","I","estimate","my","tax","return","with","my","last","pay","stub","no","dependents","your","non","taxable","income","is","9350","any","excess","of","that","will","be","taxed","at","Using","my","pay","stub","for","taxes","file","my","taxes","is","H","R","block","Can","I","use","only","my","last","paystub","to","do","my","taxes"
-"Once you get home you are going to immidiatly start drinking the . . for four hours. remember no toxins 48 hours before and no alcohol . ."
-"O","B-NP","B-VP","B-NP","B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","I-VP","B-NP","I-NP","B-NP","I-NP","B-VP","B-NP","I-NP","B-NP","I-NP","O","O","B-NP","I-NP"
-"RB","PRP","VBP","NN","PRP","VBP","VBG","TO","RB","VB","VBG","DT","NN","CD","NNS","VBP","DT","NNS","CD","NNS","IN","CC","DT","NN"
-"Once","you","get","home","you","are","going","to","immidiatly","start","drinking","the","for","four","hours","remember","no","toxins","48","hours","before","and","no","alcohol"
+"Some Quotations of ALBERT EINSTEIN (1879-1955)"
+"B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","O"
+"DT","NNS","IN","NNP","NNP","CD","CD"
+"Some","Quotations","of","ALBERT","EINSTEIN","1879","1955"
+"Letter to his son Eduard (5 February 1930), as quoted in Walter Isaacson, Einstein: His Life "
+"B-NP","B-VP","B-NP","I-NP","B-NP","I-NP","I-NP","I-NP","B-SBAR","B-VP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","TO","PRP$","NN","NNP","CD","NNP","CD","IN","VBN","IN","NNP","NNP","NNP","NNP","NN"
+"Letter","to","his","son","Eduard","5","February","1930","as","quoted","in","Walter","Isaacson","Einstein","His","Life"
+"I can deduct office rental expense from my business profit to calculate net income."
+"B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP"
+"PRP","MD","VB","NN","JJ","NN","IN","PRP$","NN","NN","TO","VB","JJ","NN"
+"I","can","deduct","office","rental","expense","from","my","business","profit","to","calculate","net","income"
+"Albert Einstein was born at Ulm, in Württemberg, Germany, on March 14, 1879."
+"B-NP","I-NP","B-VP","I-VP","B-PP","B-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","O"
+"NNP","NNP","VBD","VBN","IN","NNP","IN","NNP","NNP","IN","NNP","CD","CD"
+"Albert","Einstein","was","born","at","Ulm","in","Württemberg","Germany","on","March","14","1879"
"Ask Stacy: Do I Have to Pay Taxes If I'm Not in the US?"
"B-VP","B-NP","I-NP","B-NP","B-VP","I-VP","I-VP","B-NP","B-SBAR","B-NP","B-VP","B-ADVP","B-PP","B-NP","I-NP"
"VB","NNP","VB","PRP","VBP","TO","VB","NNS","IN","PRP","VBP","RB","IN","DT","NNP"
@@ -406,10 +458,6 @@
"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-VP","B-PP","B-NP"
"VB","TO","VB","NN","NN","IN","JJ","TO","NNS"
"remember","to","buy","milk","tomorrow","from","third","to","joes"
-"Remember the Milk is a web application with numerous mobile app . complete today will automatically get moved to the tomorrow column. ."
-"B-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","B-VP","I-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP"
-"VB","DT","NN","VBZ","DT","NN","NN","IN","JJ","JJ","NN","JJ","NN","MD","RB","VB","VBN","TO","DT","NN","NN"
-"Remember","the","Milk","is","a","web","application","with","numerous","mobile","app","complete","today","will","automatically","get","moved","to","the","tomorrow","column"
"A guide to Irish income tax and capital gains tax liability based on some commonly . In general, tax is deducted from your salary through what is known as the Pay As You Earn (PAYE) . .."
"B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","O","B-NP","I-NP","I-NP","I-NP","B-VP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-PP","B-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-PP","B-NP","B-NP","I-NP"
"DT","NN","TO","JJ","NN","NN","CC","NN","NNS","NN","NN","VBN","IN","DT","JJ","IN","JJ","NN","VBZ","VBN","IN","PRP$","NN","IN","WP","VBZ","VBN","IN","DT","NN","IN","PRP","NNP","NN"
@@ -418,10 +466,14 @@
"B-SBAR","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP"
"IN","PRP","VBP","VBN","VBG","DT","NN"
"If","you","re","finished","viewing","a","task"
-"It's sometimes hard to function and hard to want to get out of bed . Posted in baby amelia 3 Comments » . I realized that time has gotten away from me, for tomorrow Amelia would have been four weeks old. . by now, and hopefully, had become an accomplished milk-cow. . ."
+"It's sometimes hard to function and hard to want to get out of bed . Posted in baby amelia 3 Comments » . I realized that time has gotten away from me, for tomorrow Amelia would have been four weeks old. . by now, and hopefully, had become an accomplished milk-cow. . ."
"B-NP","B-VP","B-ADVP","B-ADJP","B-VP","I-VP","O","B-ADVP","B-VP","I-VP","I-VP","I-VP","B-PP","B-PP","B-NP","B-VP","B-PP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","B-SBAR","B-NP","B-VP","I-VP","B-ADVP","B-PP","B-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-ADJP","B-PP","B-ADVP","O","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","I-NP"
"PRP","VBZ","RB","JJ","TO","VB","CC","JJ","TO","VB","TO","VB","IN","IN","NN","VBN","IN","NN","NNS","CD","NNS","PRP","VBD","IN","NN","VBZ","VBN","RB","IN","PRP","IN","NN","NNP","MD","VB","VBN","CD","NNS","JJ","IN","RB","CC","RB","VBD","VBN","DT","JJ","NN","NN"
"It","s","sometimes","hard","to","function","and","hard","to","want","to","get","out","of","bed","Posted","in","baby","amelia","3","Comments","I","realized","that","time","has","gotten","away","from","me","for","tomorrow","Amelia","would","have","been","four","weeks","old","by","now","and","hopefully","had","become","an","accomplished","milk","cow"
+"Albert Einstein (March 14, 1879 – April 18, 1955) was a German born theoretical physicist."
+"B-NP","I-NP","I-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","CD","CD","NNP","CD","CD","VBD","DT","JJ","JJ","JJ","NN"
+"Albert","Einstein","March","14","1879","April","18","1955","was","a","German","born","theoretical","physicist"
"How can I get short focus zoom lens for digital camera"
"B-ADVP","O","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
"WRB","MD","PRP","VB","JJ","NN","NN","NN","IN","JJ","NN"
@@ -434,46 +486,38 @@
"B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","O","B-VP","B-NP","I-NP","B-NP","B-VP","I-VP","B-NP","B-PP","B-NP","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP","B-NP","B-VP","B-ADJP","O","B-NP","B-VP","B-NP"
"WRB","PRP","VBP","DT","DT","NN","CC","VB","DT","NN","PRP","MD","VB","NN","IN","NN","WP","VBZ","TO","VB","IN","PRP","JJ","NN","WDT","VBZ","JJ","CC","PRP","VBG","NN"
"When","you","go","the","the","store","and","buy","some","milk","you","should","tell","everyone","in","line","who","dares","to","look","at","you","funny","yeah","that","s","right","I","m","buying","milk"
+"Albert Einstein profoundly changed physics and ideas about space and time."
+"B-NP","I-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"NNP","NNP","RB","VBD","NNS","CC","NNS","IN","NN","CC","NN"
+"Albert","Einstein","profoundly","changed","physics","and","ideas","about","space","and","time"
"D-Jones. @dcomplex12."
"B-NP","I-NP","I-NP"
"NNP","NNP","NN"
"D","Jones","dcomplex12"
-"If you live in the UK permanently you'll pay tax on overseas income."
-"B-SBAR","B-NP","B-VP","B-PP","B-NP","I-NP","B-ADVP","B-NP","B-VP","I-VP","B-NP","B-PP","B-NP","I-NP"
-"IN","PRP","VBP","IN","DT","NNP","RB","PRP","MD","VB","NN","IN","JJ","NN"
-"If","you","live","in","the","UK","permanently","you","ll","pay","tax","on","overseas","income"
-"Use Tax Questions and Answers"
-"B-NP","I-NP","I-NP","I-NP","I-NP"
-"NNP","NNP","NNPS","CC","NNPS"
-"Use","Tax","Questions","and","Answers"
-"20 Best Online To Do List Apps for Freelancers"
-"B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP"
-"CD","NNP","NNP","TO","VB","NNP","NNP","IN","NNS"
-"20","Best","Online","To","Do","List","Apps","for","Freelancers"
+"I need to add my rental income to my profits, but subtract rental expenses such as repair from it."
+"B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","O","B-VP","B-NP","I-NP","B-PP","I-PP","B-NP","B-SBAR","B-NP"
+"PRP","VBP","TO","VB","PRP$","JJ","NN","TO","PRP$","NNS","CC","VB","JJ","NNS","JJ","IN","NN","IN","PRP"
+"I","need","to","add","my","rental","income","to","my","profits","but","subtract","rental","expenses","such","as","repair","from","it"
"Gay Pulitzer Prize-Winning Reporter Jose Antonio Vargas Comes Out as Undocumented Immigrant Jose Antonio Vargas, a gay journalist who won a Pulitzer Prize for his coverage of the Virginia Tech shootings in the Washington Post"
"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
"NNP","NNP","NNP","NNP","NNP","NNP","NNP","NNP","NNP","IN","IN","JJ","NNP","NNP","NNP","NNP","DT","JJ","NN","WP","VBD","DT","NNP","NNP","IN","PRP$","NN","IN","DT","NNP","NNP","NNS","IN","DT","NNP","NNP"
"Gay","Pulitzer","Prize","Winning","Reporter","Jose","Antonio","Vargas","Comes","Out","as","Undocumented","Immigrant","Jose","Antonio","Vargas","a","gay","journalist","who","won","a","Pulitzer","Prize","for","his","coverage","of","the","Virginia","Tech","shootings","in","the","Washington","Post"
-"How to deduct repair expense from rental income."
-"B-ADVP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
-"WRB","TO","VB","NN","NN","IN","JJ","NN"
-"How","to","deduct","repair","expense","from","rental","income"
-"Even if you avoid U.S."
-"B-SBAR","I-SBAR","B-NP","B-VP","B-NP","I-NP"
-"RB","IN","PRP","VBP","NNP","NNP"
-"Even","if","you","avoid","U","S"
-"Get Organized with Remember the Milk"
-"B-VP","B-NP","I-PRT","B-VP","B-NP","I-NP"
-"VB","NNP","IN","VB","DT","NN"
-"Get","Organized","with","Remember","the","Milk"
-"This series of questions and answers refers specifically to the use tax incurred on . For taxpayers who do not have records to document their use tax liability, the department will estimate liability. . Individual Income Tax Return, on or before April 15 of the following year (for tax . Can I file and pay my use tax electronically? ."
-"B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-ADJP","B-PP","B-NP","I-NP","I-NP","B-VP","B-PP","B-PP","B-NP","B-NP","B-VP","I-VP","I-VP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-NP","B-PP","I-PP","I-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","O","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","O"
-"DT","NN","IN","NNS","CC","NNS","VBZ","RB","TO","DT","NN","NN","VBN","IN","IN","NNS","WP","VBP","RB","VB","NNS","TO","VB","PRP$","NN","NN","NN","DT","NN","MD","VB","NN","NNP","NNP","NNP","NN","IN","CC","IN","NNP","CD","IN","DT","JJ","NN","IN","NN","MD","PRP","VB","CC","VB","PRP$","NN","NN","RB"
-"This","series","of","questions","and","answers","refers","specifically","to","the","use","tax","incurred","on","For","taxpayers","who","do","not","have","records","to","document","their","use","tax","liability","the","department","will","estimate","liability","Individual","Income","Tax","Return","on","or","before","April","15","of","the","following","year","for","tax","Can","I","file","and","pay","my","use","tax","electronically"
-"D-Jones (dcomplex12) on Twitter"
-"B-NP","I-NP","I-NP","B-PP","B-NP"
-"NNP","NNP","NN","IN","NN"
-"D","Jones","dcomplex12","on","Twitter"
+"The theory of relativity is a beautiful example of the basic character of the modern development of theory."
+"B-NP","I-NP","B-PP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP"
+"DT","NN","IN","NN","VBZ","DT","JJ","NN","IN","DT","JJ","NN","IN","DT","JJ","NN","IN","NN"
+"The","theory","of","relativity","is","a","beautiful","example","of","the","basic","character","of","the","modern","development","of","theory"
+"In 1879, Albert Einstein was born in Ulm, Germany."
+"B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP"
+"IN","CD","NNP","NNP","VBD","VBN","IN","NNP","NNP"
+"In","1879","Albert","Einstein","was","born","in","Ulm","Germany"
+"I am afraid I will end up paying the tax."
+"B-NP","B-VP","B-ADJP","B-NP","B-VP","I-VP","B-ADVP","B-VP","B-NP","I-NP"
+"PRP","VBP","JJ","PRP","MD","VB","RP","VBG","DT","NN"
+"I","am","afraid","I","will","end","up","paying","the","tax"
+"Get all the facts on HISTORY.com"
+"B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"VB","PDT","DT","NNS","IN","NNP","NN"
+"Get","all","the","facts","on","HISTORY","com"
"Here's a list of BPA-free coconut milk, fish, pumpkin, and tomatoes. . and some (like Trader Joe's) don't even put the label on their products. . . but just remember to check with the supplier before making your purchase. . .."
"B-ADVP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","O","B-NP","O","B-NP","B-PP","B-NP","I-NP","B-VP","B-PP","B-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","O","B-VP","I-VP","I-VP","I-VP","B-PP","B-NP","I-NP","B-PP","B-VP","B-NP","I-NP"
"RB","VBZ","DT","NN","IN","NNP","JJ","NN","NN","NN","NN","CC","NNS","CC","DT","IN","NN","NNP","VBZ","IN","NN","RB","VBD","DT","NN","IN","PRP$","NNS","CC","RB","VB","TO","VB","IN","DT","NN","IN","VBG","PRP$","NN"
@@ -482,46 +526,578 @@
"B-VP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-PP","B-NP","I-NP"
"VB","DT","NN","NNP","NNP","DT","NN","IN","NN","NN"
"Remember","The","Milk","Services","Remember","The","Milk","for","iPad","FAQ"
-"Way to minimize medical expense for my daughter"
-"B-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
-"NN","TO","VB","JJ","NN","IN","PRP$","NN"
-"Way","to","minimize","medical","expense","for","my","daughter"
-"remember to buy milk tomorrow from for d jones"
-"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-PP","B-NP","I-NP"
-"VB","TO","VB","NN","NN","IN","IN","NN","NNS"
-"remember","to","buy","milk","tomorrow","from","for","d","jones"
"I use the olive oil, I buy my milk and eggs from them. . Remember I'm talking raw non-homogenized which is even more delicate. ."
"B-NP","B-VP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","B-VP","I-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP"
"PRP","VBP","DT","JJ","NN","PRP","VBP","PRP$","NN","CC","NNS","IN","PRP","VB","PRP","VB","VBG","JJ","NN","VBN","WDT","VBZ","RB","JJR","NN"
"I","use","the","olive","oil","I","buy","my","milk","and","eggs","from","them","Remember","I","m","talking","raw","non","homogenized","which","is","even","more","delicate"
-"While you will need your W-2 or a substitute, you can figure out your tax obligation . final pay stub for a job (if you are no longer employed) or the last pay stub for the year . the year, you can estimate your income and taxes from a single pay stub. . However, if you worked at the same job the previous year, you can use the . ."
-"B-SBAR","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","O","B-NP","I-NP","B-NP","B-VP","I-VP","B-PRT","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-SBAR","B-NP","B-VP","B-ADVP","I-ADVP","B-VP","O","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-ADVP","B-SBAR","B-NP","B-VP","B-PP","B-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-NP","B-VP","I-VP","B-NP"
-"IN","PRP","MD","VB","PRP$","NN","CD","CC","DT","NN","PRP","MD","VB","RP","PRP$","NN","NN","JJ","NN","NN","IN","DT","NN","IN","PRP","VBP","RB","RB","VBN","CC","DT","JJ","NN","NN","IN","DT","NN","DT","NN","PRP","MD","VB","PRP$","NN","CC","NNS","IN","DT","JJ","NN","NN","RB","IN","PRP","VBD","IN","DT","JJ","NN","DT","JJ","NN","PRP","MD","VB","DT"
-"While","you","will","need","your","W","2","or","a","substitute","you","can","figure","out","your","tax","obligation","final","pay","stub","for","a","job","if","you","are","no","longer","employed","or","the","last","pay","stub","for","the","year","the","year","you","can","estimate","your","income","and","taxes","from","a","single","pay","stub","However","if","you","worked","at","the","same","job","the","previous","year","you","can","use","the"
+"Montefiore Medical Center ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Montefiore","Medical","Center"
+"Advertisement and repair expenses can be subtracted from the rental income."
+"B-NP","I-NP","I-NP","I-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP"
+"NN","CC","NN","NNS","MD","VB","VBN","IN","DT","JJ","NN"
+"Advertisement","and","repair","expenses","can","be","subtracted","from","the","rental","income"
"income tax, you will likely pay some form of income tax to the . My son is working overseas and pays all taxes and health insurance there. ."
"B-NP","I-NP","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-ADVP","O","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP"
"NN","NN","PRP","MD","RB","VB","DT","NN","IN","NN","NN","TO","DT","PRP$","NN","VBZ","VBG","RB","CC","VBZ","DT","NNS","CC","NN","NN"
"income","tax","you","will","likely","pay","some","form","of","income","tax","to","the","My","son","is","working","overseas","and","pays","all","taxes","and","health","insurance"
-"remember to buy milk tomorrow for for details"
-"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
-"VB","TO","VB","NN","NN","IN","NN","NNS"
-"remember","to","buy","milk","tomorrow","for","for","details"
"remember to buy milk tomorrow from trader joes"
"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
"VB","TO","VB","NN","NN","IN","NN","NNS"
"remember","to","buy","milk","tomorrow","from","trader","joes"
-"Tagged - Damaya Lady D Jones's Profile"
-"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
-"JJ","NNP","NNP","NNP","NNP","NNS","NN"
-"Tagged","Damaya","Lady","D","Jones","s","Profile"
+"Today, the practical applications of Einstein s theories include the development of the television, remote control devices, automatic door openers, lasers, and DVD-players."
+"B-NP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","B-ADVP","B-NP","I-NP","I-NP","O","B-NP","I-NP"
+"NN","DT","JJ","NNS","IN","NNP","NNS","NNS","VBP","DT","NN","IN","DT","NN","JJ","NN","NNS","JJ","NN","NNS","NNS","CC","NNP","NNS"
+"Today","the","practical","applications","of","Einstein","s","theories","include","the","development","of","the","television","remote","control","devices","automatic","door","openers","lasers","and","DVD","players"
"In 1956, he met Joe Campbell, at the Jacob Riis Park beach, a popular location for . . but remembered Milk's attitude: I think he was happier than at any time I had ever . ."
"B-PP","B-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","O","B-VP","B-NP","B-VP","B-SBAR","B-NP","B-VP","B-NP","B-VP","B-ADJP","B-PP","B-PP","B-NP","I-NP","B-NP","B-VP","I-VP"
"IN","CD","PRP","VBD","NNP","NNP","IN","DT","NNP","NNP","NNP","IN","DT","JJ","NN","IN","CC","VBD","NNP","VBZ","IN","PRP","VBP","PRP","VBD","JJR","IN","IN","DT","NN","PRP","VBD","RB"
"In","1956","he","met","Joe","Campbell","at","the","Jacob","Riis","Park","beach","a","popular","location","for","but","remembered","Milk","s","attitude","I","think","he","was","happier","than","at","any","time","I","had","ever"
+"Son of Hermann and Pauline Einstein."
+"B-NP","B-PP","B-NP","O","B-NP","I-NP"
+"NN","IN","NNP","CC","NNP","NNP"
+"Son","of","Hermann","and","Pauline","Einstein"
+"And how many folks out there "
+"O","B-ADVP","B-NP","B-VP","B-ADVP","I-ADVP"
+"CC","WRB","JJ","NNS","IN","RB"
+"And","how","many","folks","out","there"
"This car has a great engine."
"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP"
"DT","NN","VBZ","DT","JJ","NN"
"This","car","has","a","great","engine"
+"He is best known for his theory of relativity and specifically the "
+"B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","B-PP","B-NP","O","O"
+"PRP","VBZ","RB","VBN","IN","PRP$","NN","IN","NN","CC","RB"
+"He","is","best","known","for","his","theory","of","relativity","and","specifically"
+"Albert Einstein Rotten Tomatoes ."
+"B-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNPS"
+"Albert","Einstein","Rotten","Tomatoes"
+"Albert Einstein Biography The Life and Achievements of .."
+"B-NP","I-NP","I-NP","I-NP","I-NP","O","B-NP","B-PP"
+"NNP","NNP","NNP","NNP","NNP","CC","NNP","IN"
+"Albert","Einstein","Biography","The","Life","and","Achievements","of"
+"I showed my property to a business owner to rent."
+"B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP"
+"PRP","VBD","PRP$","NN","TO","DT","NN","NN","TO","NN"
+"I","showed","my","property","to","a","business","owner","to","rent"
+"BPA-Free Versions of Popular Foods | Mark's Daily Apple"
+"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
+"NNP","NNP","NNP","IN","NNP","NNP","NNP","VBZ","NNP","NNP"
+"BPA","Free","Versions","of","Popular","Foods","Mark","s","Daily","Apple"
+"$2.8 billion, but she did not pay a dime in state income tax in 2010, the . investing in local and state governments, earning money overseas and . I want my tax dollars to go to my children's schools, not the president of GE ."
+"B-NP","I-NP","I-NP","O","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","B-ADVP","O","B-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP"
+"CD","CD","CD","CC","PRP","VBD","RB","VB","DT","NN","IN","NN","NN","NN","IN","CD","DT","NN","IN","JJ","CC","NN","NNS","VBG","NN","RB","CC","PRP","VBP","PRP$","NN","NNS","TO","VB","TO","PRP$","NNS","VBZ","NNS","RB","DT","NN","IN","NNP"
+"2","8","billion","but","she","did","not","pay","a","dime","in","state","income","tax","in","2010","the","investing","in","local","and","state","governments","earning","money","overseas","and","I","want","my","tax","dollars","to","go","to","my","children","s","schools","not","the","president","of","GE"
+"remember to buy milk tomorrow for details"
+"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP"
+"VB","TO","VB","NN","NN","IN","NNS"
+"remember","to","buy","milk","tomorrow","for","details"
+"I remember I used to get excited to go shopping at the mall, now I have those same feelings towards TJ's. ."
+"B-NP","B-VP","B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","I-VP","B-NP","B-PP","B-NP","I-NP","B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"PRP","VBP","PRP","VBD","TO","VB","VBN","TO","VB","NN","IN","DT","NN","RB","FW","VBP","DT","JJ","NNS","IN","NN","NNS"
+"I","remember","I","used","to","get","excited","to","go","shopping","at","the","mall","now","I","have","those","same","feelings","towards","TJ","s"
+"Albert Einstein was a German born physicist who developed the theory of relativity."
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP"
+"NNP","NNP","VBD","DT","JJ","NN","NN","WP","VBD","DT","NN","IN","NN"
+"Albert","Einstein","was","a","German","born","physicist","who","developed","the","theory","of","relativity"
+"Pulitzer Prize-Winning Reporter is an Illegal Immigrant"
+"B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","VBZ","DT","NNP","NN"
+"Pulitzer","Prize","Winning","Reporter","is","an","Illegal","Immigrant"
+"Albert Einstein Few people are capable of expressing with equanimity opinions which differ from the prejudices of their social environment."
+"B-NP","I-NP","I-NP","I-NP","B-VP","B-ADJP","B-PP","B-VP","B-PP","B-NP","I-NP","B-NP","B-VP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"NNP","NNP","JJ","NNS","VBP","JJ","IN","VBG","IN","NN","NNS","WDT","VBP","IN","DT","NNS","IN","PRP$","JJ","NN"
+"Albert","Einstein","Few","people","are","capable","of","expressing","with","equanimity","opinions","which","differ","from","the","prejudices","of","their","social","environment"
+"It's sometimes hard to function and hard to want to get out of bed . Posted in baby amelia 3 Comments � . I realized that time has gotten away from me, for tomorrow Amelia would have been four weeks old. . by now, and hopefully, had become an accomplished milk-cow. . ."
+"B-NP","B-VP","B-ADVP","B-ADJP","B-VP","I-VP","O","B-ADVP","B-VP","I-VP","I-VP","I-VP","B-PP","B-PP","B-NP","B-VP","B-PP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","B-SBAR","B-NP","B-VP","I-VP","B-ADVP","B-PP","B-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-ADJP","B-PP","B-ADVP","O","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","I-NP"
+"PRP","VBZ","RB","JJ","TO","VB","CC","JJ","TO","VB","TO","VB","IN","IN","NN","VBN","IN","NN","NNS","CD","NNS","PRP","VBD","IN","NN","VBZ","VBN","RB","IN","PRP","IN","NN","NNP","MD","VB","VBN","CD","NNS","JJ","IN","RB","CC","RB","VBD","VBN","DT","JJ","NN","NN"
+"It","s","sometimes","hard","to","function","and","hard","to","want","to","get","out","of","bed","Posted","in","baby","amelia","3","Comments","I","realized","that","time","has","gotten","away","from","me","for","tomorrow","Amelia","would","have","been","four","weeks","old","by","now","and","hopefully","had","become","an","accomplished","milk","cow"
+"A world famous theoretical physicist, he was awarded the 1921 Nobel Prize for "
+"B-NP","I-NP","I-NP","I-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-UCP"
+"DT","NN","JJ","JJ","NN","PRP","VBD","VBN","DT","CD","NNP","NNP","IN"
+"A","world","famous","theoretical","physicist","he","was","awarded","the","1921","Nobel","Prize","for"
+"cannot conceive a god"
+"B-NP","B-VP","B-NP","I-NP"
+"NN","VBP","DT","NN"
+"cannot","conceive","a","god"
+"Sounds too good to be true but it actually is, the world's first flying car is finally here. "
+"B-VP","B-ADJP","I-ADJP","B-VP","I-VP","B-ADJP","O","B-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-ADVP","O"
+"VBZ","RB","JJ","TO","VB","JJ","CC","PRP","RB","VBZ","DT","NN","NNS","JJ","NN","NN","VBZ","RB","RB"
+"Sounds","too","good","to","be","true","but","it","actually","is","the","world","s","first","flying","car","is","finally","here"
+"Most people are even "
+"B-NP","I-NP","B-VP","O"
+"JJS","NNS","VBP","RB"
+"Most","people","are","even"
+"All my income comes from my host country and I pay plenty of taxes there! . is that earned income from overseas can be exempted from US income tax (but only .. ."
+"B-NP","I-NP","I-NP","B-VP","B-PP","B-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","B-PP","B-NP","B-ADVP","B-VP","B-SBAR","B-NP","I-NP","B-PP","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP","O","O"
+"DT","PRP$","NN","VBZ","IN","PRP$","NN","NN","CC","PRP","VBP","NN","IN","NNS","RB","VBZ","IN","JJ","NN","IN","RB","MD","VB","VBN","IN","NNP","NN","NN","CC","RB"
+"All","my","income","comes","from","my","host","country","and","I","pay","plenty","of","taxes","there","is","that","earned","income","from","overseas","can","be","exempted","from","US","income","tax","but","only"
+"Albert Einstein (14 March 1879 – 18 April 1955) was a German born theoretical physicist who developed the general theory of relativity, one of the two pillars of "
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP"
+"NNP","NNP","CD","NNP","CD","CD","NNP","CD","VBD","DT","JJ","JJ","JJ","NN","WP","VBD","DT","JJ","NN","IN","NN","CD","IN","DT","CD","NNS","IN"
+"Albert","Einstein","14","March","1879","18","April","1955","was","a","German","born","theoretical","physicist","who","developed","the","general","theory","of","relativity","one","of","the","two","pillars","of"
+"Just tryna take over the World."
+"B-NP","I-NP","B-VP","B-PP","B-NP","I-NP"
+"RB","NNS","VB","IN","DT","NNP"
+"Just","tryna","take","over","the","World"
+"Join Tagged and be friends with Damaya Lady D Jones - it's free! . Purpose Driven Life by Rick Warren, Milk In My Coffee by Eric Jerome Dickey, . I know tomorrow is not promised so I always try to remember to cherish LIFE because ."
+"B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","B-ADVP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","B-SBAR"
+"VB","JJ","CC","VB","NNS","IN","NNP","NNP","NNP","NNP","PRP","VBZ","JJ","NN","VBN","NN","IN","NNP","NNP","NNP","IN","PRP$","NNP","IN","NNP","NNP","NNP","PRP","VBP","NN","VBZ","RB","VBN","IN","PRP","RB","VBP","TO","VB","TO","JJ","NNP","IN"
+"Join","Tagged","and","be","friends","with","Damaya","Lady","D","Jones","it","s","free","Purpose","Driven","Life","by","Rick","Warren","Milk","In","My","Coffee","by","Eric","Jerome","Dickey","I","know","tomorrow","is","not","promised","so","I","always","try","to","remember","to","cherish","LIFE","because"
+"CST- Stamford ? http://Www.facebook.com/dcomplex12 ."
+"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
+"NN","NN","NN","VBD","NN","NN"
+"CST","Stamford","http","Www","facebook","com"
+"Albert Einstein was a German born theoretical physicist."
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","VBD","DT","JJ","JJ","JJ","NN"
+"Albert","Einstein","was","a","German","born","theoretical","physicist"
+"Albert Einstein | Albert Einstein Official Site ."
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP","NNP"
+"Albert","Einstein","Albert","Einstein","Official","Site"
+"Albert Einstein"
+"B-NP","I-NP"
+"NNP","NNP"
+"Albert","Einstein"
+"Life is like riding a bicycle."
+"B-NP","B-VP","B-PP","B-VP","B-NP","I-NP"
+"NN","VBZ","IN","VBG","DT","NN"
+"Life","is","like","riding","a","bicycle"
+"How do I estimate my tax return with my last pay stub"
+"B-ADVP","O","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
+"WRB","VBP","PRP","VB","PRP$","NN","NN","IN","PRP$","JJ","NN","NN"
+"How","do","I","estimate","my","tax","return","with","my","last","pay","stub"
+"He is considered the most influential physicist of the 20th century."
+"B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"PRP","VBZ","VBN","DT","RBS","JJ","NN","IN","DT","JJ","NN"
+"He","is","considered","the","most","influential","physicist","of","the","20th","century"
+"Rental expense needs to be subtracted from revenue. "
+"B-NP","I-NP","B-VP","I-VP","I-VP","I-VP","B-PP","B-NP"
+"JJ","NN","VBZ","TO","VB","VBN","IN","NN"
+"Rental","expense","needs","to","be","subtracted","from","revenue"
+"Share with your friends."
+"B-NP","B-PP","B-NP","I-NP"
+"NN","IN","PRP$","NNS"
+"Share","with","your","friends"
+"I bake with the almond meal often and buy the fresh almond milk (blue container), . ."
+"B-NP","B-VP","B-PP","B-NP","I-NP","I-NP","B-ADVP","O","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"PRP","VBP","IN","DT","JJ","NN","RB","CC","VB","DT","JJ","NN","NN","JJ","NN"
+"I","bake","with","the","almond","meal","often","and","buy","the","fresh","almond","milk","blue","container"
+"Albert Einstein College of Medicine is one of the nation’s premier institutions for medical education, basic research and clinical investigation."
+"B-NP","I-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","O","B-NP","I-NP"
+"NNP","NNP","NNP","IN","NNP","VBZ","CD","IN","DT","NN","NNS","JJ","NNS","IN","JJ","NN","NN","NN","CC","JJ","NN"
+"Albert","Einstein","College","of","Medicine","is","one","of","the","nation","s","premier","institutions","for","medical","education","basic","research","and","clinical","investigation"
+"If you live here temporarily, you'll normally pay tax only on overseas income you bring into ."
+"B-SBAR","B-NP","B-VP","B-ADVP","I-ADVP","B-NP","B-VP","I-VP","I-VP","B-NP","B-ADVP","B-PP","B-NP","I-NP","B-NP","B-VP","O"
+"IN","PRP","VBP","RB","RB","PRP","MD","RB","VB","NN","RB","IN","JJ","NN","PRP","VBP","IN"
+"If","you","live","here","temporarily","you","ll","normally","pay","tax","only","on","overseas","income","you","bring","into"
+"remember to pick up milk at seven (smste00006) get some cleaning supplies at . tomorrow (smste00025) can we have a quick dinner this week (smste00026) . .. (smste00217) remind me to call estefana+1 on the twenty third (smste00218) . of joe's+2 coffee+2 shop+2 then we'll go together (smste00220) can we have a ."
+"B-VP","I-VP","I-VP","B-PRT","B-NP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-NP","I-NP","B-VP","B-NP","B-VP","B-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","I-NP","I-NP","I-NP","B-NP","B-ADVP","B-NP","B-VP","I-VP","B-ADJP","I-ADJP","O","B-NP","B-VP","B-NP"
+"VB","TO","VB","RP","NN","IN","CD","NNS","VBP","DT","NN","NNS","IN","NN","NN","MD","PRP","VB","DT","JJ","NN","DT","NN","VBD","CD","VBP","PRP","TO","VB","NN","CD","IN","DT","CD","JJ","NN","IN","NN","NNS","CD","NN","CD","NN","CD","RB","PRP","MD","VB","RB","JJR","MD","PRP","VB","DT"
+"remember","to","pick","up","milk","at","seven","smste00006","get","some","cleaning","supplies","at","tomorrow","smste00025","can","we","have","a","quick","dinner","this","week","smste00026","smste00217","remind","me","to","call","estefana","1","on","the","twenty","third","smste00218","of","joe","s","2","coffee","2","shop","2","then","we","ll","go","together","smste00220","can","we","have","a"
+"Albert Einstein was born in Germany in 1879."
+"B-NP","I-NP","B-VP","I-VP","B-PP","B-NP","B-PP","B-NP"
+"NNP","NNP","VBD","VBN","IN","NNP","IN","CD"
+"Albert","Einstein","was","born","in","Germany","in","1879"
+"While it may seem like something straight out of a sci-fi movie, the flying car might soon become a reality. "
+"B-SBAR","B-NP","B-VP","I-VP","B-PP","B-NP","B-ADVP","B-PP","B-PP","B-NP","I-NP","B-ADVP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP"
+"IN","PRP","MD","VB","IN","NN","RB","IN","IN","DT","NNS","JJ","NN","DT","VBG","NN","MD","RB","VB","DT","NN"
+"While","it","may","seem","like","something","straight","out","of","a","sci","fi","movie","the","flying","car","might","soon","become","a","reality"
+"Item 361 - 380 ? Profile picture of djones. djones. @djones active 6 months, 3 weeks ago . There are many buy one, get one free offers for area restaurants, museums, zoo, sporting events, etc. . Vicki Todd wrote a new blog post: PLEASE REMEMBER? . NOTE: although we collect Swiss Valley milk caps and Campbell's . djones - Rock Island/Milan School District #41"
+"B-NP","B-PP","I-NP","I-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-ADJP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP","B-VP","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","O"
+"PRP","CD","CD","NNP","NN","IN","NNS","NNS","NNS","JJ","CD","NNS","CD","NNS","IN","EX","VBP","JJ","VBP","CD","IN","CD","JJ","NNS","IN","NN","NNS","NNS","NN","VBG","NNS","FW","NNP","NNP","VBD","DT","JJ","NN","NN","NNP","NNP","NNP","IN","PRP","VBP","JJ","NNP","NN","NNS","CC","NNP","VBD","NNS","NNP","NNP","NNP","NNP","NNP","CD"
+"Item","361","380","Profile","picture","of","djones","djones","djones","active","6","months","3","weeks","ago","There","are","many","buy","one","get","one","free","offers","for","area","restaurants","museums","zoo","sporting","events","etc","Vicki","Todd","wrote","a","new","blog","post","PLEASE","REMEMBER","NOTE","although","we","collect","Swiss","Valley","milk","caps","and","Campbell","s","djones","Rock","Island","Milan","School","District","41"
+"My rental profits are added to my taxable income. "
+"B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP"
+"PRP$","JJ","NNS","VBP","VBN","TO","PRP$","JJ","NN"
+"My","rental","profits","are","added","to","my","taxable","income"
+"Albert Einstein Biography Physicist, Scientist .."
+"B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP"
+"Albert","Einstein","Biography","Physicist","Scientist"
+"People are worried about having to pay a fine for not carrying health insurance coverage got more guidance this week with some new federal regulations."
+"B-NP","B-VP","I-VP","B-PP","B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","O","B-VP","B-NP","I-NP","I-NP","B-VP","B-NP","I-NP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
+"NNS","VBP","VBN","IN","VBG","TO","VB","DT","JJ","IN","RB","VBG","NN","NN","NN","VBD","JJR","NN","DT","NN","IN","DT","JJ","JJ","NNS"
+"People","are","worried","about","having","to","pay","a","fine","for","not","carrying","health","insurance","coverage","got","more","guidance","this","week","with","some","new","federal","regulations"
+"Albert Einstein was a German born theoretical physicist who discovered the theory of general relativity, effecting a revolution in physics."
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP"
+"NNP","NNP","VBD","DT","JJ","JJ","JJ","NN","WP","VBD","DT","NN","IN","JJ","NN","VBG","DT","NN","IN","NNS"
+"Albert","Einstein","was","a","German","born","theoretical","physicist","who","discovered","the","theory","of","general","relativity","effecting","a","revolution","in","physics"
+"Einstein was born at Ulm, in Württemberg, Germany, on."
+"B-NP","B-VP","I-VP","B-PP","B-NP","B-PP","B-NP","I-NP","B-PP"
+"NNP","VBD","VBN","IN","NNP","IN","NNP","NNP","IN"
+"Einstein","was","born","at","Ulm","in","Württemberg","Germany","on"
+"Albert Einstein Quotes The Quotations Page ."
+"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP"
+"NNP","NNP","NNP","DT","NNPS","NNP"
+"Albert","Einstein","Quotes","The","Quotations","Page"
+"Robert Jones 2 years ago . this month. what are my chances of passing my test tomorrow? . .."
+"B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-VP","B-NP","I-NP","I-NP"
+"NNP","NNP","CD","NNS","IN","DT","NN","WP","VBP","PRP$","NNS","IN","VBG","PRP$","NN","NN"
+"Robert","Jones","2","years","ago","this","month","what","are","my","chances","of","passing","my","test","tomorrow"
+"That's one way to remember it. . I've been getting reactions to much of what I buy at Trader Joe's. . The rice milk is made by the same company that makes Rice Dream and Rice Dream uses barley gluten in the process, than claims it is taken . ."
+"B-NP","B-VP","B-NP","I-NP","B-VP","I-VP","B-NP","B-NP","B-VP","I-VP","I-VP","B-NP","B-PP","B-NP","B-PP","B-NP","B-NP","B-VP","B-PP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","I-NP","I-NP","B-NP","B-VP","B-NP","I-NP","O","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","B-NP","B-VP","I-VP"
+"DT","VBZ","CD","NN","TO","VB","PRP","PRP","VBP","VBN","VBG","NNS","TO","JJ","IN","WP","PRP","VBP","IN","NN","NNP","VBZ","DT","NN","NN","VBZ","VBN","IN","DT","JJ","NN","WDT","VBZ","NNP","NNP","CC","NNP","NNP","VBZ","NN","NNS","IN","DT","NN","IN","NNS","PRP","VBZ","VBN"
+"That","s","one","way","to","remember","it","I","ve","been","getting","reactions","to","much","of","what","I","buy","at","Trader","Joe","s","The","rice","milk","is","made","by","the","same","company","that","makes","Rice","Dream","and","Rice","Dream","uses","barley","gluten","in","the","process","than","claims","it","is","taken"
+"People are exempt from health insurance fine if they make too little money to file an income tax return, or US citizens living abroad."
+"B-NP","B-VP","B-ADJP","B-PP","B-NP","I-NP","I-NP","B-SBAR","B-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","O","B-NP","B-NP","B-VP","O"
+"NNS","VBP","JJ","IN","NN","NN","NN","IN","PRP","VBP","RB","JJ","NN","TO","VB","DT","NN","NN","NN","CC","PRP","NNS","VBG","RB"
+"People","are","exempt","from","health","insurance","fine","if","they","make","too","little","money","to","file","an","income","tax","return","or","US","citizens","living","abroad"
+"Sync with Remember The Milk online (limit once every 24 hours). . include extra details about tasks in the 'Add task' bar (e.g., Pick up the milk tomorrow). . Detect your current location to see nearby tasks; plan the best way to get things done. ."
+"B-NP","B-PP","B-NP","B-NP","I-NP","I-NP","I-NP","B-ADVP","B-ADVP","B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-ADVP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","O"
+"NNP","IN","NNP","DT","NNP","NNP","NN","RB","RB","CD","NNS","VBP","JJ","NNS","IN","NNS","IN","DT","NNP","NN","NN","NN","NN","VBG","RP","DT","NN","NN","NNP","PRP$","JJ","NN","TO","VB","JJ","NNS","VBP","DT","JJS","NN","TO","VB","NNS","VBN"
+"Sync","with","Remember","The","Milk","online","limit","once","every","24","hours","include","extra","details","about","tasks","in","the","Add","task","bar","e","g","Pick","up","the","milk","tomorrow","Detect","your","current","location","to","see","nearby","tasks","plan","the","best","way","to","get","things","done"
+"Includes blogs, news, and community conversations about Albert Einstein."
+"B-NP","I-NP","I-NP","O","B-NP","I-NP","B-PP","B-NP","I-NP"
+"NNP","NNS","NN","CC","NN","NNS","IN","NNP","NNP"
+"Includes","blogs","news","and","community","conversations","about","Albert","Einstein"
+"I receive rental income from my office."
+"B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"PRP","VBP","JJ","NN","IN","PRP$","NN"
+"I","receive","rental","income","from","my","office"
+"Once you do, on the right hand side, a task box will contain editable details about that task. ."
+"O","B-NP","B-VP","B-PP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"RB","PRP","VBP","IN","DT","JJ","NN","NN","DT","NN","NN","MD","VB","JJ","NNS","IN","DT","NN"
+"Once","you","do","on","the","right","hand","side","a","task","box","will","contain","editable","details","about","that","task"
+"NOVA | Einstein s Big Idea ."
+"B-NP","I-NP","B-VP","B-NP","I-NP"
+"NNP","NNP","VBZ","JJ","NN"
+"NOVA","Einstein","s","Big","Idea"
+"Collected Quotes from Albert Einstein . [Note: This list of Einstein quotes was being forwarded around the Internet in e mail, so I decided to put it on my web page."
+"B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-NP","I-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","B-ADVP","B-NP","B-VP","I-VP","I-VP","B-NP","B-PP","B-NP","I-NP","I-NP"
+"NNP","NNP","IN","NNP","NNP","NNP","DT","NN","IN","NNP","NNS","VBD","VBG","VBN","IN","DT","NNP","IN","NN","NN","RB","PRP","VBD","TO","VB","PRP","IN","PRP$","NN","NN"
+"Collected","Quotes","from","Albert","Einstein","Note","This","list","of","Einstein","quotes","was","being","forwarded","around","the","Internet","in","e","mail","so","I","decided","to","put","it","on","my","web","page"
+"NARRATOR: When we think of E = mc2 we have this vision of Einstein as an old wrinkly man with white hair."
+"B-NP","B-ADVP","B-NP","B-VP","B-PP","B-NP","I-NP","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"NN","WRB","PRP","VBP","IN","NNP","NN","PRP","VBP","DT","NN","IN","NNP","IN","DT","JJ","JJ","NN","IN","JJ","NN"
+"NARRATOR","When","we","think","of","E","mc2","we","have","this","vision","of","Einstein","as","an","old","wrinkly","man","with","white","hair"
+"Because I'm somewhat obsessed with shopping at Trader Joe's, people always . .."
+"B-SBAR","B-NP","B-VP","B-ADJP","I-ADJP","B-PP","B-NP","B-PP","B-NP","I-NP","B-VP","B-NP","B-ADVP"
+"IN","PRP","VBP","RB","JJ","IN","NN","IN","NN","NNP","VBZ","NNS","RB"
+"Because","I","m","somewhat","obsessed","with","shopping","at","Trader","Joe","s","people","always"
+"Albert Einstein Celebrity Profile Check out the latest Albert Einstein photo gallery, biography, pics, pictures, interviews, news, forums and blogs at Rotten Tomatoes!"
+"B-PP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","O","B-NP","B-PP","B-NP","I-NP"
+"IN","DT","NN","NN","NN","IN","DT","JJS","IN","DT","NN","NN","NN","NNS","NNS","NNS","NN","NNS","CC","NNS","IN","JJ","NNS"
+"Albert","Einstein","Celebrity","Profile","Check","out","the","latest","Albert","Einstein","photo","gallery","biography","pics","pictures","interviews","news","forums","and","blogs","at","Rotten","Tomatoes"
+"Harvey Milk - Wikipedia, the free encyclopedia"
+"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP"
+"NNP","NNP","NNP","DT","JJ","NN"
+"Harvey","Milk","Wikipedia","the","free","encyclopedia"
+"How to get weed out of your system Fast"
+"B-ADVP","B-VP","I-VP","I-VP","B-PP","B-PP","B-NP","I-NP"
+"WRB","TO","VB","VBN","IN","IN","PRP$","NN"
+"How","to","get","weed","out","of","your","system"
+"Trader Joe's List: Gluten Free List"
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","I-NP"
+"NN","NNP","VBZ","NNP","NNP","NNP","NN"
+"Trader","Joe","s","List","Gluten","Free","List"
+"Remaining rental income needs to be added to my profit and be reported as taxable profit. "
+"B-NP","I-NP","I-NP","B-VP","I-VP","I-VP","I-VP","B-PP","B-NP","I-NP","O","B-VP","I-VP","B-PP","B-NP","I-NP"
+"JJ","JJ","NN","VBZ","TO","VB","VBN","TO","PRP$","NN","CC","VB","VBN","IN","JJ","NN"
+"Remaining","rental","income","needs","to","be","added","to","my","profit","and","be","reported","as","taxable","profit"
+"I just want to rent a space for myself."
+"B-NP","B-ADVP","B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP"
+"PRP","RB","VBP","TO","VB","DT","NN","IN","PRP"
+"I","just","want","to","rent","a","space","for","myself"
+"Its classy design and the Mercedes name make it a very cool vehicle to drive. "
+"B-NP","I-NP","I-NP","O","B-NP","I-NP","I-NP","B-VP","B-NP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP"
+"PRP$","JJ","NN","CC","DT","NNP","NN","VBP","PRP","DT","RB","JJ","NN","TO","NN"
+"Its","classy","design","and","the","Mercedes","name","make","it","a","very","cool","vehicle","to","drive"
+"He developed the general theory of relativity, one of the two pillars of modern physics (alongside quantum "
+"B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP"
+"PRP","VBD","DT","JJ","NN","IN","NN","CD","IN","DT","CD","NNS","IN","JJ","NNS","IN","NN"
+"He","developed","the","general","theory","of","relativity","one","of","the","two","pillars","of","modern","physics","alongside","quantum"
+"My employer withheld the taxes from my pay. ."
+"B-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"PRP$","NN","VBD","DT","NNS","IN","PRP$","NN"
+"My","employer","withheld","the","taxes","from","my","pay"
+"Remember The Milk - Services / Remember The Milk for Email"
+"B-VP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-PP","B-NP"
+"VB","DT","NN","NNP","NNP","DT","NN","IN","NN"
+"Remember","The","Milk","Services","Remember","The","Milk","for","Email"
+"To store goods for my retail business I rent some space."
+"B-VP","I-VP","B-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP"
+"TO","VB","NNS","IN","PRP$","JJ","NN","PRP","VB","DT","NN"
+"To","store","goods","for","my","retail","business","I","rent","some","space"
+"As a 26 year old patent clerk, Albert Einstein revolutionized science in 1905 when he published five new theories, including the theory of relativity."
+"B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-VP","B-NP","B-PP","B-NP","B-ADVP","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP"
+"IN","DT","CD","NN","JJ","NN","NN","NNP","NNP","VBD","NN","IN","CD","WRB","PRP","VBD","CD","JJ","NNS","VBG","DT","NN","IN","NN"
+"As","a","26","year","old","patent","clerk","Albert","Einstein","revolutionized","science","in","1905","when","he","published","five","new","theories","including","the","theory","of","relativity"
+"Citizens Living Abroad Taxes: Frequently Asked Questions"
+"B-NP","I-NP","I-NP","I-NP","B-ADVP","B-VP","B-NP"
+"NNPS","NNP","NNP","NNP","RB","VBD","NNS"
+"Citizens","Living","Abroad","Taxes","Frequently","Asked","Questions"
+"My mom wants me to get some groceries... - Page 3 - Social Anxiety"
+"B-NP","I-NP","B-VP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"PRP$","NN","VBZ","PRP","TO","VB","DT","NNS","NN","CD","NNP","NNP"
+"My","mom","wants","me","to","get","some","groceries","Page","3","Social","Anxiety"
+"Albert Einstein (1879 1955) was born in Germany and became an American citizen in 1940."
+"B-NP","I-NP","I-NP","I-NP","B-VP","I-VP","B-PP","B-NP","O","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP"
+"NNP","NNP","CD","CD","VBD","VBN","IN","NNP","CC","VBD","DT","JJ","NN","IN","CD"
+"Albert","Einstein","1879","1955","was","born","in","Germany","and","became","an","American","citizen","in","1940"
+"IRS Withholding Calculator"
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"IRS","Withholding","Calculator"
+"Tax Foundation's Tax Policy Calculator"
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NN","NN","NNS","NN","NN","NN"
+"Tax","Foundation","s","Tax","Policy","Calculator"
+"I rent an office space."
+"B-NP","B-VP","B-NP","I-NP","I-NP"
+"PRP","VB","DT","NN","NN"
+"I","rent","an","office","space"
+"Jermaine Jones - Milk Chocolate Shitcanned | Vote for the Worst"
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP","NN","IN","DT","JJS"
+"Jermaine","Jones","Milk","Chocolate","Shitcanned","Vote","for","the","Worst"
+"His father was a featherbed salesman."
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP"
+"PRP$","NN","VBD","DT","JJ","NN"
+"His","father","was","a","featherbed","salesman"
+"I need to add the rental income to my profit."
+"B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"PRP","VBP","TO","VB","DT","JJ","NN","TO","PRP$","NN"
+"I","need","to","add","the","rental","income","to","my","profit"
+"What will happen to my information if I log out before I submit my request? . Can I use Web Pay if my last name changed in 2011? . an electronic payment from your bank account to pay your personal and business income taxes. . Estimated tax; Tax return; Billing notice; Extension; Notice of Proposed Assessment; Tax ."
+"B-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-SBAR","B-NP","B-VP","B-PRT","B-SBAR","B-NP","B-VP","B-NP","I-NP","O","B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-PP","B-NP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"WP","MD","VB","TO","PRP$","NN","IN","PRP","VBP","RP","IN","PRP","VBP","PRP$","NN","MD","PRP","VB","NNP","NN","IN","PRP$","JJ","NN","VBD","IN","CD","DT","JJ","NN","IN","PRP$","NN","NN","TO","VB","PRP$","JJ","CC","NN","NN","NNS","JJ","NN","NNP","NN","NNP","NN","NNP","NNP","IN","NNP","NNP","NNP"
+"What","will","happen","to","my","information","if","I","log","out","before","I","submit","my","request","Can","I","use","Web","Pay","if","my","last","name","changed","in","2011","an","electronic","payment","from","your","bank","account","to","pay","your","personal","and","business","income","taxes","Estimated","tax","Tax","return","Billing","notice","Extension","Notice","of","Proposed","Assessment","Tax"
+"I neither calculate deduction of individual or business tax."
+"B-NP","B-ADVP","B-VP","B-NP","B-PP","B-NP","I-NP","I-NP","I-NP"
+"PRP","RB","VBP","NN","IN","JJ","CC","NN","NN"
+"I","neither","calculate","deduction","of","individual","or","business","tax"
+"I need to add my rental income to my profits, but subtract rental expenses such as repair from it. "
+"B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","O","B-VP","B-NP","I-NP","B-PP","I-PP","B-NP","B-SBAR","B-NP"
+"PRP","VBP","TO","VB","PRP$","JJ","NN","TO","PRP$","NNS","CC","VB","JJ","NNS","JJ","IN","NN","IN","PRP"
+"I","need","to","add","my","rental","income","to","my","profits","but","subtract","rental","expenses","such","as","repair","from","it"
+"This car provides you a very good mileage."
+"B-NP","I-NP","B-VP","B-NP","B-NP","I-NP","I-NP","I-NP"
+"DT","NN","VBZ","PRP","DT","RB","JJ","NN"
+"This","car","provides","you","a","very","good","mileage"
+"Albert Einstein College of Medicine | Medical Education .."
+"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"NNP","NNP","NNP","IN","NNP","NNP","NNP"
+"Albert","Einstein","College","of","Medicine","Medical","Education"
+"CST- Stamford · http://Www.facebook.com/dcomplex12 ."
+"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
+"NN","NN","NN","VBD","NN","NN"
+"CST","Stamford","http","Www","facebook","com"
+"Can I share lists with other Remember The Milk users?"
+"O","B-NP","B-VP","B-NP","B-PP","B-NP","I-NP","B-NP","I-NP","I-NP"
+"MD","PRP","VB","NNS","IN","JJ","NNP","DT","NN","NNS"
+"Can","I","share","lists","with","other","Remember","The","Milk","users"
+"Einstein, Albert Encyclopædia Britannica, Inc."
+"B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP"
+"Einstein","Albert","Encyclopædia","Britannica","Inc"
+"Albert Einstein Facts & Summary HISTORY.com ."
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP","NN"
+"Albert","Einstein","Facts","Summary","HISTORY","com"
+"Albert Einstein Quotes (Author of Relativity) Goodreads ."
+"B-NP","B-ADVP","B-VP","B-NP","B-PP","B-NP","I-NP"
+"NN","RB","VBZ","NN","IN","NN","NNS"
+"Albert","Einstein","Quotes","Author","of","Relativity","Goodreads"
+"While the app doesn't currently . ."
+"B-SBAR","B-NP","I-NP","I-NP","I-NP","O"
+"IN","DT","NN","NN","NN","RB"
+"While","the","app","doesn","t","currently"
+"It's sometimes hard to function and hard to want to get out of bed . Posted in baby amelia 3 Comments ? . I realized that time has gotten away from me, for tomorrow Amelia would have been four weeks old. . by now, and hopefully, had become an accomplished milk-cow. . ."
+"B-NP","B-VP","B-ADVP","B-ADJP","B-VP","I-VP","O","B-ADVP","B-VP","I-VP","I-VP","I-VP","B-PP","B-PP","B-NP","B-VP","B-PP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","B-SBAR","B-NP","B-VP","I-VP","B-ADVP","B-PP","B-NP","B-PP","B-NP","I-NP","B-VP","I-VP","I-VP","B-NP","I-NP","B-ADJP","B-PP","B-ADVP","O","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","I-NP"
+"PRP","VBZ","RB","JJ","TO","VB","CC","JJ","TO","VB","TO","VB","IN","IN","NN","VBN","IN","NN","NNS","CD","NNS","PRP","VBD","IN","NN","VBZ","VBN","RB","IN","PRP","IN","NN","NNP","MD","VB","VBN","CD","NNS","JJ","IN","RB","CC","RB","VBD","VBN","DT","JJ","NN","NN"
+"It","s","sometimes","hard","to","function","and","hard","to","want","to","get","out","of","bed","Posted","in","baby","amelia","3","Comments","I","realized","that","time","has","gotten","away","from","me","for","tomorrow","Amelia","would","have","been","four","weeks","old","by","now","and","hopefully","had","become","an","accomplished","milk","cow"
+"For the United States income tax return, you will have several options available to you regarding claiming a . ."
+"B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","B-ADJP","B-PP","B-NP","B-VP","I-VP","B-UCP"
+"IN","DT","NNP","NNPS","NN","NN","NN","PRP","MD","VB","JJ","NNS","JJ","TO","PRP","VBG","VBG","DT"
+"For","the","United","States","income","tax","return","you","will","have","several","options","available","to","you","regarding","claiming","a"
+"remember to buy milk tomorrow from 3 to jones"
+"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"VB","TO","VB","NN","NN","IN","CD","TO","NNS"
+"remember","to","buy","milk","tomorrow","from","3","to","jones"
+"Can I get auto focus lens for digital camera"
+"O","B-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"MD","PRP","VB","NN","NN","NNS","IN","JJ","NN"
+"Can","I","get","auto","focus","lens","for","digital","camera"
+"I have to claim it as a profit in my tax forms."
+"B-NP","B-VP","I-VP","I-VP","B-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"PRP","VBP","TO","VB","PRP","IN","DT","NN","IN","PRP$","NN","NNS"
+"I","have","to","claim","it","as","a","profit","in","my","tax","forms"
+"Albert Einstein Simple English Wikipedia, the free .."
+"B-NP","I-NP","I-NP","I-NP","I-NP","B-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP","DT","JJ"
+"Albert","Einstein","Simple","English","Wikipedia","the","free"
+"Going to Work Abroad - Revenue Commissioners"
+"B-VP","I-VP","I-VP","B-NP","I-NP","I-NP"
+"VBG","TO","VB","RB","NN","NNS"
+"Going","to","Work","Abroad","Revenue","Commissioners"
+"Learn his theories, find facts and quotes from the man with an IQ of 160."
+"B-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP"
+"VB","PRP$","NNS","VBP","NNS","CC","NNS","IN","DT","NN","IN","DT","NNP","IN","CD"
+"Learn","his","theories","find","facts","and","quotes","from","the","man","with","an","IQ","of","160"
+"Albert Einstein Biography ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","Biography"
+"To run my business, I have to rent an office."
+"B-VP","I-VP","B-NP","I-NP","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP"
+"TO","VB","PRP$","NN","PRP","VBP","TO","VB","DT","NN"
+"To","run","my","business","I","have","to","rent","an","office"
+"But what do we know about his Jewish self identification?"
+"O","B-NP","B-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP","I-NP"
+"CC","WP","VBP","PRP","VB","IN","PRP$","JJ","NN","NN"
+"But","what","do","we","know","about","his","Jewish","self","identification"
+"However, when I repair my house, I can deduct the repair expense from my rental income."
+"B-ADVP","B-ADVP","B-NP","B-VP","B-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"RB","WRB","PRP","VBD","PRP$","NN","PRP","MD","VB","DT","NN","NN","IN","PRP$","JJ","NN"
+"However","when","I","repair","my","house","I","can","deduct","the","repair","expense","from","my","rental","income"
+"Albert began reading and "
+"B-NP","B-VP","I-VP","O"
+"NNP","VBD","VBG","CC"
+"Albert","began","reading","and"
+"I do not want to wait till I am sick to buy health insurance."
+"B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","B-SBAR","B-NP","B-VP","B-ADJP","B-VP","I-VP","B-NP","I-NP"
+"PRP","VBP","RB","VB","TO","VB","IN","PRP","VBP","JJ","TO","VB","NN","NN"
+"I","do","not","want","to","wait","till","I","am","sick","to","buy","health","insurance"
+"Once you get home you are going to immidiatly start drinking the . . for four hours. remember no toxins 48 hours before and no alcohol . ."
+"O","B-NP","B-VP","B-NP","B-NP","B-VP","I-VP","I-VP","I-VP","I-VP","I-VP","B-NP","I-NP","B-NP","I-NP","B-VP","B-NP","I-NP","B-NP","I-NP","O","O","B-NP","I-NP"
+"RB","PRP","VBP","NN","PRP","VBP","VBG","TO","RB","VB","VBG","DT","NN","CD","NNS","VBP","DT","NNS","CD","NNS","IN","CC","DT","NN"
+"Once","you","get","home","you","are","going","to","immidiatly","start","drinking","the","for","four","hours","remember","no","toxins","48","hours","before","and","no","alcohol"
+"Remember the Milk is a web application with numerous mobile app . complete today will automatically get moved to the tomorrow column. ."
+"B-VP","B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","B-VP","I-VP","I-VP","I-VP","B-PP","B-NP","I-NP","I-NP"
+"VB","DT","NN","VBZ","DT","NN","NN","IN","JJ","JJ","NN","JJ","NN","MD","RB","VB","VBN","TO","DT","NN","NN"
+"Remember","the","Milk","is","a","web","application","with","numerous","mobile","app","complete","today","will","automatically","get","moved","to","the","tomorrow","column"
+"I cannot conceive of a god who rewards and punishes his creatures or has a will of the kind that we experience in ourselves."
+"B-NP","B-VP","I-VP","B-PP","B-NP","I-NP","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","O","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP","B-SBAR","B-NP","B-VP","B-PP","B-NP"
+"PRP","MD","VB","IN","DT","NN","WP","VBZ","CC","VBZ","PRP$","NNS","CC","VBZ","DT","NN","IN","DT","NN","IN","PRP","VBP","IN","NNS"
+"I","cannot","conceive","of","a","god","who","rewards","and","punishes","his","creatures","or","has","a","will","of","the","kind","that","we","experience","in","ourselves"
+"Questions and Answers on Albert Einstein."
+"B-NP","I-NP","I-NP","B-PP","B-NP","I-NP"
+"NNS","CC","NNS","IN","NNP","NNP"
+"Questions","and","Answers","on","Albert","Einstein"
+"To calculate my net income, I subtract from revenue my rental business expense."
+"B-VP","I-VP","B-NP","I-NP","I-NP","B-NP","B-VP","B-PP","B-NP","B-NP","I-NP","I-NP","I-NP"
+"TO","VB","PRP$","JJ","NN","PRP","VBD","IN","NN","PRP$","JJ","NN","NN"
+"To","calculate","my","net","income","I","subtract","from","revenue","my","rental","business","expense"
+"For this achievement "
+"B-PP","B-NP","I-NP"
+"IN","DT","NN"
+"For","this","achievement"
+"Albert Einstein reinterpreted the inner workings of nature, the very essence of light, time, energy and gravity."
+"B-NP","I-NP","B-VP","B-NP","I-NP","I-NP","B-PP","B-NP","B-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","VBD","DT","JJ","NNS","IN","NN","DT","JJ","NN","IN","JJ","NN","NN","CC","NN"
+"Albert","Einstein","reinterpreted","the","inner","workings","of","nature","the","very","essence","of","light","time","energy","and","gravity"
+"Montefiore Medical Center, the University Hospital for Albert Einstein College of Medicine, is a premier academic medical center and nationally recognized leader in "
+"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-PP","B-NP","B-PP","B-NP","B-PP","B-NP","B-VP","B-NP","I-NP","I-NP","I-NP","I-NP","O","B-VP","I-VP","B-NP","B-PP"
+"NN","JJ","NN","DT","NN","NN","IN","NN","IN","NN","IN","NN","VBZ","DT","JJ","JJ","JJ","NN","CC","RB","VBD","NN","IN"
+"Montefiore","Medical","Center","the","University","Hospital","for","Albert","Einstein","College","of","Medicine","is","a","premier","academic","medical","center","and","nationally","recognized","leader","in"
+"Albert Einstein (Author of Relativity) ."
+"B-NP","I-NP","I-NP","B-PP","B-NP"
+"NNP","NNP","NNP","IN","NN"
+"Albert","Einstein","Author","of","Relativity"
+"If you live in the UK permanently you'll pay tax on overseas income."
+"B-SBAR","B-NP","B-VP","B-PP","B-NP","I-NP","B-ADVP","B-NP","B-VP","I-VP","B-NP","B-PP","B-NP","I-NP"
+"IN","PRP","VBP","IN","DT","NNP","RB","PRP","MD","VB","NN","IN","JJ","NN"
+"If","you","live","in","the","UK","permanently","you","ll","pay","tax","on","overseas","income"
+"Use Tax Questions and Answers"
+"B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNPS","CC","NNPS"
+"Use","Tax","Questions","and","Answers"
+"Albert Einstein The Sanctuary Network, Sanctuary For All ."
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-UCP","I-UCP"
+"NNP","NNP","NNP","NNP","NNP","NNP","IN","DT"
+"Albert","Einstein","The","Sanctuary","Network","Sanctuary","For","All"
+"20 Best Online To Do List Apps for Freelancers"
+"B-NP","I-NP","I-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP"
+"CD","NNP","NNP","TO","VB","NNP","NNP","IN","NNS"
+"20","Best","Online","To","Do","List","Apps","for","Freelancers"
+"Even if you avoid U.S."
+"B-SBAR","I-SBAR","B-NP","B-VP","B-NP","I-NP"
+"RB","IN","PRP","VBP","NNP","NNP"
+"Even","if","you","avoid","U","S"
+"How to deduct repair expense from rental income."
+"B-ADVP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"WRB","TO","VB","NN","NN","IN","JJ","NN"
+"How","to","deduct","repair","expense","from","rental","income"
+"Get Organized with Remember the Milk"
+"B-VP","B-NP","I-PRT","B-VP","B-NP","I-NP"
+"VB","NNP","IN","VB","DT","NN"
+"Get","Organized","with","Remember","the","Milk"
+"This series of questions and answers refers specifically to the use tax incurred on . For taxpayers who do not have records to document their use tax liability, the department will estimate liability. . Individual Income Tax Return, on or before April 15 of the following year (for tax . Can I file and pay my use tax electronically? ."
+"B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-VP","B-ADJP","B-PP","B-NP","I-NP","I-NP","B-VP","B-PP","B-PP","B-NP","B-NP","B-VP","I-VP","I-VP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","B-NP","I-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","I-NP","B-PP","I-PP","I-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP","B-PP","B-NP","O","B-NP","B-VP","I-VP","I-VP","B-NP","I-NP","I-NP","O"
+"DT","NN","IN","NNS","CC","NNS","VBZ","RB","TO","DT","NN","NN","VBN","IN","IN","NNS","WP","VBP","RB","VB","NNS","TO","VB","PRP$","NN","NN","NN","DT","NN","MD","VB","NN","NNP","NNP","NNP","NN","IN","CC","IN","NNP","CD","IN","DT","JJ","NN","IN","NN","MD","PRP","VB","CC","VB","PRP$","NN","NN","RB"
+"This","series","of","questions","and","answers","refers","specifically","to","the","use","tax","incurred","on","For","taxpayers","who","do","not","have","records","to","document","their","use","tax","liability","the","department","will","estimate","liability","Individual","Income","Tax","Return","on","or","before","April","15","of","the","following","year","for","tax","Can","I","file","and","pay","my","use","tax","electronically"
+"I rent some space for my business."
+"B-NP","B-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"PRP","VB","DT","NN","IN","PRP$","NN"
+"I","rent","some","space","for","my","business"
+"His insights fundamentally changed the way we look at "
+"B-NP","I-NP","B-ADVP","B-VP","B-NP","I-NP","B-NP","B-VP","B-PP"
+"PRP$","NNS","RB","VBD","DT","NN","PRP","VBP","IN"
+"His","insights","fundamentally","changed","the","way","we","look","at"
+"D-Jones (dcomplex12) on Twitter"
+"B-NP","I-NP","I-NP","B-PP","B-NP"
+"NNP","NNP","NN","IN","NN"
+"D","Jones","dcomplex12","on","Twitter"
+"Quotations by Albert Einstein, German Physicist, Born March 14, 1879."
+"B-NP","B-PP","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","O"
+"NNS","IN","NNP","NNP","NNP","NNP","NNP","NNP","CD","CD"
+"Quotations","by","Albert","Einstein","German","Physicist","Born","March","14","1879"
+"CST- Stamford � http://Www.facebook.com/dcomplex12 ."
+"B-NP","I-NP","I-NP","B-VP","B-NP","I-NP"
+"NN","NN","NN","VBD","NN","NN"
+"CST","Stamford","http","Www","facebook","com"
+"Way to minimize medical expense for my daughter"
+"B-NP","B-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"NN","TO","VB","JJ","NN","IN","PRP$","NN"
+"Way","to","minimize","medical","expense","for","my","daughter"
+"One story Einstein liked to "
+"B-NP","I-NP","B-ADVP","I-VP"
+"CD","NN","RB","VBD"
+"One","story","Einstein","liked"
+"remember to buy milk tomorrow from for d jones"
+"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-PP","B-NP","I-NP"
+"VB","TO","VB","NN","NN","IN","IN","NN","NNS"
+"remember","to","buy","milk","tomorrow","from","for","d","jones"
+"Albert Einstein Biographical ."
+"B-NP","I-NP","I-NP"
+"NNP","NNP","NNP"
+"Albert","Einstein","Biographical"
+"While you will need your W-2 or a substitute, you can figure out your tax obligation . final pay stub for a job (if you are no longer employed) or the last pay stub for the year . the year, you can estimate your income and taxes from a single pay stub. . However, if you worked at the same job the previous year, you can use the . ."
+"B-SBAR","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","O","B-NP","I-NP","B-NP","B-VP","I-VP","B-PRT","B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-SBAR","B-NP","B-VP","B-ADVP","I-ADVP","B-VP","O","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-NP","I-NP","B-NP","B-VP","I-VP","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","I-NP","I-NP","B-ADVP","B-SBAR","B-NP","B-VP","B-PP","B-NP","I-NP","I-NP","B-NP","I-NP","I-NP","B-NP","B-VP","I-VP","B-NP"
+"IN","PRP","MD","VB","PRP$","NN","CD","CC","DT","NN","PRP","MD","VB","RP","PRP$","NN","NN","JJ","NN","NN","IN","DT","NN","IN","PRP","VBP","RB","RB","VBN","CC","DT","JJ","NN","NN","IN","DT","NN","DT","NN","PRP","MD","VB","PRP$","NN","CC","NNS","IN","DT","JJ","NN","NN","RB","IN","PRP","VBD","IN","DT","JJ","NN","DT","JJ","NN","PRP","MD","VB","DT"
+"While","you","will","need","your","W","2","or","a","substitute","you","can","figure","out","your","tax","obligation","final","pay","stub","for","a","job","if","you","are","no","longer","employed","or","the","last","pay","stub","for","the","year","the","year","you","can","estimate","your","income","and","taxes","from","a","single","pay","stub","However","if","you","worked","at","the","same","job","the","previous","year","you","can","use","the"
+"I rent out a first floor unit of my house to a travel business."
+"B-NP","B-VP","B-PRT","B-NP","I-NP","I-NP","I-NP","B-PP","B-NP","I-NP","B-PP","B-NP","I-NP","I-NP"
+"PRP","VB","RP","DT","JJ","NN","NN","IN","PRP$","NN","TO","DT","NN","NN"
+"I","rent","out","a","first","floor","unit","of","my","house","to","a","travel","business"
+"Albert Einstein New World Encyclopedia ."
+"B-NP","I-NP","I-NP","I-NP","I-NP"
+"NNP","NNP","NNP","NNP","NNP"
+"Albert","Einstein","New","World","Encyclopedia"
+"Einstein s Big Idea."
+"B-NP","B-VP","B-NP","I-NP"
+"NNP","VBZ","JJ","NN"
+"Einstein","s","Big","Idea"
+"remember to buy milk tomorrow for for details"
+"B-VP","I-VP","I-VP","B-NP","I-NP","B-PP","B-NP","I-NP"
+"VB","TO","VB","NN","NN","IN","NN","NNS"
+"remember","to","buy","milk","tomorrow","for","for","details"
+"Albert Einstein - Wikipedia, the free encyclopedia"
+"B-NP","I-NP","I-NP","B-NP","I-NP","I-NP"
+"NNP","NNP","NNP","DT","JJ","NN"
+"Albert","Einstein","Wikipedia","the","free","encyclopedia"
+"Tagged - Damaya Lady D Jones's Profile"
+"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
+"JJ","NNP","NNP","NNP","NNP","NNS","NN"
+"Tagged","Damaya","Lady","D","Jones","s","Profile"
+"Hardly anyone will end up paying the tax when the health reform law takes full effect in 2014."
+"B-ADVP","B-NP","B-VP","I-VP","B-ADVP","B-VP","B-NP","I-NP","B-ADVP","B-NP","I-NP","I-NP","I-NP","B-VP","B-NP","I-NP","B-PP","B-NP"
+"RB","NN","MD","VB","RP","VBG","DT","NN","WRB","DT","NN","NN","NN","VBZ","JJ","NN","IN","CD"
+"Hardly","anyone","will","end","up","paying","the","tax","when","the","health","reform","law","takes","full","effect","in","2014"
"Big Paychecks, Tiny Tax Burdens: How 21,000 Wealthy Americans"
"B-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP","I-NP"
"NNP","NNP","NNP","NNP","NNPS","NNP","CD","CD","NNP","NNPS"
