| Apache Accumulo Wikipedia Search Example | |
| This project contains a sample application for ingesting and querying wikipedia data. | |
| Ingest | |
| ------ | |
| Prerequisites | |
| ------------- | |
| 1. Accumulo, Hadoop, and ZooKeeper must be installed and running | |
| 2. One or more wikipedia dump files (http://dumps.wikimedia.org/backup-index.html) placed in an HDFS directory. | |
| You will want to grab the files with the link name of pages-articles.xml.bz2 | |
| 3. Though not strictly required, the ingest will go more quickly if the files are decompressed: | |
| $ bunzip2 < enwiki-*-pages-articles.xml.bz2 | hadoop fs -put - /wikipedia/enwiki-pages-articles.xml | |
| INSTRUCTIONS | |
| ------------ | |
| 1. Copy the ingest/conf/wikipedia.xml.example to ingest/conf/wikipedia.xml and change it to specify Accumulo information. | |
| 2. Copy the ingest/lib/wikisearch-*.jar and ingest/lib/protobuf*.jar to $ACCUMULO_HOME/lib/ext | |
| 3. Then run ingest/bin/ingest.sh with one argument (the name of the directory in HDFS where the wikipedia XML | |
| files reside) and this will kick off a MapReduce job to ingest the data into Accumulo. | |
| Query | |
| ----- | |
| Prerequisites | |
| ------------- | |
| 1. The query software was tested using JBoss AS 6. Install this unless you feel like messing with the installation. | |
| NOTE: Ran into a bug (https://issues.jboss.org/browse/RESTEASY-531) that did not allow an EJB3.1 war file. The | |
| workaround is to separate the RESTEasy servlet from the EJBs by creating an EJB jar and a WAR file. | |
| INSTRUCTIONS | |
| ------------- | |
| 1. Copy the query/src/main/resources/META-INF/ejb-jar.xml.example file to | |
| query/src/main/resources/META-INF/ejb-jar.xml. Modify to the file to contain the same | |
| information that you put into the wikipedia.xml file from the Ingest step above. | |
| 2. Re-build the query distribution by running 'mvn package assembly:single' in the top-level directory. | |
| 3. Untar the resulting file in the $JBOSS_HOME/server/default directory. | |
| $ cd $JBOSS_HOME/server/default | |
| $ tar -xzf $ACCUMULO_HOME/src/examples/wikisearch/query/target/wikisearch-query*.tar.gz | |
| This will place the dependent jars in the lib directory and the EJB jar into the deploy directory. | |
| 4. Next, copy the wikisearch*.war file in the query-war/target directory to $JBOSS_HOME/server/default/deploy. | |
| 5. Start JBoss ($JBOSS_HOME/bin/run.sh) | |
| 6. Use the Accumulo shell and give the user permissions for the wikis that you loaded, for example: | |
| setauths -u <user> -s all,enwiki,eswiki,frwiki,fawiki | |
| 7. Copy the following jars to the $ACCUMULO_HOME/lib/ext directory from the $JBOSS_HOME/server/default/lib directory: | |
| commons-lang*.jar | |
| kryo*.jar | |
| minlog*.jar | |
| commons-jexl*.jar | |
| google-collections*.jar | |
| 8. Copy the $JBOSS_HOME/server/default/deploy/wikisearch-query*.jar to $ACCUMULO_HOME/lib/ext. | |
| 9. At this point you should be able to open a browser and view the page: http://localhost:8080/accumulo-wikisearch/ui/ui.jsp. | |
| You can issue the queries using this user interface or via the following REST urls: <host>/accumulo-wikisearch/rest/Query/xml, | |
| <host>/accumulo-wikisearch/rest/Query/html, <host>/accumulo-wikisearch/rest/Query/yaml, or <host>/accumulo-wikisearch/rest/Query/json. | |
| There are two parameters to the REST service, query and auths. The query parameter is the same string that you would type | |
| into the search box at ui.jsp, and the auths parameter is a comma-separated list of wikis that you want to search (i.e. | |
| enwiki,frwiki,dewiki, etc. Or you can use all) |