Summary

TABLE OF CONTENTS

• Getting Started

  • Installation
  • Install as permanent functions
  • Input Format

• Tips for Effective Hivemall

  • Explicit addBias() for better prediction
  • Use rand_amplify() to better prediction results
  • Real-time Prediction on RDBMS
  • Ensemble learning for stable prediction
  • Mixing models for a better prediction convergence (MIX server)
  • Run Hivemall on Amazon Elastic MapReduce

• General Hive/Hadoop tips

  • Adding rowid for each row
  • Hadoop tuning for Hivemall

• Troubleshooting

  • OutOfMemoryError in training
  • SemanticException Generate Map Join Task Error: Cannot serialize object
  • Asterisk argument for UDTF does not work
  • The number of mappers is less than input splits in Hadoop 2.x
  • Map-side Join causes ClassCastException on Tez

Part II - Generic Features

• List of generic Hivemall functions
• Efficient Top-K query processing
• English/Japanese Text Tokenizer

Part III - Feature Engineering

• Feature Scaling

• Feature Hashing

• TF-IDF calculation

• FEATURE TRANSFORMATION

  • Vectorize Features
  • Quantify non-number features

Part IV - Evaluation

• Statistical evaluation of a prediction model

• Data Generation

  • Logistic Regression data generation

Part V - Binary classification

• a9a Tutorial

  • Data preparation
  • Logistic Regression
  • Mini-batch Gradient Descent

• News20 Tutorial

  • Data preparation
  • Perceptron, Passive Aggressive
  • CW, AROW, SCW
  • AdaGradRDA, AdaGrad, AdaDelta

• KDD2010a Tutorial

  • Data preparation
  • PA, CW, AROW, SCW

• KDD2010b Tutorial

  • Data preparation
  • AROW

• Webspam Tutorial

  • Data pareparation
  • PA1, AROW, SCW

• Kaggle Titanic Tutorial

Part VI - Multiclass classification

• News20 Multiclass Tutorial

  • Data preparation
  • Data preparation for one-vs-the-rest classifiers
  • PA
  • CW, AROW, SCW
  • Ensemble learning
  • one-vs-the-rest classifier

• Iris Tutorial

  • Data preparation
  • SCW
  • RandomForest

Part VII - Regression

• E2006-tfidf regression Tutorial

  • Data preparation
  • Passive Aggressive, AROW

• KDDCup 2012 track 2 CTR prediction Tutorial

  • Data preparation
  • Logistic Regression, Passive Aggressive
  • Logistic Regression with Amplifier
  • AdaGrad, AdaDelta

Part VIII - Recommendation

• Collaborative Filtering

  • Item-based Collaborative Filtering

• News20 related article recommendation Tutorial

  • Data preparation
  • LSH/Minhash and Jaccard Similarity
  • LSH/Minhash and Brute-Force Search
  • kNN search using b-Bits Minhash

• MovieLens movie recommendation Tutorial

  • Data preparation
  • Matrix Factorization
  • Factorization Machine
  • 10-fold Cross Validation (Matrix Factorization)

Part IX - Anomaly Detection

• Outlier Detection using Local Outlier Factor (LOF)

Part X - External References

• Hivemall on Apache Spark
• Hivemall on Apache Pig