Recently I have been working on my first (serious) Kaggle competition and spent a lot of time reading papers and tech blogs. Some of them are just genius’ work and I would like to write a small piece of digest here to keep the wisdom lingering around as long as possible.
Ensembling
Model Ensembling is a powerful technique to improve accuracy on a variety of Machine Leanring tasks, especially Kaggle competitions. It reduces generalization error with no brainstorming and trivial computational cost. The idea is to simply mix diffenrent models up into (weighted-)average predictions. The key to success is to choose models that have as least correlation as possible.
LSTM Networks
LSTM Networks, a special kind of RNN, are capable of learning long-term dependencies. LSTM is great in not only time series predictions but also any sequence-related problems, i.e. semantic analysis in NLP.
A common LSTM unit is composed of a cell, a forget gate, an input gate and an output gate. GRU is a famous variation of LSTM unit and worths a check if you are pushing for better accuracy and effieciency.