Recently, I watched Hinton’s talk on recent developments in deep learning. Main points are as below:
1. Replace sigmoid function with rectified linear function: easily for training and test, plus efficient.
2. Dropout training and test could improve accuracy significantly, becasue this is basicly aggregating different highly regularized deep learning model by a geometric mean.
This might be a standard recipe for current deep learning. Based on this recipe, several students of his have won many Kaggle chagllenge.