Debugging Tips for Neural Networks

Often times the bottleneck in a neural network-based project isn't the network implementation. Rather, after you've written all the code and tried a whole bunch of hyperparameter configurations, sometimes the network will just not work. I've been there before. After some time dealing with finicky networks, I've collected a few methods that have helped me debug them. These methods aren't a guarantee of any sort — even if you do everything I suggest, there's a chance your network will still be broken. I hope, however, that these tips will in the long run decrease the time you spend debugging your neural networks.