Adafactor

Adafactor is an optimization algorithm for training deep learning models, particularly well-suited for large-scale distributed settings. It is designed to address some limitations of traditional optimization algorithms, such as adaptive learning rate methods like Adam and RMSprop. Adafactor was introduced by Noam Shazeer and Mitchell Stern in their paper titled “Adafactor: Adaptive Learning Rates with Sublinear Memory Cost” in 2018.

External Links

Please note that the content of external links are not endorsed or verified by us and can change with no notice. Use at your own risk.