Oak Ridge National Laboratory, United States of America
Neural architecture search (NAS) is a popular topic in deep learning that focuses on optimizing the architecture of a deep network for a particular problem. In practice, the single deep network that gives optimal performance is not typically used, as it may be limited in its knowledge of the data’s distribution or poorly fitted to the training data. Instead, an ensemble of multiple networks produced by the NAS is used in order to boost results. High performance computing offers the opportunity to produce many more models than would otherwise be possible, and thus provides an excellent opportunity to not only optimize individual network structures, but also ensembles of network structures that perform well together on problems of interest. Neural network ensembles combine the outputs of multiple deep neural network classifiers with different parameters that have been trained on the same data and have been demonstrated to offer significantly improved prediction accuracies over individual models. The diversity of network structures produced by NAS drives a natural bias towards diversity of predictions produced by the individual networks. This results in an ensemble that performs better than one that simply contains duplicates of the best network architecture retrained to have unique weights.