Session: The 5th Deep Learning on Supercomputers Workshop

Towards a Scalable and Distributed Infrastructure for Deep Learning Applications

Wednesday, November 11, 2020

3:00 PM – 3:30 PM ET

Channel: Track 7

Author/Presenter(s)

BH

Bita Hasheminezhad

Louisiana State University, Center for Computation and Technology, United States of America
SS

Shahrzad Shirzad

Louisiana State University, Center for Computation and Technology, United States of America
NW

Nanmiao Wu

Louisiana State University, Center for Computation and Technology, United States of America
Patrick Diehl

Louisiana State University; Louisiana State University, Center for Computation and Technology, United States of America
HS

Hannes Schulz

Microsoft Research, Montreal, Canada
Hartmut Kaiser

Louisiana State University; Louisiana State University, Center for Computation and Technology, United States of America

Although recent scaling up approaches to train deep neural networks have proven to be effective, the computational intensity of large and complex models, as well as the availability of large-scale datasets require deep learning frameworks to utilize scaling out techniques. Parallelization approaches and distribution requirements are not considered in the primary designs of most available distributed deep learning frameworks and most of them still are not able to perform effective and efficient fine-grained inter-node communication. We present Phylanx, which has the potential to alleviate these shortcomings. Phylanx presents a productivity-oriented frontend where user Python code is translated to a futurized execution tree that can be executed efficiently on multiple nodes using the C++ standard library for parallelism and concurrency (HPX), leveraging fine-grained threading and an active messaging task-based runtime system.