Blog: Evolving Programming Models for Massively Scalable Scientific Applications

18 Nov 2025 - Douglas Thain

Last week, I gave a talk at Stevens Institute of Technology on Evolving Programming Models for Massively Scalable Scientific Applications.

Here is a quick summary: The TaskVine workflow system is designed to enable the construction of applications that scale up to tens of thousands of cluster nodes. Over the last five years, we have built up a series of programming models for such workflows, starting with a low-level API for constructing graphs, adding a FaaS like interface for function deployment, integrating dataflow languages like Dask and Parsl, and most recently creating Dynamic Data Reduction, a compact framework for generating graphs on the fly:

Thank you to my colleague, Dr. Hao Wang, who was kind enough to host me for the day, introduce my to his students, and show off some monster GPU machines:

---

« Prev: Wrangling Complex Notebook Workflows with Floability