Accelerating Python Applications with Dask and ProxyStore

Understand

Applications are increasingly written as dynamic workflows underpinned by an execution framework that manages asynchronous computations across distributed hardware.

• However, execution frameworks typically offer one-size-fits-all solutions for data flow management, which can restrict performance and scalability.
• ProxyStore, a middleware layer that optimizes data flow via an advanced pass-by-reference paradigm, has shown to be an effective mechanism for addressing these limitations.
• Here, we investigate integrating ProxyStore with Dask Distributed, one of the most popular libraries for distributed computing in Python, with the goal of supporting scalable and portable scientific workflows.