Alchemist: An Apache Spark <=> MPI Interface

Understand

The Apache Spark framework for distributed computation is popular in the data analytics community due to its ease of use, but its MapReduce-style programming model can incur significant overheads when performing computations that do not map directly onto this model.

• One way to mitigate these costs is to off-load computations onto MPI codes.
• In recent work, we introduced Alchemist, a system for the analysis of large-scale data sets.
• Alchemist calls MPI-based libraries from within Spark applications, and it has minimal coding, communication, and memory overheads.