Memory Planning for Deep Neural Networks

Understand

We study memory allocation patterns in DNNs during inference, in the context of large-scale systems.

• We observe that such memory allocation patterns, in the context of multi-threading, are subject to high latencies, due to \texttt{mutex} contention in the system memory allocator.
• Latencies incurred due to such \texttt{mutex} contention produce undesirable bottlenecks in user-facing services.
• Thus, we propose a "memorization" based technique, \texttt{MemoMalloc}, for optimizing overall latency, with only moderate increases in peak memory usage.