On the Replicability and Reproducibility of Deep Learning in Software Engineering

Understand

Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years.

• This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge.
• Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors: (1) replicability - whether the reported experimental result can be approximately reproduced in high probability with the same DL model and the same data; and (2) reproducibility - whether one reported experimental findings can be reproduced by new experiments with the same experimental protocol and DL model, but different sampled real-world data.
• Unlike traditional machine learning (ML) models, DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work.