Carbon Emissions and Large Neural Network Training

Understand

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs.

• Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information.
• We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer.
• We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters.