Convergence of Two-Layer Regression with Nonlinear Units

Understand

Large language models (LLMs), such as ChatGPT and GPT4, have shown outstanding performance in many human life task.

• Attention computation plays an important role in training LLMs.
• Softmax unit and ReLU unit are the key structure in attention computation.
• Inspired by them, we put forward a softmax ReLU regression problem.