Tighter Convergence Rates for Local SGD with Data Heterogeneity

This research significantly advances the theoretical understanding of Local SGD, also known as Federated Averaging, a widely adopted distributed optimization algorithm. While Local SGD often demonstrates superior practical performance compared to alternatives like Mini-batch SGD, the theoretical underpinnings for its efficiency, especially under realistic data heterogeneity, have been incomplete. Previous work suggested that bounded second-order heterogeneity could explain Local SGD's effectiveness for strongly convex objectives, and this paper extends that principle. The authors successfully prove this conjecture by establishing a refined convergence guarantee for Local SGD when applied to general convex objectives under the assumption of bounded second-order heterogeneity. This theoretical breakthrough provides a clearer picture of when and why local updates are beneficial in distributed learning environments. Furthermore, the study improves the best-known lower bounds for Local SGD in this specific setting, demonstrating that the newly derived upper bounds are nearly tight. These combined results offer a more precise and fine-grained convergence theory for Local SGD, enhancing our ability to predict and optimize its performance. The techniques also yield a lower bound for serial SGD with replacement, illustrating how second-order heterogeneity influences the impact of rare, high-curvature clients.