large-scale transformer training
Megatron-LM is a GPU-optimized library for training transformer models at scale, providing advanced parallelism strategies (TP, PP, DP, EP, CP) and mixed precision support.
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
Lightning-based training framework with ready-made recipes, simpler than Megatron's custom parallelism.
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
PaddlePaddle-based transformer training library for users committed to that framework.
PaddleFormers is an easy-to-use library of pre-trained large language model zoo based on PaddlePaddle.
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
Narrower, focused on GPT-style autoregressive models with a simpler Megatron-based implementation.
An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries
Older JAX-based model-parallel training, unmaintained but relevant for JAX users.
Model parallel transformers in JAX and Haiku
Archived PyTorch training library from Meta, an older alternative for large-scale pretraining.
Repo for external large-scale work
These projects were analysed and named Megatron-LM among their alternatives. The relationship is not symmetric — how Megatron-LM rates them is a separate judgement, made when Megatron-LM is analysed in its own right.
calls Megatron-LM “Actively maintained original Megatron framework; choose it for current upstream support and the broadest ecosystem rather than GPT-NeoX's DeepSpeed-specific integrations.”
An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries
calls Megatron-LM “specialized large-scale transformer training framework, more complex but powerful for extreme parallelism, not a drop-in replacement”
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
calls Megatron-LM “PyTorch research framework for distributed transformer training; choose it for Megatron parallelism on Nvidia GPUs.”
PaddleFormers is an easy-to-use library of pre-trained large language model zoo based on PaddlePaddle.
calls Megatron-LM “Pick it for PyTorch-based model-parallel transformer training with broader GPU ecosystem support.”
Model parallel transformers in JAX and Haiku