NVIDIA/Megatron-LMGitHub
Pick it for PyTorch-based model-parallel transformer training with broader GPU ecosystem support.
Ongoing research training transformer models at scale
- 17.7k
- Python
- NOASSERTION
active1mo ago
kingoflolz/mesh-transformer-jax Alternatives
model-parallel transformer training/inference
A JAX/Haiku library for model-parallel training and inference of large transformer models, especially on TPUs (e.g., GPT-J-6B).
Same job, different approach
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
EleutherAI/gpt-neoxGitHub
Use when you need GPU/Megatron-DeepSpeed model-parallel training instead of JAX/TPU.
An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries
- 7.5k
- Python
- Apache-2.0
active4mo ago
Older generation, or narrower
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
EleutherAI/gpt-neoGitHub
Use only for legacy Mesh-TensorFlow GPT-Neo work; it is archived and unmaintained.
An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.
- 8.3k
- Python
- MIT
archived4.6y ago
Listed as an alternative to
These projects were analysed and named mesh-transformer-jax among their alternatives. The relationship is not symmetric — how mesh-transformer-jax rates them is a separate judgement, made when mesh-transformer-jax is analysed in its own right.
NVIDIA/Megatron-LMGitHub
calls mesh-transformer-jax “Older JAX-based model-parallel training, unmaintained but relevant for JAX users.”
Ongoing research training transformer models at scale
17.7kPythonactive