large-scale LLM pretraining / distributed transformer training
Open-source library for pretraining large autoregressive language models from scratch on GPU clusters, combining Megatron-style model parallelism with DeepSpeed techniques.
Same problem, same approach. Swapping one for another is a config change, not a rewrite.
Actively maintained original Megatron framework; choose it for current upstream support and the broadest ecosystem rather than GPT-NeoX's DeepSpeed-specific integrations.
Ongoing research training transformer models at scale
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
Alibaba's patched Megatron stack for LLM/VLM training; choose it if you need its model recipes and multi-modal scale-out, while accepting Megatron-LM as the base.
The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.
Independent training framework with Kunlun XPU support and multimodal coverage; choose it if you need those accelerators or workload types, not for a Megatron-compatible codebase.
A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs.
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
Same Megatron+DeepSpeed training stack but unmaintained for over two years; choose it only when reproducing BLOOM-era work.
Ongoing research training transformer language models at scale, including: BERT & GPT-2
Older Megatron-derived distributed LLM trainer with no recent maintenance; use it only if you need that legacy codebase rather than GPT-NeoX.
distributed trainer for LLMs
Does the same job, but ships as an app. Useful to a person, not swappable into a codebase.
GUI/app research environment for launching training runs; choose it for interactive workflows, not as an embeddable Python training library.
The open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters.
These projects were analysed and named gpt-neox among their alternatives. The relationship is not symmetric — how gpt-neox rates them is a separate judgement, made when gpt-neox is analysed in its own right.
calls gpt-neox “Use when you need GPU/Megatron-DeepSpeed model-parallel training instead of JAX/TPU.”
Model parallel transformers in JAX and Haiku
calls gpt-neox “Narrower, focused on GPT-style autoregressive models with a simpler Megatron-based implementation.”
Ongoing research training transformer models at scale