large language model training and finetuning
LitGPT is a framework for pretraining, finetuning, and deploying large language models at scale, providing from-scratch implementations with support for 20+ LLMs, distributed training, and fine-tuning techniques.
Solves the same problem with a different architecture or at a different layer. Expect to rewrite the integration.
specialized large-scale transformer training framework, more complex but powerful for extreme parallelism, not a drop-in replacement
Ongoing research training transformer models at scale
An earlier or more limited way to do the job. Still the right call when you need something small, proven, or CPU-only.
minimal from-scratch training script for small-scale models, lacks distributed and finetuning features
A straightforward method for training your LLM, from downloading data to generating text.
lightweight code to train a tiny LLM on a single GPU, not for large-scale use
🧠 Train a 64M-parameter LLM from scratch in just 2h!
These projects were analysed and named litgpt among their alternatives. The relationship is not symmetric — how litgpt rates them is a separate judgement, made when litgpt is analysed in its own right.
calls litgpt “Choose it when you want a lightning-native pretrain/finetune/serve framework with more model recipes but fewer platform orchestration features.”
Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!
calls litgpt “For developers who want a scriptable Python/Lightning training and inference library without a GUI.”
Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4, MiniMax-H3, Gemma 4, FLUX and more.
calls litgpt “choose when you want a lightweight, self-contained PyTorch codebase with pretrain/finetune recipes for a curated set of 20+ open models.”
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
calls litgpt “Choose this if you want a production-grade framework to pretrain/finetune larger LLMs at scale instead of educational from-scratch code.”
🧠 Train a 64M-parameter LLM from scratch in just 2h!
calls litgpt “Lightning-based training framework with ready-made recipes, simpler than Megatron's custom parallelism.”
Ongoing research training transformer models at scale
calls litgpt “Choose when you want a lightweight Lightning/Fabric-based pretrain-finetune-serve pipeline for a curated set of LLMs rather than the huge ms-swift model zoo.”
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).
calls litgpt “PyTorch/Lightning LLM pretraining/finetuning stack; pick if your stack is PyTorch, not PaddlePaddle.”
PaddleFormers is an easy-to-use library of pre-trained large language model zoo based on PaddlePaddle.
calls litgpt “Choose when you need a production-grade training/finetuning framework with many model recipes instead of hand-written educational code.”
A straightforward method for training your LLM, from downloading data to generating text.