This repository is a hands on guide to building a ChatGPT like LLM in PyTorch. It breaks the architecture into simple parts and explains each one step by step.
Let us have a birds eye view of the Generative Pretrained Transformer (GPT) like LLM architecture.
Example: Every moment is a beginning
LLMs work by predicting one word or token at a time. LLMs generate text iteratively. Each predicted word token is appended to the previous input to form the context for the next prediction.
- Tokenization
- Token Embeddings
- Positional Embeddings
- Self Attention Mechanism
- Multi-Head Self Attention
- FeedForward Neural Networks
- Residual Connections
- Layer Normalization
- Transformer Block
- Cross Entropy Loss
- Training and Generation
Dive into the hands-on examples for each LLM component using interactive Jupyter notebooks.
| Topic | Code |
|---|---|
| Tokenization | 01_tokenization.ipynb |
| Token Embeddings | 02_token_embeddings.ipynb |
| Positional Embeddings | 03_positional_embeddings.ipynb |
| Self Attention Mechanism | TODO |
| Multi-Head Self Attention | TODO |
| FeedForward Neural Networks | TODO |
| Residual Connections | TODO |
| Layer Normalization | TODO |
| Transformer Block | TODO |
| Cross Entropy Loss | TODO |
| Training and Generation | TODO |
pip install -r requirements.txt
If you're installing torch with CUDA support, make sure to use the correct installation command from PyTorch's official website, as some versions require a specific installation method.
Read the full breakdown and insights in the accompanying blogs.
- A Visual Guide to LLMs (Part 1): Text to Numbers: Tokenization and Embeddings
- A Visual Guide to LLMs (Part 2): Inside the Transformer Architecture
✅ Learn AI for FREE with visuals, easy-to-follow insights.
✅ Get cutting-edge topics like GenAI, RAGs, and LLMs in your inbox every week.
We welcome contributions! If you have improvements, new notebooks, or fixes to suggest:
- Fork the repository.
- Create a feature branch:
git checkout -b feature/YourTopic. - Add or update notebooks in the
notebooks/folder. - Commit your changes:
git commit -m 'Add or update YourTopic notebook'. - Push your branch:
git push origin feature/YourTopic. - Open a pull request for review.
This project is licensed under MIT License
⭐️ If you find this repository helpful, please consider giving it a star!
Keywords: AI, Machine Learning, Deep Learning, PyTorch, Generative AI, LLMs, Transformers
