Introduction

In this post, Jay Mody implements a Generative Pre-trained Transformer from scratch in just 60 lines of numpy. He then loads the trained GPT-2 model weights released by OpenAI into the implementation and generates some text.

This post assumes familiarity with Python, NumPy, and some basic experience training neural networks. This implementation is missing tons of features on purpose to keep it as simple as possible while remaining complete. The goal is to provide a simple yet complete technical introduction to the GPT as an educational tool.