Deep Learning Architectures
Class Notes: From the Perceptron to the Transformer Era
- Machine Learning vs. Deep Learning
Both are subsets of AI, but they differ in how they learn features, how much data/compute they need, and how interpretable they are.
Rule of thumb: small/structured data + need for interpretability → go with ML. Large volumes of unstructured data + enough compute → go with DL.
- Perceptron → MLP → CNN → RNN → LSTM → GRU
1.1Perceptron
Year
1958 Creator: Frank Rosenblatt
What it does
A single layer of neurons mapping inputs directly to a binary output
Uses a step activation function to fire 0 or 1
Learns weights by adjusting them when it misclassifies a training example
Key problem solved
- Proved that a machine could learn to classify simple, linearly separable patterns — the birth of neural learning.
Key limitation
- Cannot solve non-linearly separable problems (famously fails on XOR); no hidden layers means very limited capacity.
Real-world example
- Classifying simple shapes (circle vs. square) under controlled lab conditions.
1.2Multi-Layer Perceptron (MLP / ANN / DNN)
Year
1980s Creator: Rumelhart, Hinton & Williams (backpropagation)
What it does
Stacks an input layer, one or more hidden layers, and an output layer, fully connected
Uses non-linear activations (Sigmoid, Tanh, ReLU) between layers
Trained end-to-end with backpropagation + gradient descent
Key problem solved
- Solved non-linear problems like XOR by adding hidden layers, enabling deep pattern learning on tabular data.
Key limitation
- Treats every input feature independently — no sense of spatial layout (images) or temporal order (sequences).
Real-world example
- Credit scoring and customer-churn prediction from tabular/CSV data.