Neural networks are the engine behind modern AI. Every time you use a language model, an image classifier, or a recommendation system, there’s a neural network doing the heavy lifting. But for most software engineers, the inner workings feel like a black box.
They don’t have to.
If you can understand a function, a loop, and a dictionary, you have everything you need to understand how a neural network works. This article builds that intuition from the ground up, with real code you can run.
The Core Idea
A neural network is, at its most fundamental level, a composition of functions.
You pass in an input (a number, an image, a sentence). The network applies a series of transformations. You get an output (a prediction, a classification, a probability).
The “learning” part is this: those transformations contain tunable parameters called weights and biases. Training a neural network means finding values for those parameters that make the network’s outputs consistently close to the correct answers.
That’s it. Everything else (layers, activation functions, backpropagation) is machinery that makes this work at scale and on complex data.
Building a Neural Network from Scratch
Let’s make this concrete with a minimal neural network in PyTorch.
Setup
import torch
import torch.nn as nn
import torch.optim as optimDefine the Network
class TinyNet(nn.Module):
def __init__(self):
super(TinyNet, self).__init__()
self.layer1 = nn.Linear(2, 4) # 2 inputs → 4 neurons
self.layer2 = nn.Linear(4, 1) # 4 neurons → 1 output
self.relu = nn.ReLU()
self.sigmoid = nn.Sigmoid()
def forward(self, x):
x = self.relu(self.layer1(x))
x = self.sigmoid(self.layer2(x))
return x
model = TinyNet()
print(model)Output:
TinyNet(
(layer1): Linear(in_features=2, out_features=4, bias=True)
(layer2): Linear(4, 1)
(relu): ReLU()
(sigmoid): Sigmoid()
)This network takes 2 numbers as input, runs them through 4 neurons in a hidden layer, and produces 1 output: a probability between 0 and 1. Small, but it illustrates every key concept.
Understanding the Components
Weights and Biases
nn.Linear(in, out) is a linear transformation: output = input × W + b
Wis the weight matrix (one weight per input-output connection)bis the bias vector (one bias per output neuron)
These are the parameters the network learns. Initially they’re random. Training adjusts them to reduce prediction error.
# Inspect the initial (random) weights
print("Layer 1 weights:", model.layer1.weight)
print("Layer 1 bias:", model.layer1.bias)Activation Functions
Without activation functions, stacking linear layers is mathematically equivalent to a single linear layer. Activation functions introduce non-linearity, which is what allows networks to learn complex patterns.
ReLU (Rectified Linear Unit) is the workhorse:
# ReLU: max(0, x)
# Negative values → 0. Positive values → unchanged.
relu = nn.ReLU()
print(relu(torch.tensor([-2.0, 0.0, 1.5, 3.0])))
# tensor([0.0000, 0.0000, 1.5000, 3.0000])Sigmoid squashes values to (0, 1), useful for binary classification outputs:
sigmoid = nn.Sigmoid()
print(sigmoid(torch.tensor([-2.0, 0.0, 2.0])))
# tensor([0.1192, 0.5000, 0.8808])The Forward Pass
The forward() method defines what happens when you call the model. Data flows through layers sequentially, transformed at each step:
# Create a sample input: 2 features
x = torch.tensor([[0.5, 0.8]])
# Forward pass
output = model(x)
print(f"Output: {output.item():.4f}") # e.g., Output: 0.4823Training: How Networks Learn
Loss Function
The loss function measures how wrong the model’s prediction is. For binary classification, binary cross-entropy is standard:
criterion = nn.BCELoss()Optimizer
The optimizer decides how to adjust weights given the gradients computed during backpropagation:
optimizer = optim.Adam(model.parameters(), lr=0.01)Adam is the default choice for most tasks; it adapts the learning rate per parameter and handles sparse gradients gracefully.
Training Loop
# XOR problem: a classic non-linearly-separable task
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])
for epoch in range(1000):
# 1. Forward pass
outputs = model(X)
loss = criterion(outputs, y)
# 2. Zero gradients from previous step
optimizer.zero_grad()
# 3. Backward pass — compute gradients
loss.backward()
# 4. Update weights
optimizer.step()
if (epoch + 1) % 100 == 0:
print(f'Epoch [{epoch+1}/1000], Loss: {loss.item():.4f}')Sample output:
Epoch [100/1000], Loss: 0.6842
Epoch [200/1000], Loss: 0.5931
Epoch [300/1000], Loss: 0.4287
Epoch [400/1000], Loss: 0.2943
Epoch [500/1000], Loss: 0.1876
Epoch [600/1000], Loss: 0.1204
Epoch [700/1000], Loss: 0.0823
Epoch [800/1000], Loss: 0.0601
Epoch [900/1000], Loss: 0.0469
Epoch [1000/1000], Loss: 0.0383The loss decreases. The model is learning.
Backpropagation: The Mechanism
loss.backward() is where the magic happens. Backpropagation computes the gradient of the loss with respect to every parameter in the network: how much would the loss change if we nudged each weight slightly?
PyTorch builds a computation graph during the forward pass that records every operation. backward() traverses this graph in reverse, applying the chain rule of calculus to propagate gradients from the output layer back to the input layer.
The result: every parameter has a .grad attribute containing its gradient.
# After loss.backward(), check gradients
print("Gradient for layer1 weights:", model.layer1.weight.grad)The optimizer then uses these gradients to update each parameter:
weight = weight - learning_rate × gradientThis is gradient descent. Do this thousands of times across enough data, and the network converges to a set of weights that minimize the loss.
Evaluating the Trained Model
model.eval() # Disable dropout and batch norm training behavior
with torch.no_grad(): # Disable gradient computation
predictions = model(X)
predicted_classes = (predictions > 0.5).float()
print("Predictions:")
for i in range(len(X)):
print(f" Input: {X[i].tolist()} → "
f"Raw: {predictions[i].item():.4f} → "
f"Class: {int(predicted_classes[i].item())}")Output after training:
Predictions:
Input: [0.0, 0.0] → Raw: 0.0312 → Class: 0 ✓
Input: [0.0, 1.0] → Raw: 0.9687 → Class: 1 ✓
Input: [1.0, 0.0] → Raw: 0.9701 → Class: 1 ✓
Input: [1.0, 1.0] → Raw: 0.0289 → Class: 0 ✓The network correctly learned XOR, a non-linearly separable function that a single linear model cannot solve.
The Mental Model for Engineers
If you think of a neural network as software, here’s the mapping:
| Neural Network Concept | Software Analogy |
|---|---|
| Weights & biases | Tunable configuration parameters |
| Forward pass | Function call with return value |
| Loss function | Test suite that measures prediction quality |
| Backpropagation | Automated differentiation of the error signal |
| Gradient descent | Parameter update loop that minimizes loss |
| Training | The build/compile step that produces a trained model |
| Inference | Running the compiled artifact on new inputs |
The fundamental difference from traditional software: you don’t write the logic. You define the architecture (the function shape) and the objective (the loss function), then let the optimization process find the logic through exposure to data.
What This Unlocks
Once you have this foundation, the rest of the AI landscape becomes navigable:
- Convolutional Neural Networks (CNNs): the same architecture with specialized layers for spatial data (images)
- Transformers: the architecture behind LLMs, using attention mechanisms to handle sequences
- Embeddings: compressed numerical representations that capture semantic meaning, enabling similarity search
- Fine-tuning: taking a pre-trained model and continuing training on your specific domain data
Each of these is an extension of the same core ideas: parameters, forward pass, loss, backpropagation, optimization.
The field moves fast, but the fundamentals are stable. Start here, and everything else builds naturally.