Neural Networks Basics Cheat Sheet

A reference for foundational neural network concepts covering feedforward architectures in PyTorch and Keras, backpropagation, activations, and regularization.

2 PagesBeginnerMar 10, 2026

Feedforward Network in PyTorch

Define a simple multilayer perceptron.

python

import torchimport torch.nn as nnclass MLP(nn.Module):    def __init__(self, in_dim, hidden_dim, out_dim):        super().__init__()        self.net = nn.Sequential(            nn.Linear(in_dim, hidden_dim),            nn.ReLU(),            nn.Linear(hidden_dim, out_dim)        )    def forward(self, x):        return self.net(x)model = MLP(784, 128, 10)

Training Loop

The standard PyTorch train step pattern.

python

criterion = nn.CrossEntropyLoss()optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)for epoch in range(10):    for X_batch, y_batch in train_loader:        optimizer.zero_grad()        outputs = model(X_batch)        loss = criterion(outputs, y_batch)        loss.backward()          # backpropagation        optimizer.step()

Keras Equivalent

The same network defined with the Keras API.

python

from tensorflow import kerasmodel = keras.Sequential([    keras.layers.Dense(128, activation='relu', input_shape=(784,)),    keras.layers.Dense(10, activation='softmax')])model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])model.fit(X_train, y_train, epochs=10, batch_size=32, validation_split=0.1)

Key Concepts

Core theory behind neural networks.

Activation function- Introduces non-linearity (ReLU, sigmoid, tanh); without it, stacked layers collapse into one linear function
Backpropagation- Computes the loss gradient with respect to every weight via the chain rule, layer by layer
Weight initialization- Poor initialization (e.g. all zeros) causes symmetric, non-learning neurons; use He or Xavier initialization
Learning rate- Step size for gradient updates; too high diverges, too low converges very slowly
Overfitting- Combat with dropout, weight decay (L2), early stopping, or more training data
Batch size- Number of samples per gradient update; affects training stability, speed, and memory use

Pro Tip

Vanishing gradients in deep networks are often a symptom of saturating sigmoid or tanh activations — switch to ReLU (or variants like LeakyReLU or GELU) and add batch normalization to keep gradients flowing through deep stacks.

Was this cheat sheet helpful?

Principal Component Analysis Cheat Sheet

Convolutional Neural Networks Cheat Sheet

Explore Topics

#NeuralNetworksBasics#NeuralNetworksBasicsCheatSheet#DataScience#Beginner#FeedforwardNetworkInPyTorch#TrainingLoop#KerasEquivalent#KeyConcepts#Networking#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services